Hi Jack,

Thanks so much for your feedback!

Based on my conversations with CF users to date, this is definitely
something that we would want to be "opt-in" behavior; the consensus-desired
default appears to be to disallow the downloading of old/deprecated
versions.

Notably, though, what we'll continue to support is the specification of a
buildpack using the `cf push` `-b` option:

```
-b Custom buildpack by name (e.g. my-buildpack) or GIT URL
```

Buildpacks used in this manner will behave in "online" mode, meaning
they'll attempt to download dependencies from the public internet. Does
that satisfy your needs, at least in the short-term?

-m


On Mon, Apr 13, 2015 at 1:59 PM, Jack Cai <greensight(a)gmail.com> wrote:

We will no longer provide "online" buildpack releases, which download

dependencies from the public internet.

I think it would make sense to retain the ability to download additional
runtime versions on demand (that's not packaged in the buildpack) if the
user explicitly requests it. So basically it will be a hybrid model, where
the most recent versions are "cached", while old versions are still
available.

Jack


On Wed, Apr 8, 2015 at 11:36 PM, Onsi Fakhouri <ofakhouri(a)pivotal.io>
wrote:

Hey Patrick,

Sorry about that - the diego-dev-notes is an internal documentation repo
that the Diego team uses to stay on the same page and toss ideas around.

There isn't much that's terribly interesting at that link - just some
ideas on how to extend diego's existing caching capabilities to avoid
copying cached artifacts into containers (we'd mount them in
directly instead).

Happy to share more detail if there is interest.

Onsi

On Wednesday, April 8, 2015, Patrick Mueller <pmuellr(a)gmail.com> wrote:

I got a 404 on
https://github.com/pivotal-cf-experimental/diego-dev-notes/blob/master/proposals/better-buildpack-caching.md

On Wed, Apr 8, 2015 at 11:10 AM, Mike Dalessio <mdalessio(a)pivotal.io>
wrote:

Hello vcap-dev!

This email details a proposed change to how Cloud Foundry buildpacks
are packaged, with respect to the ever-increasing number of binary
dependencies being cached within them.

This proposal's permanent residence is here:

https://github.com/cloudfoundry-incubator/buildpack-packager/issues/4

Feel free to comment there or reply to this email.
------------------------------
Buildpack SizesWhere we are today

Many of you have seen, and possibly been challenged by, the enormous
sizes of some of the buildpacks that are currently shipping with cf-release.

Here's the state of the world right now, as of v205:

php-buildpack: 1.1G
ruby-buildpack: 922M
go-buildpack: 675M
python-buildpack: 654M
nodejs-buildpack: 403M
----------------------
total: 3.7G

These enormous sizes are the result of the current policy of packaging
every-version-of-everything-ever-supported ("EVOEES") within the buildpack.

Most recently, this problem was exacerbated by the fact that
buildpacks now contain binaries for two rootfses.
Why this is a problem

If continued, buildpacks will only continue to increase in size,
leading to longer and longer build and deploy times, longer test times,
slacker feedback loops, and therefore less frequent buildpack releases.

Additionally, this also means that we're shipping versions of
interpreters, web servers, and libraries that are deprecated, insecure, or
both. Feedback from CF users has made it clear that many companies view
this as an unnecessary security risk.

This policy is clearly unsustainable.
What we can do about it

There are many things being discussed to ameliorate the impact that
buildpack size is having on the operations of CF.

Notably, Onsi has proposed a change to buildpack caching, to improve
Diego staging times (link to proposal
<https://github.com/pivotal-cf-experimental/diego-dev-notes/blob/master/proposals/better-buildpack-caching.md>
).

However, there is an immediate solution available, which addresses
both the size concerns as well as the security concern: packaging fewer
binary dependencies within the buildpack.
The proposal

I'm proposing that we reduce the binary dependencies in each buildpack
in a very specific way.

Aside on terms I'll use below:

- Versions of the form "1.2.3" are broken down as:
MAJOR.MINOR.TEENY. Many language ecosystems refer to the "TEENY" as "PATCH"
interchangeably, but we're going to use "TEENY" in this proposal.
- We'll assume that TEENY gets bumped for API/ABI compatible
changes.
- We'll assume that MINOR and MAJOR get bumped when there are
API/ABI *incompatible* changes.

I'd like to move forward soon with the following changes:

1. For language interpreters/compilers, we'll package the two
most-recent TEENY versions on each MAJOR.MINOR release.
2. For all other dependencies, we'll package only the single
most-recent TEENY version on each MAJOR.MINOR release.
3. We will discontinue packaging versions of dependencies that
have been deprecated.
4. We will no longer provide "EVOEES" buildpack releases.
5. We will no longer provide "online" buildpack releases, which
download dependencies from the public internet.
6. We will document the process, and provide tooling, for CF
operators to build their own buildpacks, choosing the dependencies that
their organization wants to support or creating "online" buildpacks at
operators' discretion.

An example for #1 is that we'll go from packaging 34 versions of node v0.10.x
to only packaging two: 0.10.37 and 0.10.38.

An example for #2 is that we'll go from packaging 3 versions of nginx 1.5
in the PHP buildpack to only packaging one: 1.5.12.

An example for #3 is that we'll discontinue packaging ruby 1.9.3 in
the ruby-buildpack, which reached end-of-life in February 2015.
Outcomes

With these changes, the total buildpack size will be reduced greatly.
As an example, we expect the ruby-buildpack size to go from 922M to 338M.

We also want to set the expectation that, as new interpreter versions
are released, either for new features or (more urgently) for security
fixes, we'll release new buildpacks much more quickly than we do today. My
hope is that we'll be able to do it within 24 hours of a new release.
Planning

These changes will be relatively easy to make, since all the
buildpacks are now using a manifest.yml file to declare what's being
packaged. We expect to be able to complete this work within the next two
weeks.

Stories are in the Tracker backlog under the Epic named
"skinny-buildpacks", which you can see here:

https://www.pivotaltracker.com/epic/show/1747328

------------------------------

Please let me know how these changes will impact you and your
organizations, and let me know of any counter-proposals or variations you'd
like to consider.

Thanks,

-mike


--
You received this message because you are subscribed to the Google
Groups "Cloud Foundry Developers" group.
To view this discussion on the web visit
https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/CAGeQLZwDbON2B6cAynyJY12tCWXO8XPKSCmhCc%3D%3DBu4KsHe%3DhA%40mail.gmail.com
<https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/CAGeQLZwDbON2B6cAynyJY12tCWXO8XPKSCmhCc%3D%3DBu4KsHe%3DhA%40mail.gmail.com?utm_medium=email&utm_source=footer>
.

To unsubscribe from this group and stop receiving emails from it, send
an email to vcap-dev+unsubscribe(a)cloudfoundry.org.

--
Patrick Mueller
http://muellerware.org

--
You received this message because you are subscribed to the Google
Groups "Cloud Foundry Developers" group.
To view this discussion on the web visit
https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/CACAoQH0-dCrN6o%2B%3Ds1kn3poJSusUWbV6Zzsk29FTRs_kQfYtaQ%40mail.gmail.com
<https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/CACAoQH0-dCrN6o%2B%3Ds1kn3poJSusUWbV6Zzsk29FTRs_kQfYtaQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
.

To unsubscribe from this group and stop receiving emails from it, send
an email to vcap-dev+unsubscribe(a)cloudfoundry.org.

--
You received this message because you are subscribed to the Google
Groups "Cloud Foundry Developers" group.
To view this discussion on the web visit
https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/CAFwdB-x8JD36rQFCHOnuDpOojYpVFZ_xBH1JzoTHWaEKC5Vqog%40mail.gmail.com
<https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/CAFwdB-x8JD36rQFCHOnuDpOojYpVFZ_xBH1JzoTHWaEKC5Vqog%40mail.gmail.com?utm_medium=email&utm_source=footer>
.

To unsubscribe from this group and stop receiving emails from it, send
an email to vcap-dev+unsubscribe(a)cloudfoundry.org.

--
You received this message because you are subscribed to the Google Groups
"Cloud Foundry Developers" group.
To view this discussion on the web visit
https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/CANbAaBTNMr9QwyopJJJ6VpZWz%2BGYr-Q_xL0UsnX2ha6OzkAcag%40mail.gmail.com
<https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/CANbAaBTNMr9QwyopJJJ6VpZWz%2BGYr-Q_xL0UsnX2ha6OzkAcag%40mail.gmail.com?utm_medium=email&utm_source=footer>
.

To unsubscribe from this group and stop receiving emails from it, send an
email to vcap-dev+unsubscribe(a)cloudfoundry.org.

--
You received this message because you are subscribed to the Google Groups
"Cloud Foundry Developers" group.
To view this discussion on the web visit
https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/CAGeQLZyo9VszAcgQRCuAfQ3r%3Di2PksDq2Hkv6N-kObs053hOFg%40mail.gmail.com
<https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/CAGeQLZyo9VszAcgQRCuAfQ3r%3Di2PksDq2Hkv6N-kObs053hOFg%40mail.gmail.com?utm_medium=email&utm_source=footer>
.

To unsubscribe from this group and stop receiving emails from it, send an
email to vcap-dev+unsubscribe(a)cloudfoundry.org.

For the cached package of the buildpacks, I thought it would refuse to
provide a runtime version that's not cached. Yesterday I was playing with
the node.js buildpack and found it actually will download a non-cached
node.js runtime. Does it mean we kind of moved to the "hybrid" model I
suggested earlier in this thread? Does it work the same way for
java/go/php/ruby/python buildpacks as well?

Jack




On Mon, Apr 13, 2015 at 3:08 PM, Mike Dalessio <mdalessio(a)pivotal.io>
wrote:

Hi Jack,

Thanks so much for your feedback!

Based on my conversations with CF users to date, this is definitely
something that we would want to be "opt-in" behavior; the consensus-desired
default appears to be to disallow the downloading of old/deprecated
versions.

Notably, though, what we'll continue to support is the specification of a
buildpack using the `cf push` `-b` option:

```
-b Custom buildpack by name (e.g. my-buildpack) or GIT URL
```

Buildpacks used in this manner will behave in "online" mode, meaning
they'll attempt to download dependencies from the public internet. Does
that satisfy your needs, at least in the short-term?

-m


On Mon, Apr 13, 2015 at 1:59 PM, Jack Cai <greensight(a)gmail.com> wrote:

We will no longer provide "online" buildpack releases, which download

dependencies from the public internet.

I think it would make sense to retain the ability to download additional
runtime versions on demand (that's not packaged in the buildpack) if the
user explicitly requests it. So basically it will be a hybrid model, where
the most recent versions are "cached", while old versions are still
available.

Jack


On Wed, Apr 8, 2015 at 11:36 PM, Onsi Fakhouri <ofakhouri(a)pivotal.io>
wrote:

Hey Patrick,

Sorry about that - the diego-dev-notes is an internal documentation
repo that the Diego team uses to stay on the same page and toss ideas
around.

There isn't much that's terribly interesting at that link - just some
ideas on how to extend diego's existing caching capabilities to avoid
copying cached artifacts into containers (we'd mount them in
directly instead).

Happy to share more detail if there is interest.

Onsi

On Wednesday, April 8, 2015, Patrick Mueller <pmuellr(a)gmail.com> wrote:

I got a 404 on
https://github.com/pivotal-cf-experimental/diego-dev-notes/blob/master/proposals/better-buildpack-caching.md

On Wed, Apr 8, 2015 at 11:10 AM, Mike Dalessio <mdalessio(a)pivotal.io>
wrote:

Hello vcap-dev!

This email details a proposed change to how Cloud Foundry buildpacks
are packaged, with respect to the ever-increasing number of binary
dependencies being cached within them.

This proposal's permanent residence is here:

https://github.com/cloudfoundry-incubator/buildpack-packager/issues/4

Feel free to comment there or reply to this email.
------------------------------
Buildpack SizesWhere we are today

Many of you have seen, and possibly been challenged by, the enormous
sizes of some of the buildpacks that are currently shipping with cf-release.

Here's the state of the world right now, as of v205:

php-buildpack: 1.1G
ruby-buildpack: 922M
go-buildpack: 675M
python-buildpack: 654M
nodejs-buildpack: 403M
----------------------
total: 3.7G

These enormous sizes are the result of the current policy of
packaging every-version-of-everything-ever-supported ("EVOEES") within the
buildpack.

Most recently, this problem was exacerbated by the fact that
buildpacks now contain binaries for two rootfses.
Why this is a problem

If continued, buildpacks will only continue to increase in size,
leading to longer and longer build and deploy times, longer test times,
slacker feedback loops, and therefore less frequent buildpack releases.

Additionally, this also means that we're shipping versions of
interpreters, web servers, and libraries that are deprecated, insecure, or
both. Feedback from CF users has made it clear that many companies view
this as an unnecessary security risk.

This policy is clearly unsustainable.
What we can do about it

There are many things being discussed to ameliorate the impact that
buildpack size is having on the operations of CF.

Notably, Onsi has proposed a change to buildpack caching, to improve
Diego staging times (link to proposal
<https://github.com/pivotal-cf-experimental/diego-dev-notes/blob/master/proposals/better-buildpack-caching.md>
).

However, there is an immediate solution available, which addresses
both the size concerns as well as the security concern: packaging fewer
binary dependencies within the buildpack.
The proposal

I'm proposing that we reduce the binary dependencies in each
buildpack in a very specific way.

Aside on terms I'll use below:

- Versions of the form "1.2.3" are broken down as:
MAJOR.MINOR.TEENY. Many language ecosystems refer to the "TEENY" as "PATCH"
interchangeably, but we're going to use "TEENY" in this proposal.
- We'll assume that TEENY gets bumped for API/ABI compatible
changes.
- We'll assume that MINOR and MAJOR get bumped when there are
API/ABI *incompatible* changes.

I'd like to move forward soon with the following changes:

1. For language interpreters/compilers, we'll package the two
most-recent TEENY versions on each MAJOR.MINOR release.
2. For all other dependencies, we'll package only the single
most-recent TEENY version on each MAJOR.MINOR release.
3. We will discontinue packaging versions of dependencies that
have been deprecated.
4. We will no longer provide "EVOEES" buildpack releases.
5. We will no longer provide "online" buildpack releases, which
download dependencies from the public internet.
6. We will document the process, and provide tooling, for CF
operators to build their own buildpacks, choosing the dependencies that
their organization wants to support or creating "online" buildpacks at
operators' discretion.

An example for #1 is that we'll go from packaging 34 versions of node v0.10.x
to only packaging two: 0.10.37 and 0.10.38.

An example for #2 is that we'll go from packaging 3 versions of nginx 1.5
in the PHP buildpack to only packaging one: 1.5.12.

An example for #3 is that we'll discontinue packaging ruby 1.9.3 in
the ruby-buildpack, which reached end-of-life in February 2015.
Outcomes

With these changes, the total buildpack size will be reduced greatly.
As an example, we expect the ruby-buildpack size to go from 922M to 338M.

We also want to set the expectation that, as new interpreter versions
are released, either for new features or (more urgently) for security
fixes, we'll release new buildpacks much more quickly than we do today. My
hope is that we'll be able to do it within 24 hours of a new release.
Planning

These changes will be relatively easy to make, since all the
buildpacks are now using a manifest.yml file to declare what's being
packaged. We expect to be able to complete this work within the next two
weeks.

Stories are in the Tracker backlog under the Epic named
"skinny-buildpacks", which you can see here:

https://www.pivotaltracker.com/epic/show/1747328

------------------------------

Please let me know how these changes will impact you and your
organizations, and let me know of any counter-proposals or variations you'd
like to consider.

Thanks,

-mike


--
You received this message because you are subscribed to the Google
Groups "Cloud Foundry Developers" group.
To view this discussion on the web visit
https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/CAGeQLZwDbON2B6cAynyJY12tCWXO8XPKSCmhCc%3D%3DBu4KsHe%3DhA%40mail.gmail.com
<https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/CAGeQLZwDbON2B6cAynyJY12tCWXO8XPKSCmhCc%3D%3DBu4KsHe%3DhA%40mail.gmail.com?utm_medium=email&utm_source=footer>
.

To unsubscribe from this group and stop receiving emails from it,
send an email to vcap-dev+unsubscribe(a)cloudfoundry.org.

--
Patrick Mueller
http://muellerware.org

--
You received this message because you are subscribed to the Google
Groups "Cloud Foundry Developers" group.
To view this discussion on the web visit
https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/CACAoQH0-dCrN6o%2B%3Ds1kn3poJSusUWbV6Zzsk29FTRs_kQfYtaQ%40mail.gmail.com
<https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/CACAoQH0-dCrN6o%2B%3Ds1kn3poJSusUWbV6Zzsk29FTRs_kQfYtaQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
.

To unsubscribe from this group and stop receiving emails from it, send
an email to vcap-dev+unsubscribe(a)cloudfoundry.org.

--
You received this message because you are subscribed to the Google
Groups "Cloud Foundry Developers" group.
To view this discussion on the web visit
https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/CAFwdB-x8JD36rQFCHOnuDpOojYpVFZ_xBH1JzoTHWaEKC5Vqog%40mail.gmail.com
<https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/CAFwdB-x8JD36rQFCHOnuDpOojYpVFZ_xBH1JzoTHWaEKC5Vqog%40mail.gmail.com?utm_medium=email&utm_source=footer>
.

To unsubscribe from this group and stop receiving emails from it, send
an email to vcap-dev+unsubscribe(a)cloudfoundry.org.

--
You received this message because you are subscribed to the Google
Groups "Cloud Foundry Developers" group.
To view this discussion on the web visit
https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/CANbAaBTNMr9QwyopJJJ6VpZWz%2BGYr-Q_xL0UsnX2ha6OzkAcag%40mail.gmail.com
<https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/CANbAaBTNMr9QwyopJJJ6VpZWz%2BGYr-Q_xL0UsnX2ha6OzkAcag%40mail.gmail.com?utm_medium=email&utm_source=footer>
.

To unsubscribe from this group and stop receiving emails from it, send
an email to vcap-dev+unsubscribe(a)cloudfoundry.org.

--
You received this message because you are subscribed to the Google Groups
"Cloud Foundry Developers" group.
To view this discussion on the web visit
https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/CAGeQLZyo9VszAcgQRCuAfQ3r%3Di2PksDq2Hkv6N-kObs053hOFg%40mail.gmail.com
<https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/CAGeQLZyo9VszAcgQRCuAfQ3r%3Di2PksDq2Hkv6N-kObs053hOFg%40mail.gmail.com?utm_medium=email&utm_source=footer>
.

To unsubscribe from this group and stop receiving emails from it, send an
email to vcap-dev+unsubscribe(a)cloudfoundry.org.

Thanks Mike. Is it true that even for the online version (aka when doing
"cf push -b https://github.com/cloudfoundry/nodejs-buildpack.git"), users
are now limited to use the runtime versions defined in the manifest.yml?

Jack


On Thu, Nov 12, 2015 at 12:05 PM, Mike Dalessio <mdalessio(a)pivotal.io>
wrote:

This is a bug in the nodejs-buildpack v1.5.1, which we should have a fix
for later today.

Github issue is here:
https://github.com/cloudfoundry/nodejs-buildpack/issues/35

Tracker story is here:
https://www.pivotaltracker.com/story/show/107946000

Apologies for the inconvenience.

On Thu, Nov 12, 2015 at 12:03 PM, Jack Cai <greensight(a)gmail.com> wrote:

For the cached package of the buildpacks, I thought it would refuse to
provide a runtime version that's not cached. Yesterday I was playing with
the node.js buildpack and found it actually will download a non-cached
node.js runtime. Does it mean we kind of moved to the "hybrid" model I
suggested earlier in this thread? Does it work the same way for
java/go/php/ruby/python buildpacks as well?

Jack




On Mon, Apr 13, 2015 at 3:08 PM, Mike Dalessio <mdalessio(a)pivotal.io>
wrote:

Hi Jack,

Thanks so much for your feedback!

Based on my conversations with CF users to date, this is definitely
something that we would want to be "opt-in" behavior; the consensus-desired
default appears to be to disallow the downloading of old/deprecated
versions.

Notably, though, what we'll continue to support is the specification of
a buildpack using the `cf push` `-b` option:

```
-b Custom buildpack by name (e.g. my-buildpack) or GIT URL
```

Buildpacks used in this manner will behave in "online" mode, meaning
they'll attempt to download dependencies from the public internet. Does
that satisfy your needs, at least in the short-term?

-m


On Mon, Apr 13, 2015 at 1:59 PM, Jack Cai <greensight(a)gmail.com> wrote:

We will no longer provide "online" buildpack releases, which

download dependencies from the public internet.

I think it would make sense to retain the ability to download
additional runtime versions on demand (that's not packaged in the
buildpack) if the user explicitly requests it. So basically it will be a
hybrid model, where the most recent versions are "cached", while old
versions are still available.

Jack


On Wed, Apr 8, 2015 at 11:36 PM, Onsi Fakhouri <ofakhouri(a)pivotal.io>
wrote:

Hey Patrick,

Sorry about that - the diego-dev-notes is an internal documentation
repo that the Diego team uses to stay on the same page and toss ideas
around.

There isn't much that's terribly interesting at that link - just some
ideas on how to extend diego's existing caching capabilities to avoid
copying cached artifacts into containers (we'd mount them in
directly instead).

Happy to share more detail if there is interest.

Onsi

On Wednesday, April 8, 2015, Patrick Mueller <pmuellr(a)gmail.com>
wrote:

I got a 404 on
https://github.com/pivotal-cf-experimental/diego-dev-notes/blob/master/proposals/better-buildpack-caching.md

On Wed, Apr 8, 2015 at 11:10 AM, Mike Dalessio <mdalessio(a)pivotal.io

wrote:

Hello vcap-dev!

This email details a proposed change to how Cloud Foundry
buildpacks are packaged, with respect to the ever-increasing number of
binary dependencies being cached within them.

This proposal's permanent residence is here:


https://github.com/cloudfoundry-incubator/buildpack-packager/issues/4

Feel free to comment there or reply to this email.
------------------------------
Buildpack SizesWhere we are today

Many of you have seen, and possibly been challenged by, the
enormous sizes of some of the buildpacks that are currently shipping with
cf-release.

Here's the state of the world right now, as of v205:

php-buildpack: 1.1G
ruby-buildpack: 922M
go-buildpack: 675M
python-buildpack: 654M
nodejs-buildpack: 403M
----------------------
total: 3.7G

These enormous sizes are the result of the current policy of
packaging every-version-of-everything-ever-supported ("EVOEES") within the
buildpack.

Most recently, this problem was exacerbated by the fact that
buildpacks now contain binaries for two rootfses.
Why this is a problem

If continued, buildpacks will only continue to increase in size,
leading to longer and longer build and deploy times, longer test times,
slacker feedback loops, and therefore less frequent buildpack releases.

Additionally, this also means that we're shipping versions of
interpreters, web servers, and libraries that are deprecated, insecure, or
both. Feedback from CF users has made it clear that many companies view
this as an unnecessary security risk.

This policy is clearly unsustainable.
What we can do about it

There are many things being discussed to ameliorate the impact that
buildpack size is having on the operations of CF.

Notably, Onsi has proposed a change to buildpack caching, to
improve Diego staging times (link to proposal
<https://github.com/pivotal-cf-experimental/diego-dev-notes/blob/master/proposals/better-buildpack-caching.md>
).

However, there is an immediate solution available, which addresses
both the size concerns as well as the security concern: packaging fewer
binary dependencies within the buildpack.
The proposal

I'm proposing that we reduce the binary dependencies in each
buildpack in a very specific way.

Aside on terms I'll use below:

- Versions of the form "1.2.3" are broken down as:
MAJOR.MINOR.TEENY. Many language ecosystems refer to the "TEENY" as "PATCH"
interchangeably, but we're going to use "TEENY" in this proposal.
- We'll assume that TEENY gets bumped for API/ABI compatible
changes.
- We'll assume that MINOR and MAJOR get bumped when there are
API/ABI *incompatible* changes.

I'd like to move forward soon with the following changes:

1. For language interpreters/compilers, we'll package the two
most-recent TEENY versions on each MAJOR.MINOR release.
2. For all other dependencies, we'll package only the single
most-recent TEENY version on each MAJOR.MINOR release.
3. We will discontinue packaging versions of dependencies that
have been deprecated.
4. We will no longer provide "EVOEES" buildpack releases.
5. We will no longer provide "online" buildpack releases, which
download dependencies from the public internet.
6. We will document the process, and provide tooling, for CF
operators to build their own buildpacks, choosing the dependencies that
their organization wants to support or creating "online" buildpacks at
operators' discretion.

An example for #1 is that we'll go from packaging 34 versions of
node v0.10.x to only packaging two: 0.10.37 and 0.10.38.

An example for #2 is that we'll go from packaging 3 versions of
nginx 1.5 in the PHP buildpack to only packaging one: 1.5.12.

An example for #3 is that we'll discontinue packaging ruby 1.9.3 in
the ruby-buildpack, which reached end-of-life in February 2015.
Outcomes

With these changes, the total buildpack size will be reduced
greatly. As an example, we expect the ruby-buildpack size to go from 922M
to 338M.

We also want to set the expectation that, as new interpreter
versions are released, either for new features or (more urgently) for
security fixes, we'll release new buildpacks much more quickly than we do
today. My hope is that we'll be able to do it within 24 hours of a new
release.
Planning

These changes will be relatively easy to make, since all the
buildpacks are now using a manifest.yml file to declare what's
being packaged. We expect to be able to complete this work within the next
two weeks.

Stories are in the Tracker backlog under the Epic named
"skinny-buildpacks", which you can see here:

https://www.pivotaltracker.com/epic/show/1747328

------------------------------

Please let me know how these changes will impact you and your
organizations, and let me know of any counter-proposals or variations you'd
like to consider.

Thanks,

-mike


--
You received this message because you are subscribed to the Google
Groups "Cloud Foundry Developers" group.
To view this discussion on the web visit
https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/CAGeQLZwDbON2B6cAynyJY12tCWXO8XPKSCmhCc%3D%3DBu4KsHe%3DhA%40mail.gmail.com
<https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/CAGeQLZwDbON2B6cAynyJY12tCWXO8XPKSCmhCc%3D%3DBu4KsHe%3DhA%40mail.gmail.com?utm_medium=email&utm_source=footer>
.

To unsubscribe from this group and stop receiving emails from it,
send an email to vcap-dev+unsubscribe(a)cloudfoundry.org.

--
Patrick Mueller
http://muellerware.org

--
You received this message because you are subscribed to the Google
Groups "Cloud Foundry Developers" group.
To view this discussion on the web visit
https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/CACAoQH0-dCrN6o%2B%3Ds1kn3poJSusUWbV6Zzsk29FTRs_kQfYtaQ%40mail.gmail.com
<https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/CACAoQH0-dCrN6o%2B%3Ds1kn3poJSusUWbV6Zzsk29FTRs_kQfYtaQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
.

To unsubscribe from this group and stop receiving emails from it,
send an email to vcap-dev+unsubscribe(a)cloudfoundry.org.

--
You received this message because you are subscribed to the Google
Groups "Cloud Foundry Developers" group.
To view this discussion on the web visit
https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/CAFwdB-x8JD36rQFCHOnuDpOojYpVFZ_xBH1JzoTHWaEKC5Vqog%40mail.gmail.com
<https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/CAFwdB-x8JD36rQFCHOnuDpOojYpVFZ_xBH1JzoTHWaEKC5Vqog%40mail.gmail.com?utm_medium=email&utm_source=footer>
.

To unsubscribe from this group and stop receiving emails from it,
send an email to vcap-dev+unsubscribe(a)cloudfoundry.org.

--
You received this message because you are subscribed to the Google
Groups "Cloud Foundry Developers" group.
To view this discussion on the web visit
https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/CANbAaBTNMr9QwyopJJJ6VpZWz%2BGYr-Q_xL0UsnX2ha6OzkAcag%40mail.gmail.com
<https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/CANbAaBTNMr9QwyopJJJ6VpZWz%2BGYr-Q_xL0UsnX2ha6OzkAcag%40mail.gmail.com?utm_medium=email&utm_source=footer>
.

To unsubscribe from this group and stop receiving emails from it, send
an email to vcap-dev+unsubscribe(a)cloudfoundry.org.

--
You received this message because you are subscribed to the Google
Groups "Cloud Foundry Developers" group.
To view this discussion on the web visit
https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/CAGeQLZyo9VszAcgQRCuAfQ3r%3Di2PksDq2Hkv6N-kObs053hOFg%40mail.gmail.com
<https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/CAGeQLZyo9VszAcgQRCuAfQ3r%3Di2PksDq2Hkv6N-kObs053hOFg%40mail.gmail.com?utm_medium=email&utm_source=footer>
.

To unsubscribe from this group and stop receiving emails from it, send
an email to vcap-dev+unsubscribe(a)cloudfoundry.org.

Jack,

This is correct.

The online version and the offline version of the buildpack only differ
one way. The offline version has the dependencies, defined in the
`manifest.yml`, packaged with it.

They both keep the same contract that we'll only support certain versions
of runtimes (Ruby, Python, etc).


-- JT

On Thu, Nov 12, 2015 at 2:46 PM, Jack Cai <greensight(a)gmail.com> wrote:

Thanks Mike. Is it true that even for the online version (aka when doing
"cf push -b https://github.com/cloudfoundry/nodejs-buildpack.git"),
users are now limited to use the runtime versions defined in the
manifest.yml?

Jack


On Thu, Nov 12, 2015 at 12:05 PM, Mike Dalessio <mdalessio(a)pivotal.io>
wrote:

This is a bug in the nodejs-buildpack v1.5.1, which we should have a fix
for later today.

Github issue is here:
https://github.com/cloudfoundry/nodejs-buildpack/issues/35

Tracker story is here:
https://www.pivotaltracker.com/story/show/107946000

Apologies for the inconvenience.

On Thu, Nov 12, 2015 at 12:03 PM, Jack Cai <greensight(a)gmail.com> wrote:

For the cached package of the buildpacks, I thought it would refuse to
provide a runtime version that's not cached. Yesterday I was playing with
the node.js buildpack and found it actually will download a non-cached
node.js runtime. Does it mean we kind of moved to the "hybrid" model I
suggested earlier in this thread? Does it work the same way for
java/go/php/ruby/python buildpacks as well?

Jack




On Mon, Apr 13, 2015 at 3:08 PM, Mike Dalessio <mdalessio(a)pivotal.io>
wrote:

Hi Jack,

Thanks so much for your feedback!

Based on my conversations with CF users to date, this is definitely
something that we would want to be "opt-in" behavior; the consensus-desired
default appears to be to disallow the downloading of old/deprecated
versions.

Notably, though, what we'll continue to support is the specification
of a buildpack using the `cf push` `-b` option:

```
-b Custom buildpack by name (e.g. my-buildpack) or GIT URL
```

Buildpacks used in this manner will behave in "online" mode, meaning
they'll attempt to download dependencies from the public internet. Does
that satisfy your needs, at least in the short-term?

-m


On Mon, Apr 13, 2015 at 1:59 PM, Jack Cai <greensight(a)gmail.com>
wrote:

We will no longer provide "online" buildpack releases, which

download dependencies from the public internet.

I think it would make sense to retain the ability to download
additional runtime versions on demand (that's not packaged in the
buildpack) if the user explicitly requests it. So basically it will be a
hybrid model, where the most recent versions are "cached", while old
versions are still available.

Jack


On Wed, Apr 8, 2015 at 11:36 PM, Onsi Fakhouri <ofakhouri(a)pivotal.io>
wrote:

Hey Patrick,

Sorry about that - the diego-dev-notes is an internal documentation
repo that the Diego team uses to stay on the same page and toss ideas
around.

There isn't much that's terribly interesting at that link - just
some ideas on how to extend diego's existing caching capabilities to avoid
copying cached artifacts into containers (we'd mount them in
directly instead).

Happy to share more detail if there is interest.

Onsi

On Wednesday, April 8, 2015, Patrick Mueller <pmuellr(a)gmail.com>
wrote:

I got a 404 on
https://github.com/pivotal-cf-experimental/diego-dev-notes/blob/master/proposals/better-buildpack-caching.md

On Wed, Apr 8, 2015 at 11:10 AM, Mike Dalessio <
mdalessio(a)pivotal.io> wrote:

Hello vcap-dev!

This email details a proposed change to how Cloud Foundry
buildpacks are packaged, with respect to the ever-increasing number of
binary dependencies being cached within them.

This proposal's permanent residence is here:


https://github.com/cloudfoundry-incubator/buildpack-packager/issues/4

Feel free to comment there or reply to this email.
------------------------------
Buildpack SizesWhere we are today

Many of you have seen, and possibly been challenged by, the
enormous sizes of some of the buildpacks that are currently shipping with
cf-release.

Here's the state of the world right now, as of v205:

php-buildpack: 1.1G
ruby-buildpack: 922M
go-buildpack: 675M
python-buildpack: 654M
nodejs-buildpack: 403M
----------------------
total: 3.7G

These enormous sizes are the result of the current policy of
packaging every-version-of-everything-ever-supported ("EVOEES") within the
buildpack.

Most recently, this problem was exacerbated by the fact that
buildpacks now contain binaries for two rootfses.
Why this is a problem

If continued, buildpacks will only continue to increase in size,
leading to longer and longer build and deploy times, longer test times,
slacker feedback loops, and therefore less frequent buildpack releases.

Additionally, this also means that we're shipping versions of
interpreters, web servers, and libraries that are deprecated, insecure, or
both. Feedback from CF users has made it clear that many companies view
this as an unnecessary security risk.

This policy is clearly unsustainable.
What we can do about it

There are many things being discussed to ameliorate the impact
that buildpack size is having on the operations of CF.

Notably, Onsi has proposed a change to buildpack caching, to
improve Diego staging times (link to proposal
<https://github.com/pivotal-cf-experimental/diego-dev-notes/blob/master/proposals/better-buildpack-caching.md>
).

However, there is an immediate solution available, which addresses
both the size concerns as well as the security concern: packaging fewer
binary dependencies within the buildpack.
The proposal

I'm proposing that we reduce the binary dependencies in each
buildpack in a very specific way.

Aside on terms I'll use below:

- Versions of the form "1.2.3" are broken down as:
MAJOR.MINOR.TEENY. Many language ecosystems refer to the "TEENY" as "PATCH"
interchangeably, but we're going to use "TEENY" in this proposal.
- We'll assume that TEENY gets bumped for API/ABI compatible
changes.
- We'll assume that MINOR and MAJOR get bumped when there are
API/ABI *incompatible* changes.

I'd like to move forward soon with the following changes:

1. For language interpreters/compilers, we'll package the two
most-recent TEENY versions on each MAJOR.MINOR release.
2. For all other dependencies, we'll package only the single
most-recent TEENY version on each MAJOR.MINOR release.
3. We will discontinue packaging versions of dependencies that
have been deprecated.
4. We will no longer provide "EVOEES" buildpack releases.
5. We will no longer provide "online" buildpack releases,
which download dependencies from the public internet.
6. We will document the process, and provide tooling, for CF
operators to build their own buildpacks, choosing the dependencies that
their organization wants to support or creating "online" buildpacks at
operators' discretion.

An example for #1 is that we'll go from packaging 34 versions of
node v0.10.x to only packaging two: 0.10.37 and 0.10.38.

An example for #2 is that we'll go from packaging 3 versions of
nginx 1.5 in the PHP buildpack to only packaging one: 1.5.12.

An example for #3 is that we'll discontinue packaging ruby 1.9.3
in the ruby-buildpack, which reached end-of-life in February 2015.
Outcomes

With these changes, the total buildpack size will be reduced
greatly. As an example, we expect the ruby-buildpack size to go from 922M
to 338M.

We also want to set the expectation that, as new interpreter
versions are released, either for new features or (more urgently) for
security fixes, we'll release new buildpacks much more quickly than we do
today. My hope is that we'll be able to do it within 24 hours of a new
release.
Planning

These changes will be relatively easy to make, since all the
buildpacks are now using a manifest.yml file to declare what's
being packaged. We expect to be able to complete this work within the next
two weeks.

Stories are in the Tracker backlog under the Epic named
"skinny-buildpacks", which you can see here:

https://www.pivotaltracker.com/epic/show/1747328

------------------------------

Please let me know how these changes will impact you and your
organizations, and let me know of any counter-proposals or variations you'd
like to consider.

Thanks,

-mike


--
You received this message because you are subscribed to the Google
Groups "Cloud Foundry Developers" group.
To view this discussion on the web visit
https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/CAGeQLZwDbON2B6cAynyJY12tCWXO8XPKSCmhCc%3D%3DBu4KsHe%3DhA%40mail.gmail.com
<https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/CAGeQLZwDbON2B6cAynyJY12tCWXO8XPKSCmhCc%3D%3DBu4KsHe%3DhA%40mail.gmail.com?utm_medium=email&utm_source=footer>
.

To unsubscribe from this group and stop receiving emails from it,
send an email to vcap-dev+unsubscribe(a)cloudfoundry.org.

--
Patrick Mueller
http://muellerware.org

--
You received this message because you are subscribed to the Google
Groups "Cloud Foundry Developers" group.
To view this discussion on the web visit
https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/CACAoQH0-dCrN6o%2B%3Ds1kn3poJSusUWbV6Zzsk29FTRs_kQfYtaQ%40mail.gmail.com
<https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/CACAoQH0-dCrN6o%2B%3Ds1kn3poJSusUWbV6Zzsk29FTRs_kQfYtaQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
.

To unsubscribe from this group and stop receiving emails from it,
send an email to vcap-dev+unsubscribe(a)cloudfoundry.org.

--
You received this message because you are subscribed to the Google
Groups "Cloud Foundry Developers" group.
To view this discussion on the web visit
https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/CAFwdB-x8JD36rQFCHOnuDpOojYpVFZ_xBH1JzoTHWaEKC5Vqog%40mail.gmail.com
<https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/CAFwdB-x8JD36rQFCHOnuDpOojYpVFZ_xBH1JzoTHWaEKC5Vqog%40mail.gmail.com?utm_medium=email&utm_source=footer>
.

To unsubscribe from this group and stop receiving emails from it,
send an email to vcap-dev+unsubscribe(a)cloudfoundry.org.

--
You received this message because you are subscribed to the Google
Groups "Cloud Foundry Developers" group.
To view this discussion on the web visit
https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/CANbAaBTNMr9QwyopJJJ6VpZWz%2BGYr-Q_xL0UsnX2ha6OzkAcag%40mail.gmail.com
<https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/CANbAaBTNMr9QwyopJJJ6VpZWz%2BGYr-Q_xL0UsnX2ha6OzkAcag%40mail.gmail.com?utm_medium=email&utm_source=footer>
.

To unsubscribe from this group and stop receiving emails from it,
send an email to vcap-dev+unsubscribe(a)cloudfoundry.org.

--
You received this message because you are subscribed to the Google
Groups "Cloud Foundry Developers" group.
To view this discussion on the web visit
https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/CAGeQLZyo9VszAcgQRCuAfQ3r%3Di2PksDq2Hkv6N-kObs053hOFg%40mail.gmail.com
<https://groups.google.com/a/cloudfoundry.org/d/msgid/vcap-dev/CAGeQLZyo9VszAcgQRCuAfQ3r%3Di2PksDq2Hkv6N-kObs053hOFg%40mail.gmail.com?utm_medium=email&utm_source=footer>
.

To unsubscribe from this group and stop receiving emails from it, send
an email to vcap-dev+unsubscribe(a)cloudfoundry.org.