Buildpack Cache


Stephan Merker
 

Hi,

 

is there documentation available that describes how the buildpack cache works?

 

I found the following info:

https://docs.cloudfoundry.org/concepts/how-applications-are-staged.html

“The Task downloads buildpacks and the app’s buildpack cache, if present.”

 

https://github.com/cloudfoundry/java-buildpack/blob/master/docs/extending-caches.md

 

I wonder how often the buildpacks and artifacts like JVM, Tomcat etc. are downloaded and how they are cached? Is this cache per app, per diego cell, CF global?

I’m talking about the default online buildpacks.

 

Thanks,

Stephan

 


Stephen Levine
 

Hi Stephan,

The buildpack cache (aka "app cache," or "build artifacts cache") is a per-app cache that is stored as a tarball in the blob store and recovered when an existing app is restaged (regardless of app code or buildpack changes).

Different buildpacks use this cache for different things. Ben Hale (CCd) may have more details about how the Java buildpack handles caching. In general, the purpose of the cache is to speed up build time and/or reduce data transfer during staging.


-Stephen


On Thu, Jun 14, 2018 at 10:31 AM Stephan Merker <stephan.merker@...> wrote:

Hi,

 

is there documentation available that describes how the buildpack cache works?

 

I found the following info:

https://docs.cloudfoundry.org/concepts/how-applications-are-staged.html

“The Task downloads buildpacks and the app’s buildpack cache, if present.”

 

https://github.com/cloudfoundry/java-buildpack/blob/master/docs/extending-caches.md

 

I wonder how often the buildpacks and artifacts like JVM, Tomcat etc. are downloaded and how they are cached? Is this cache per app, per diego cell, CF global?

I’m talking about the default online buildpacks.

 

Thanks,

Stephan

 


Stephan Merker
 

Thanks for the explanation.

 

I have one more question: Is there any caching for the buildpack itself?

The CC coding looks like the buildpack is downloaded for each staging operation (either as zip file or via git clone).

Is there a difference between system buildpacks and custom buildpacks when it comes to download and caching?

 

Best regards,

Stephan

 

From: cf-dev@... [mailto:cf-dev@...] On Behalf Of Stephen Levine
Sent: Donnerstag, 14. Juni 2018 22:04
To: Discussions about Cloud Foundry projects and the system overall. <cf-dev@...>; Ben Hale <bhale@...>
Subject: Re: [cf-dev] Buildpack Cache

 

Hi Stephan,

 

The buildpack cache (aka "app cache," or "build artifacts cache") is a per-app cache that is stored as a tarball in the blob store and recovered when an existing app is restaged (regardless of app code or buildpack changes).

 

Different buildpacks use this cache for different things. Ben Hale (CCd) may have more details about how the Java buildpack handles caching. In general, the purpose of the cache is to speed up build time and/or reduce data transfer during staging.

 

 

-Stephen

 

 

On Thu, Jun 14, 2018 at 10:31 AM Stephan Merker <stephan.merker@...> wrote:

Hi,

 

is there documentation available that describes how the buildpack cache works?

 

I found the following info:

https://docs.cloudfoundry.org/concepts/how-applications-are-staged.html

“The Task downloads buildpacks and the app’s buildpack cache, if present.”

 

https://github.com/cloudfoundry/java-buildpack/blob/master/docs/extending-caches.md

 

I wonder how often the buildpacks and artifacts like JVM, Tomcat etc. are downloaded and how they are cached? Is this cache per app, per diego cell, CF global?

I’m talking about the default online buildpacks.

 

Thanks,

Stephan

 


Stephen Levine
 

The system buildpacks are cached on the cell and bind mounted read-only into each staging container. Buildpack URLs (to a git repo or a zip file) are downloaded every time staging occurs.


On Mon, Jun 18, 2018 at 12:50 PM Stephan Merker <stephan.merker@...> wrote:

Thanks for the explanation.

 

I have one more question: Is there any caching for the buildpack itself?

The CC coding looks like the buildpack is downloaded for each staging operation (either as zip file or via git clone).

Is there a difference between system buildpacks and custom buildpacks when it comes to download and caching?

 

Best regards,

Stephan

 

From: cf-dev@... [mailto:cf-dev@...] On Behalf Of Stephen Levine
Sent: Donnerstag, 14. Juni 2018 22:04
To: Discussions about Cloud Foundry projects and the system overall. <cf-dev@...>; Ben Hale <bhale@...>
Subject: Re: [cf-dev] Buildpack Cache

 

Hi Stephan,

 

The buildpack cache (aka "app cache," or "build artifacts cache") is a per-app cache that is stored as a tarball in the blob store and recovered when an existing app is restaged (regardless of app code or buildpack changes).

 

Different buildpacks use this cache for different things. Ben Hale (CCd) may have more details about how the Java buildpack handles caching. In general, the purpose of the cache is to speed up build time and/or reduce data transfer during staging.

 

 

-Stephen

 

 

On Thu, Jun 14, 2018 at 10:31 AM Stephan Merker <stephan.merker@...> wrote:

Hi,

 

is there documentation available that describes how the buildpack cache works?

 

I found the following info:

https://docs.cloudfoundry.org/concepts/how-applications-are-staged.html

“The Task downloads buildpacks and the app’s buildpack cache, if present.”

 

https://github.com/cloudfoundry/java-buildpack/blob/master/docs/extending-caches.md

 

I wonder how often the buildpacks and artifacts like JVM, Tomcat etc. are downloaded and how they are cached? Is this cache per app, per diego cell, CF global?

I’m talking about the default online buildpacks.

 

Thanks,

Stephan

 


Ben Hale <bhale@...>
 

Different buildpacks use this cache for different things. Ben Hale (CCd) may have more details about how the Java buildpack handles caching. In general, the purpose of the cache is to speed up build time and/or reduce data transfer during staging.
The Java Buildpack treats the Buildpack cache as an opaque directory. This means that there are larger system policies about whether it's cached per-cell, per-foundation, etc. but from the perspective of the Java Buildpack, there is only a directory that may or may not contain cached artifacts.

All artifacts that the Java Buildpack might download are cached upon retrieval. So if you are using an online buildpack and download a JVM, we'll only download once per buildpack-cache (again, the system-wide policies are what really govern how often this happens) and store it for future staging. In addition to the artifact itself, we also store the `Last-Modified` timestamp and the `ETag` if either of these is returned when the artifact is downloaded. For all subsequent uses of an artifact, the buildpack first attempts a download with `If-Modified-Since` and `If-None-Match`. If all goes well (i.e. the artifact hasn't changed on the server), a `304 Not Modified` response will be returned and we know that our cached copies are safe for use and they'll be served up to the application without and bytes downloaded.

Cache hits are called out as part of staging output:

```
-----> Java Buildpack v4.12 (offline) | https://github.com/cloudfoundry/java-buildpack.git#5dca820
-----> Downloading Jvmkill Agent 1.12.0_RELEASE from https://java-buildpack.cloudfoundry.org/jvmkill/trusty/x86_64/jvmkill-1.12.0_RELEASE.so (found in cache)
-----> Downloading Open Jdk JRE 1.8.0_172 from https://java-buildpack.cloudfoundry.org/openjdk/trusty/x86_64/openjdk-1.8.0_172.tar.gz (found in cache)
Expanding Open Jdk JRE to .java-buildpack/open_jdk_jre (1.1s)
JVM DNS caching disabled in lieu of BOSH DNS caching
-----> Downloading Open JDK Like Memory Calculator 3.13.0_RELEASE from https://java-buildpack.cloudfoundry.org/memory-calculator/trusty/x86_64/memory-calculator-3.13.0_RELEASE.tar.gz (found in cache)
Loaded Classes: 16818, Threads: 250
-----> Downloading Client Certificate Mapper 1.6.0_RELEASE from https://java-buildpack.cloudfoundry.org/client-certificate-mapper/client-certificate-mapper-1.6.0_RELEASE.jar (found in cache)
-----> Downloading Container Security Provider 1.13.0_RELEASE from https://java-buildpack.cloudfoundry.org/container-security-provider/container-security-provider-1.13.0_RELEASE.jar (found in cache)
-----> Downloading Spring Auto Reconfiguration 2.4.0_RELEASE from https://java-buildpack.cloudfoundry.org/auto-reconfiguration/auto-reconfiguration-2.4.0_RELEASE.jar (found in cache)
```

versus:

```
-----> Java Buildpack 222ce62 | https://github.com/cloudfoundry/java-buildpack.git#222ce62
-----> Downloading Jvmkill Agent 1.14.0_RELEASE from https://java-buildpack.cloudfoundry.org/jvmkill/trusty/x86_64/jvmkill-1.14.0_RELEASE.so (0.0s)
-----> Downloading Open Jdk JRE 1.8.0_172 from https://java-buildpack.cloudfoundry.org/openjdk/trusty/x86_64/openjdk-1.8.0_172.tar.gz (0.9s)
Expanding Open Jdk JRE to .java-buildpack/open_jdk_jre (1.4s)
JVM DNS caching disabled in lieu of BOSH DNS caching
-----> Downloading Open JDK Like Memory Calculator 3.13.0_RELEASE from https://java-buildpack.cloudfoundry.org/memory-calculator/trusty/x86_64/memory-calculator-3.13.0_RELEASE.tar.gz (0.0s)
Loaded Classes: 16818, Threads: 250
-----> Downloading Client Certificate Mapper 1.6.0_RELEASE from https://java-buildpack.cloudfoundry.org/client-certificate-mapper/client-certificate-mapper-1.6.0_RELEASE.jar (0.0s)
-----> Downloading Container Security Provider 1.14.0_RELEASE from https://java-buildpack.cloudfoundry.org/container-security-provider/container-security-provider-1.14.0_RELEASE.jar (0.0s)
-----> Downloading Spring Auto Reconfiguration 2.4.0_RELEASE from https://java-buildpack.cloudfoundry.org/auto-reconfiguration/auto-reconfiguration-2.4.0_RELEASE.jar (0.0s)
```


-Ben Hale
Cloud Foundry Java Experience


Stephan Merker
 

This means that there are larger system policies about whether it's cached per-cell, per-foundation, etc.
Do you mean with "larger system policies" the usage of online vs. offline buildpacks? Or are there configuration parameters in e.g. the CC or Diego to configure the caching policy?
In other words: what is the global buildpack artifact caching policy for a standard https://github.com/cloudfoundry/cf-deployment/releases/tag/v1.40.0

Best regards,
Stephan

-----Original Message-----
From: cf-dev@lists.cloudfoundry.org [mailto:cf-dev@lists.cloudfoundry.org] On Behalf Of Ben Hale
Sent: Montag, 18. Juni 2018 20:23
To: Discussions about Cloud Foundry projects and the system overall. <cf-dev@lists.cloudfoundry.org>
Subject: Re: [cf-dev] Buildpack Cache

Different buildpacks use this cache for different things. Ben Hale (CCd) may have more details about how the Java buildpack handles caching. In general, the purpose of the cache is to speed up build time and/or reduce data transfer during staging.
The Java Buildpack treats the Buildpack cache as an opaque directory. This means that there are larger system policies about whether it's cached per-cell, per-foundation, etc. but from the perspective of the Java Buildpack, there is only a directory that may or may not contain cached artifacts.

All artifacts that the Java Buildpack might download are cached upon retrieval. So if you are using an online buildpack and download a JVM, we'll only download once per buildpack-cache (again, the system-wide policies are what really govern how often this happens) and store it for future staging. In addition to the artifact itself, we also store the `Last-Modified` timestamp and the `ETag` if either of these is returned when the artifact is downloaded. For all subsequent uses of an artifact, the buildpack first attempts a download with `If-Modified-Since` and `If-None-Match`. If all goes well (i.e. the artifact hasn't changed on the server), a `304 Not Modified` response will be returned and we know that our cached copies are safe for use and they'll be served up to the application without and bytes downloaded.

Cache hits are called out as part of staging output:

```
-----> Java Buildpack v4.12 (offline) | https://github.com/cloudfoundry/java-buildpack.git#5dca820
-----> Downloading Jvmkill Agent 1.12.0_RELEASE from https://java-buildpack.cloudfoundry.org/jvmkill/trusty/x86_64/jvmkill-1.12.0_RELEASE.so (found in cache)
-----> Downloading Open Jdk JRE 1.8.0_172 from https://java-buildpack.cloudfoundry.org/openjdk/trusty/x86_64/openjdk-1.8.0_172.tar.gz (found in cache)
Expanding Open Jdk JRE to .java-buildpack/open_jdk_jre (1.1s)
JVM DNS caching disabled in lieu of BOSH DNS caching
-----> Downloading Open JDK Like Memory Calculator 3.13.0_RELEASE from https://java-buildpack.cloudfoundry.org/memory-calculator/trusty/x86_64/memory-calculator-3.13.0_RELEASE.tar.gz (found in cache)
Loaded Classes: 16818, Threads: 250
-----> Downloading Client Certificate Mapper 1.6.0_RELEASE from https://java-buildpack.cloudfoundry.org/client-certificate-mapper/client-certificate-mapper-1.6.0_RELEASE.jar (found in cache)
-----> Downloading Container Security Provider 1.13.0_RELEASE from https://java-buildpack.cloudfoundry.org/container-security-provider/container-security-provider-1.13.0_RELEASE.jar (found in cache)
-----> Downloading Spring Auto Reconfiguration 2.4.0_RELEASE from https://java-buildpack.cloudfoundry.org/auto-reconfiguration/auto-reconfiguration-2.4.0_RELEASE.jar (found in cache)
```

versus:

```
-----> Java Buildpack 222ce62 | https://github.com/cloudfoundry/java-buildpack.git#222ce62
-----> Downloading Jvmkill Agent 1.14.0_RELEASE from https://java-buildpack.cloudfoundry.org/jvmkill/trusty/x86_64/jvmkill-1.14.0_RELEASE.so (0.0s)
-----> Downloading Open Jdk JRE 1.8.0_172 from https://java-buildpack.cloudfoundry.org/openjdk/trusty/x86_64/openjdk-1.8.0_172.tar.gz (0.9s)
Expanding Open Jdk JRE to .java-buildpack/open_jdk_jre (1.4s)
JVM DNS caching disabled in lieu of BOSH DNS caching
-----> Downloading Open JDK Like Memory Calculator 3.13.0_RELEASE from https://java-buildpack.cloudfoundry.org/memory-calculator/trusty/x86_64/memory-calculator-3.13.0_RELEASE.tar.gz (0.0s)
Loaded Classes: 16818, Threads: 250
-----> Downloading Client Certificate Mapper 1.6.0_RELEASE from https://java-buildpack.cloudfoundry.org/client-certificate-mapper/client-certificate-mapper-1.6.0_RELEASE.jar (0.0s)
-----> Downloading Container Security Provider 1.14.0_RELEASE from https://java-buildpack.cloudfoundry.org/container-security-provider/container-security-provider-1.14.0_RELEASE.jar (0.0s)
-----> Downloading Spring Auto Reconfiguration 2.4.0_RELEASE from https://java-buildpack.cloudfoundry.org/auto-reconfiguration/auto-reconfiguration-2.4.0_RELEASE.jar (0.0s)
```


-Ben Hale
Cloud Foundry Java Experience


Stephen Levine
 

The buildpack cache in CF is always per-app and foundation-wide. It can only be erased by deleting the app or dropping all cache artifacts on the platform.

The system buildpacks are cached on the cell, and if you use an offline buildpack with dependencies included, the buildpack will generally not need to download those dependencies. Therefore, they won't get cached in the app's buildpack cache (but other things might still be).


On Tue, Jun 19, 2018 at 11:04 AM Stephan Merker <stephan.merker@...> wrote:
> This means that there are larger system policies about whether it's cached per-cell, per-foundation, etc.

Do you mean with "larger system policies" the usage of online vs. offline buildpacks? Or are there configuration parameters in e.g. the CC  or Diego to configure the caching policy?
In other words: what is the global buildpack artifact caching policy for a standard https://github.com/cloudfoundry/cf-deployment/releases/tag/v1.40.0

Best regards,
Stephan

-----Original Message-----
From: cf-dev@... [mailto:cf-dev@...] On Behalf Of Ben Hale
Sent: Montag, 18. Juni 2018 20:23
To: Discussions about Cloud Foundry projects and the system overall. <cf-dev@...>
Subject: Re: [cf-dev] Buildpack Cache

> Different buildpacks use this cache for different things. Ben Hale (CCd) may have more details about how the Java buildpack handles caching. In general, the purpose of the cache is to speed up build time and/or reduce data transfer during staging.

The Java Buildpack treats the Buildpack cache as an opaque directory.  This means that there are larger system policies about whether it's cached per-cell, per-foundation, etc. but from the perspective of the Java Buildpack, there is only a directory that may or may not contain cached artifacts.

All artifacts that the Java Buildpack might download are cached upon retrieval.  So if you are using an online buildpack and download a JVM, we'll only download once per buildpack-cache (again, the system-wide policies are what really govern how often this happens) and store it for future staging.  In addition to the artifact itself, we also store the `Last-Modified` timestamp and the `ETag` if either of these is returned when the artifact is downloaded.  For all subsequent uses of an artifact, the buildpack first attempts a download with `If-Modified-Since` and `If-None-Match`.  If all goes well (i.e. the artifact hasn't changed on the server), a `304 Not Modified` response will be returned and we know that our cached copies are safe for use and they'll be served up to the application without and bytes downloaded.

Cache hits are called out as part of staging output:

```
-----> Java Buildpack v4.12 (offline) | https://github.com/cloudfoundry/java-buildpack.git#5dca820
-----> Downloading Jvmkill Agent 1.12.0_RELEASE from https://java-buildpack.cloudfoundry.org/jvmkill/trusty/x86_64/jvmkill-1.12.0_RELEASE.so (found in cache)
-----> Downloading Open Jdk JRE 1.8.0_172 from https://java-buildpack.cloudfoundry.org/openjdk/trusty/x86_64/openjdk-1.8.0_172.tar.gz (found in cache)
       Expanding Open Jdk JRE to .java-buildpack/open_jdk_jre (1.1s)
       JVM DNS caching disabled in lieu of BOSH DNS caching
-----> Downloading Open JDK Like Memory Calculator 3.13.0_RELEASE from https://java-buildpack.cloudfoundry.org/memory-calculator/trusty/x86_64/memory-calculator-3.13.0_RELEASE.tar.gz (found in cache)
       Loaded Classes: 16818, Threads: 250
-----> Downloading Client Certificate Mapper 1.6.0_RELEASE from https://java-buildpack.cloudfoundry.org/client-certificate-mapper/client-certificate-mapper-1.6.0_RELEASE.jar (found in cache)
-----> Downloading Container Security Provider 1.13.0_RELEASE from https://java-buildpack.cloudfoundry.org/container-security-provider/container-security-provider-1.13.0_RELEASE.jar (found in cache)
-----> Downloading Spring Auto Reconfiguration 2.4.0_RELEASE from https://java-buildpack.cloudfoundry.org/auto-reconfiguration/auto-reconfiguration-2.4.0_RELEASE.jar (found in cache)
```

versus:

```
-----> Java Buildpack 222ce62 | https://github.com/cloudfoundry/java-buildpack.git#222ce62
-----> Downloading Jvmkill Agent 1.14.0_RELEASE from https://java-buildpack.cloudfoundry.org/jvmkill/trusty/x86_64/jvmkill-1.14.0_RELEASE.so (0.0s)
-----> Downloading Open Jdk JRE 1.8.0_172 from https://java-buildpack.cloudfoundry.org/openjdk/trusty/x86_64/openjdk-1.8.0_172.tar.gz (0.9s)
       Expanding Open Jdk JRE to .java-buildpack/open_jdk_jre (1.4s)
       JVM DNS caching disabled in lieu of BOSH DNS caching
-----> Downloading Open JDK Like Memory Calculator 3.13.0_RELEASE from https://java-buildpack.cloudfoundry.org/memory-calculator/trusty/x86_64/memory-calculator-3.13.0_RELEASE.tar.gz (0.0s)
       Loaded Classes: 16818, Threads: 250
-----> Downloading Client Certificate Mapper 1.6.0_RELEASE from https://java-buildpack.cloudfoundry.org/client-certificate-mapper/client-certificate-mapper-1.6.0_RELEASE.jar (0.0s)
-----> Downloading Container Security Provider 1.14.0_RELEASE from https://java-buildpack.cloudfoundry.org/container-security-provider/container-security-provider-1.14.0_RELEASE.jar (0.0s)
-----> Downloading Spring Auto Reconfiguration 2.4.0_RELEASE from https://java-buildpack.cloudfoundry.org/auto-reconfiguration/auto-reconfiguration-2.4.0_RELEASE.jar (0.0s)
```


-Ben Hale
Cloud Foundry Java Experience