Update deployment with new packages leaves old packages on deployed machine


Alexander Claus
 

Hey,
first: sorry if this question is already here somewhere in the list. Unfortunately the search is not working. I always get:
Sorry no email could be found for this query.
No matter, which search term I enter. Am I doing anything wrong or is the search defect?

My real issue:
When updating an existing deployment with a new release, artifacts from the former release still reside on the deployment vm after the finished update.
In our case we have multiple packages which are not marginal in size, each around 100MB. In sum we have therefore 500-600MB for the packages which form the release.
When we now update an existing deployment, one can see during the update via 'bosh deployments' that the deployment has the former and the new release, and after the 'bosh deploy' command for the update finished the same view shows only the new release. So far so good.
When going via 'bosh ssh' to the machine, one can see that in /var/vcap/data/packages/<package-name> multiple folders exist: for the current version of the package and for the former one.
I checked with multiple successive updates (versions 1>2>3>4, ..), that this mechanism only holds the last version in parallel, but not more.
Furthermore I found out that the naming of the folder containing the different versions of the packages is obviously depending on the CPI. On bosh-lite one can see a relation to the package fingerprints and SHA1 of compiled packages. On Bosh(a)AWS I cannot recognize any relation to any fingerprint or SHA1 or blobstore id or whatever.

Here comes my question:
Is it a bug or a feature that the old packages remain on the deployed vm after a finished update? I could not find anything about this in the bosh documentation.

Since in our case the disk space used by these duplicate unused packages is significant: Is there any way to deactivate this "feature", i.e. only to keep the current version of a package?
I know that I could do the update via 'bosh deploy --recreate' and the result would be a machine with only the needed package version, but during that the complete vm is destroyed and recreated which takes significant amount of time depending on the cloud provider. So I'd favour an option to update without '--recreate'.

Hope that someone can help.
Alexander


Benjamin Gandon
 

Sounds like a workaround to me.

I'm not aware of any feature in BOSH that would need old packages to stay around on BOSH instances.

But it could be due to BOSH delegating process restarts to Monit and not knowing exactly when the old binaries are not needed anymore. Sure Monit has specs about how restarting things, but it also has its own agenda.

This is only a guess, though. The truth is in the code. Or when Dmitry speaks. :)


One other observation is that if the downtime implied by '--recreate' is an issue to you, then you might be operating Pets instead of Cattle and my advice is that you should pay attention for any bigger reliability problem there.

Using '--recreate' is something that should be done harmlessly on a regular basis. Like with the 'repave' part of the rotate-repave-repair 3R's of security.

Le 3 mai 2017 à 16:58, Alexander Claus <alexander.claus(a)sap.com> a écrit :

Hey,
first: sorry if this question is already here somewhere in the list. Unfortunately the search is not working. I always get:
Sorry no email could be found for this query.
No matter, which search term I enter. Am I doing anything wrong or is the search defect?

My real issue:
When updating an existing deployment with a new release, artifacts from the former release still reside on the deployment vm after the finished update.
In our case we have multiple packages which are not marginal in size, each around 100MB. In sum we have therefore 500-600MB for the packages which form the release.
When we now update an existing deployment, one can see during the update via 'bosh deployments' that the deployment has the former and the new release, and after the 'bosh deploy' command for the update finished the same view shows only the new release. So far so good.
When going via 'bosh ssh' to the machine, one can see that in /var/vcap/data/packages/<package-name> multiple folders exist: for the current version of the package and for the former one.
I checked with multiple successive updates (versions 1>2>3>4, ..), that this mechanism only holds the last version in parallel, but not more.
Furthermore I found out that the naming of the folder containing the different versions of the packages is obviously depending on the CPI. On bosh-lite one can see a relation to the package fingerprints and SHA1 of compiled packages. On Bosh(a)AWS I cannot recognize any relation to any fingerprint or SHA1 or blobstore id or whatever.

Here comes my question:
Is it a bug or a feature that the old packages remain on the deployed vm after a finished update? I could not find anything about this in the bosh documentation.

Since in our case the disk space used by these duplicate unused packages is significant: Is there any way to deactivate this "feature", i.e. only to keep the current version of a package?
I know that I could do the update via 'bosh deploy --recreate' and the result would be a machine with only the needed package version, but during that the complete vm is destroyed and recreated which takes significant amount of time depending on the cloud provider. So I'd favour an option to update without '--recreate'.

Hope that someone can help.
Alexander


Tyler Schultz
 

A copy of the old package is kept around so that reverting to the previous
version requires less downtime. This minimizes downtime in the case of a
failed deploy. This is an intended feature. Currently, there is no way to
disable this behavior.

--Tyler

On Sun, May 7, 2017 at 2:17 PM Benjamin Gandon <benjamin(a)gandon.org> wrote:

Sounds like a workaround to me.

I'm not aware of any feature in BOSH that would need old packages to stay
around on BOSH instances.

But it could be due to BOSH delegating process restarts to Monit and not
knowing exactly when the old binaries are not needed anymore. Sure Monit
has specs about how restarting things, but it also has its own agenda.

This is only a guess, though. The truth is in the code. Or when Dmitry
speaks. :)


One other observation is that if the downtime implied by '--recreate' is
an issue to you, then you might be operating *Pets* instead of *Cattle*
and my advice is that you should pay attention for any bigger reliability
problem there.

Using '--recreate' is something that should be done harmlessly on a
regular basis. Like with the 'repave' part of the rotate-repave-repair 3R's
of security.



Le 3 mai 2017 à 16:58, Alexander Claus <alexander.claus(a)sap.com> a écrit :

Hey,
first: sorry if this question is already here somewhere in the list.
Unfortunately the search is not working. I always get:
Sorry no email could be found for this query.
No matter, which search term I enter. Am I doing anything wrong or is the
search defect?

My real issue:
When updating an existing deployment with a new release, artifacts from
the former release still reside on the deployment vm after the finished
update.
In our case we have multiple packages which are not marginal in size, each
around 100MB. In sum we have therefore 500-600MB for the packages which
form the release.
When we now update an existing deployment, one can see during the update
via 'bosh deployments' that the deployment has the former and the new
release, and after the 'bosh deploy' command for the update finished the
same view shows only the new release. So far so good.
When going via 'bosh ssh' to the machine, one can see that in
/var/vcap/data/packages/<package-name> multiple folders exist: for the
current version of the package and for the former one.
I checked with multiple successive updates (versions 1>2>3>4, ..), that
this mechanism only holds the last version in parallel, but not more.
Furthermore I found out that the naming of the folder containing the
different versions of the packages is obviously depending on the CPI. On
bosh-lite one can see a relation to the package fingerprints and SHA1 of
compiled packages. On Bosh(a)AWS I cannot recognize any relation to any
fingerprint or SHA1 or blobstore id or whatever.

Here comes my question:
Is it a bug or a feature that the old packages remain on the deployed vm
after a finished update? I could not find anything about this in the bosh
documentation.

Since in our case the disk space used by these duplicate unused packages
is significant: Is there any way to deactivate this "feature", i.e. only to
keep the current version of a package?
I know that I could do the update via 'bosh deploy --recreate' and the
result would be a machine with only the needed package version, but during
that the complete vm is destroyed and recreated which takes significant
amount of time depending on the cloud provider. So I'd favour an option to
update without '--recreate'.

Hope that someone can help.
Alexander