1. posterid:472955
  2. Messages

Re: BOSH Director Performance on Jammy Stemcells FAQ

Benjamin Gandon
 

Thank you for the info, Brian.

When you intend the Bosh Agent to remove the Clang compiler on non-compilation VMs, I’m wondering about how consistent this would be with the env.bosh.remove_dev_tools option [1] in instance groups (defaulting to false), that already controls the removal of GCC on runtime VMs.

With the design ou expose, we would end up with GCC being kept on runtime VMs by default, with a possibility to remove it, whereas Clang would be systematically be removed.

In order to have a consistent behavior about compilers removal, what about those two alternate solutions:

1. Add a new env.bosh.remove_clang option that would default to true.

2. Have Clang removal be controlled by the env.bosh.remove_dev_tools option, and make it true by default. After all, There are no Bosh Releases that need the compilers to be kept on runtime VMs.


Second remark, there hasn’t been any public CentOS 7 Stemcell generated for ≈3 years, as the latest v3763.61 tag [2] was set in Dec 2019 [3]. I can’t reasonably think that any organization would actually run software on such old OS.

It seems that the pipeline has been stopped back then, and might not have been maintained since then.
Therefore in the Stemcells page [4], would it be possible to add a Deprecation Warning for CentOS 7 Stemcells?

Best,
Benjamin


Le 28 oct. 2022 à 21:46, Brian Cunnie <brian.cunnie@...> a écrit :

BOSH Director Performance on Jammy Stemcells FAQ

What’s the problem?

Ruby programs using Ruby compiled with GCC (GNU Compiler Collection) on Jammy stemcells have a much larger RSS (Resident Set Size) memory footprint, which can cause memory pressure. This affects Ruby-based programs such as the BOSH Director and the BOSH Azure, AWS, and vSphere CPIs. This can cause BOSH operations such as “bosh deploy” to take much longer and even time out.

What’s the fix?

We plan to compile the Ruby interpreter on Jammy with Clang (an Apple-sponsored GCC-compatible compiler). Ruby interpreters compiled with Clang don’t appear to have the same memory bloat when running the BOSH Director or the Ruby-based CPIs.

How will we accomplish that?

We plan to include the Clang compiler on the Jammy stemcells. We also plan to modify the Ruby BOSH package to use Clang if it’s available, otherwise fall back to GCC.

Doesn’t the Clang compiler take up a lot of disk space?

Yes, the Clang compiler takes up 700-800 MB of disk space; however, we plan to instruct the BOSH agent to remove the Clang compiler on boot unless the VM is a compilation VM. In other words, the Clang compiler won’t take up precious space on the root disk for the typically deployed VM.

What about the Xenial, Bionic, and CentOS stemcells?

We don’t plan to install Clang on Xenial and Bionic stemcells; they don’t exhibit the performance problem, so Clang has little to offer.

We’re not sure whether the CentOS stemcell is affected.

Other than memory, is there a performance impact of using Clang-based Ruby?

In our testing, it appears that a Jammy-based Clang-based Ruby Director with a Clang-based vSphere CPI offers 5-25% performance boost over a Xenial-based GCC-based Ruby Director.

What is the root cause of the problem?

We’re not sure; it appears that the problem is related to the version of GCC used to compile Ruby. We notice that the memory footprint of Ruby’s threads grew from Ubuntu Disco to Ubuntu Eoan: 204 kiB → 10400 kiB.

--
Brian Cunnie, 650.968.6262
Join cf-bosh@lists.cloudfoundry.org to automatically receive all group messages.