Hey cf-dev,

The Release Integration team has had a few reports from other CF engineering teams that their long-running environments have had their internal TLS certificates expire. Since certificates generated by the BOSH CLI get a one-year expiration date, and it's been about a year since early adopters started using cf-deployment, we suspect that some older environments in the CF community are fast approaching this issue as well. We hope to provide enough of a warning that folks in the community can address this.

Check your certificate expiration dates
This is pretty simple to do. You can copy a certificate -- service_cf_internal_ca is a good one to try -- and paste it into the form on this site: https://www.sslshopper.com/certificate-decoder.html. You'll find the expiration date in the "Valid To" section. If your certificates going to expire soon, continue to the process below.

How to rotate certificates
This is not an easy process, but it's doable. I'll warn you right now that, during the transition, your CF will experience `cf push` downtime, but apps should remain available. Also, if you're deploying with the windows-cell.yml or secure-service-credentials.yml ops-files, the process will be a bit more complicated, so please reach out to the RelInt team for help.

1. Deploy with concatenated CA certificates
   
1. Generate new certs by running
2. bosh int cf-deployment.yml [-o ... ] --vars-store new-vars.yml -v system_domain=$SYSTEM_DOMAIN
3. For each new CA cert, concatenate the new CA certificate to both the `ca` and `certificate` field.
4. Deploy
2. Deploy with new leaf certificates
   
1. For each leaf certificate in your vars-store, replace with the corresponding certificate from new-vars.yml. These leaf certificates are signed by the new CA's.
2. Deploy. When the api instances roll, users will no longer be able to push apps, until you remove the old CA certificates.
3. Deploy without the old CA certificates.
   
1. For each CA certificate in your vars-store, remove the first certificate in the `ca` and `certificate` fields. The result should be that only the new CA certificates created in step 1.1 should be included in your vars-store.

The RelInt team has also worked through a process for rotating certificates that have already expired. If you have any questions or concerns, jump into the #release-integration channel in the Cloud Foundry slack and feel free to get a hold of the team there.

Thanks!
CF Release Integration