SSL termination for private domains


Carlo Alberto Ferraris
 

I have a question about the SSL termination epic[1], whose goal IIUC is to provide the ability for operators to have multiple TLS certificates: it seems only shared domains are being considered (because the stories talk about *operators* setting up multiple certs); are there no plans for private domains? Put otherwise: are there plans for allowing *users* to provide the cert for a domain they registered in their org?

[1] https://www.pivotaltracker.com/epic/show/2135866

(I originally posted the question on slack but got no reply, so crossposting here)


Shannon Coen
 

Some time ago I sketched out an epic to add support for multiple certs to
gorouter, configured via BOSH manifest property, but these stories have
languished in the icebox while we've addressed more urgent work.

I would like to hear from the community whether an operator managed feature
would be of value, as it would be relatively cheap.

I have also heard requests for user self-service management of certs for
private domains, as Carlo described. This would be a much more complex
feature to deliver, but I can certainly see the value.

Tell me about the pain of managing TLS certificates. How are you dealing
with this today? Which of these approaches would be more helpful in
enabling your developers? Which of these features would you be more
disappointed to hear would not be delivered?

Thank you!

Shannon Coen
Product Manager, Cloud Foundry
Pivotal, Inc.

On Mon, Sep 19, 2016 at 6:11 PM, Carlo Alberto Ferraris <
carlo.ferraris(a)rakuten.com> wrote:

I have a question about the SSL termination epic[1], whose goal IIUC is to
provide the ability for operators to have multiple TLS certificates: it
seems only shared domains are being considered (because the stories talk
about *operators* setting up multiple certs); are there no plans for
private domains? Put otherwise: are there plans for allowing *users* to
provide the cert for a domain they registered in their org?

[1] https://www.pivotaltracker.com/epic/show/2135866

(I originally posted the question on slack but got no reply, so
crossposting here)


Mike Youngstrom <youngm@...>
 

For us we handle all ssl termination in our FLB (Frontend Load Balancer).
If a customer adds a custom domain then my team needs to add a vip and
associated cert for that domain. This is something I don't think CF could
do for us because we are using our FLB. So, FWIW this isn't a feature we
would use since we use or FLB to manage this instead.

Mike

On Mon, Sep 19, 2016 at 11:25 PM, Shannon Coen <scoen(a)pivotal.io> wrote:

Some time ago I sketched out an epic to add support for multiple certs to
gorouter, configured via BOSH manifest property, but these stories have
languished in the icebox while we've addressed more urgent work.

I would like to hear from the community whether an operator managed
feature would be of value, as it would be relatively cheap.

I have also heard requests for user self-service management of certs for
private domains, as Carlo described. This would be a much more complex
feature to deliver, but I can certainly see the value.

Tell me about the pain of managing TLS certificates. How are you dealing
with this today? Which of these approaches would be more helpful in
enabling your developers? Which of these features would you be more
disappointed to hear would not be delivered?

Thank you!

Shannon Coen
Product Manager, Cloud Foundry
Pivotal, Inc.

On Mon, Sep 19, 2016 at 6:11 PM, Carlo Alberto Ferraris <
carlo.ferraris(a)rakuten.com> wrote:

I have a question about the SSL termination epic[1], whose goal IIUC is
to provide the ability for operators to have multiple TLS certificates: it
seems only shared domains are being considered (because the stories talk
about *operators* setting up multiple certs); are there no plans for
private domains? Put otherwise: are there plans for allowing *users* to
provide the cert for a domain they registered in their org?

[1] https://www.pivotaltracker.com/epic/show/2135866

(I originally posted the question on slack but got no reply, so
crossposting here)


Shannon Coen
 

Mike,

What if the way the gorouters were configured with user-provided certs was
a point of extension that could also be used to configure your FLB?

How often do you have to manage certs on your LB? Is this of low value?

Shannon Coen
Product Manager, Cloud Foundry
Pivotal, Inc.

On Tue, Sep 20, 2016 at 12:43 PM, Mike Youngstrom <youngm(a)gmail.com> wrote:

For us we handle all ssl termination in our FLB (Frontend Load Balancer).
If a customer adds a custom domain then my team needs to add a vip and
associated cert for that domain. This is something I don't think CF could
do for us because we are using our FLB. So, FWIW this isn't a feature we
would use since we use or FLB to manage this instead.

Mike

On Mon, Sep 19, 2016 at 11:25 PM, Shannon Coen <scoen(a)pivotal.io> wrote:

Some time ago I sketched out an epic to add support for multiple certs to
gorouter, configured via BOSH manifest property, but these stories have
languished in the icebox while we've addressed more urgent work.

I would like to hear from the community whether an operator managed
feature would be of value, as it would be relatively cheap.

I have also heard requests for user self-service management of certs for
private domains, as Carlo described. This would be a much more complex
feature to deliver, but I can certainly see the value.

Tell me about the pain of managing TLS certificates. How are you dealing
with this today? Which of these approaches would be more helpful in
enabling your developers? Which of these features would you be more
disappointed to hear would not be delivered?

Thank you!

Shannon Coen
Product Manager, Cloud Foundry
Pivotal, Inc.

On Mon, Sep 19, 2016 at 6:11 PM, Carlo Alberto Ferraris <
carlo.ferraris(a)rakuten.com> wrote:

I have a question about the SSL termination epic[1], whose goal IIUC is
to provide the ability for operators to have multiple TLS certificates: it
seems only shared domains are being considered (because the stories talk
about *operators* setting up multiple certs); are there no plans for
private domains? Put otherwise: are there plans for allowing *users* to
provide the cert for a domain they registered in their org?

[1] https://www.pivotaltracker.com/epic/show/2135866

(I originally posted the question on slack but got no reply, so
crossposting here)


Mike Youngstrom <youngm@...>
 

An extension point would be more useful than something that only worked on
the gorouters.

Another thing that mitigates our need for this feature is that most all of
our organization's applications (CF deployed or not) use one of 2 main
wildcard domains. Use of domains outside these 2 are rare.

We built a custom iteration with our DNS solution and CF that looks for new
CF routes using one of those 2 domains and automatically add a dns entry
(if not already taken) pointing to shared VIPs on our FLB that have a
matching wildcard cert already configured. That allows us to add those 2
domains as CF shared domains that anyone can create routes for. Even
though the domains are not dedicated to CF.

I suppose that would be another reason why this isn't currently a major
pain point for my users.

Mike

On Tue, Sep 20, 2016 at 3:54 PM, Shannon Coen <scoen(a)pivotal.io> wrote:

Mike,

What if the way the gorouters were configured with user-provided certs was
a point of extension that could also be used to configure your FLB?

How often do you have to manage certs on your LB? Is this of low value?

Shannon Coen
Product Manager, Cloud Foundry
Pivotal, Inc.

On Tue, Sep 20, 2016 at 12:43 PM, Mike Youngstrom <youngm(a)gmail.com>
wrote:

For us we handle all ssl termination in our FLB (Frontend Load
Balancer). If a customer adds a custom domain then my team needs to add a
vip and associated cert for that domain. This is something I don't think
CF could do for us because we are using our FLB. So, FWIW this isn't a
feature we would use since we use or FLB to manage this instead.

Mike

On Mon, Sep 19, 2016 at 11:25 PM, Shannon Coen <scoen(a)pivotal.io> wrote:

Some time ago I sketched out an epic to add support for multiple certs
to gorouter, configured via BOSH manifest property, but these stories have
languished in the icebox while we've addressed more urgent work.

I would like to hear from the community whether an operator managed
feature would be of value, as it would be relatively cheap.

I have also heard requests for user self-service management of certs for
private domains, as Carlo described. This would be a much more complex
feature to deliver, but I can certainly see the value.

Tell me about the pain of managing TLS certificates. How are you dealing
with this today? Which of these approaches would be more helpful in
enabling your developers? Which of these features would you be more
disappointed to hear would not be delivered?

Thank you!

Shannon Coen
Product Manager, Cloud Foundry
Pivotal, Inc.

On Mon, Sep 19, 2016 at 6:11 PM, Carlo Alberto Ferraris <
carlo.ferraris(a)rakuten.com> wrote:

I have a question about the SSL termination epic[1], whose goal IIUC is
to provide the ability for operators to have multiple TLS certificates: it
seems only shared domains are being considered (because the stories talk
about *operators* setting up multiple certs); are there no plans for
private domains? Put otherwise: are there plans for allowing *users* to
provide the cert for a domain they registered in their org?

[1] https://www.pivotaltracker.com/epic/show/2135866

(I originally posted the question on slack but got no reply, so
crossposting here)


Carlo Alberto Ferraris
 

Mike,
thanks for keeping the ball rolling!
For the TLS termination part we are currently using a setup very similar to the one described by Mike. We sit behind a bunch of SLBs that handle termination for us. The main difference is that we're moving out of the "one VIP per cert" model Mike describes to "one SNI VIP for all certs" - a choice we made exactly to keep options open when it comes to automating this process.
The biggest pain comes from the fact that the SLB in our organization is handled by a different team and that therefore every cert add/update/delete operation requires a manual operation spanning three teams (application team, our team, SLB team); in the worst cases such operations can take days. We may be different in this from other CF operators, but this situation happens fairly frequently.
To put it simply, if CF (gorouter or a different component) had a way to dynamically apply certificates specified by the users (and operators) we would gladly switch away from our current setup.
We were also considering (idea stage, nothing really planned yet) using either nginx or a custom-built TLS terminator for this very purpose (the main reason we're considering something custom built is because it's somewhat hard to get session ticket key rotation right with nginx when you have multiple servers) - but if something functionally equivalent were to appear upstream we would definitely prefer it.

I hope everything makes sense, if not I'll gladly answer any question you may have.

Thanks for looking into this!

Carlo


Shannon Coen
 

Carlo, Mike, others,

Do you store certs in the LB config itself, or federate/offload TLS
termination to some secure store? I'm thinking about storing user-provided
certs in the Routing API and offering them to routers/LBs from there. Would
we instead have to send the certs to some other proprietary system from
where the router/LB would have to pull from?

I've heard a few requests for integrating with systems that store the certs
so that the routers don't have access to the keys.

Shannon Coen
Product Manager, Cloud Foundry
Pivotal, Inc.

On Tue, Sep 20, 2016 at 5:44 PM, Carlo Alberto Ferraris <
carlo.ferraris(a)rakuten.com> wrote:

Mike,
thanks for keeping the ball rolling!
For the TLS termination part we are currently using a setup very similar
to the one described by Mike. We sit behind a bunch of SLBs that handle
termination for us. The main difference is that we're moving out of the
"one VIP per cert" model Mike describes to "one SNI VIP for all certs" - a
choice we made exactly to keep options open when it comes to automating
this process.
The biggest pain comes from the fact that the SLB in our organization is
handled by a different team and that therefore every cert add/update/delete
operation requires a manual operation spanning three teams (application
team, our team, SLB team); in the worst cases such operations can take
days. We may be different in this from other CF operators, but this
situation happens fairly frequently.
To put it simply, if CF (gorouter or a different component) had a way to
dynamically apply certificates specified by the users (and operators) we
would gladly switch away from our current setup.
We were also considering (idea stage, nothing really planned yet) using
either nginx or a custom-built TLS terminator for this very purpose (the
main reason we're considering something custom built is because it's
somewhat hard to get session ticket key rotation right with nginx when you
have multiple servers) - but if something functionally equivalent were to
appear upstream we would definitely prefer it.

I hope everything makes sense, if not I'll gladly answer any question you
may have.

Thanks for looking into this!

Carlo


Carlo Alberto Ferraris
 

We don't have the requirement to use an external secure store, but for that custom terminator component we were thinking to use an external secure store (Vault or something along those lines) to make sure that the the private keys and session ticket keys never hit persistent storage.


Carlo Alberto Ferraris
 

While we're talking about TLS, but this is only partially related, it would be awesome if we were to implement (or some hooks were provided to be able to complete) either the http or tls ACME challenges. That would be the ultimate dream. :D


Anthony
 

+1 to user provided certs for private domains. Today, we use multiple vips with diff certs or our main vip with a cert that has multiple SANs.

Our goal is for cf operators to really be out of the way for app specific concerns and we view the certs and private domains (vanity dns names) as an application concern.

Having this ability provides immense value to us.

Regards,
Anthony

On Sep 20, 2016, at 8:03 PM, Shannon Coen <scoen(a)pivotal.io> wrote:

Carlo, Mike, others,

Do you store certs in the LB config itself, or federate/offload TLS termination to some secure store? I'm thinking about storing user-provided certs in the Routing API and offering them to routers/LBs from there. Would we instead have to send the certs to some other proprietary system from where the router/LB would have to pull from?

I've heard a few requests for integrating with systems that store the certs so that the routers don't have access to the keys.

Shannon Coen
Product Manager, Cloud Foundry
Pivotal, Inc.

On Tue, Sep 20, 2016 at 5:44 PM, Carlo Alberto Ferraris <carlo.ferraris(a)rakuten.com> wrote:
Mike,
thanks for keeping the ball rolling!
For the TLS termination part we are currently using a setup very similar to the one described by Mike. We sit behind a bunch of SLBs that handle termination for us. The main difference is that we're moving out of the "one VIP per cert" model Mike describes to "one SNI VIP for all certs" - a choice we made exactly to keep options open when it comes to automating this process.
The biggest pain comes from the fact that the SLB in our organization is handled by a different team and that therefore every cert add/update/delete operation requires a manual operation spanning three teams (application team, our team, SLB team); in the worst cases such operations can take days. We may be different in this from other CF operators, but this situation happens fairly frequently.
To put it simply, if CF (gorouter or a different component) had a way to dynamically apply certificates specified by the users (and operators) we would gladly switch away from our current setup.
We were also considering (idea stage, nothing really planned yet) using either nginx or a custom-built TLS terminator for this very purpose (the main reason we're considering something custom built is because it's somewhat hard to get session ticket key rotation right with nginx when you have multiple servers) - but if something functionally equivalent were to appear upstream we would definitely prefer it.

I hope everything makes sense, if not I'll gladly answer any question you may have.

Thanks for looking into this!

Carlo


James Leavers
 

It sounds like we are in a similar situation to Carlo, i.e.


- We have an external pair of LBs
- These are used for SSL termination
- We upload SSL certificates to the LBs for various domains, which point
to the same VIP


If something became available that would easily allow app developers /
users to upload their own certificates, I too would happily move SSL
termination from the LBs to gorouter, as it would mean one less automation
workflow for us :-)

On 21 September 2016 at 02:04:48, Shannon Coen (scoen(a)pivotal.io) wrote:

Carlo, Mike, others,

Do you store certs in the LB config itself, or federate/offload TLS
termination to some secure store? I'm thinking about storing user-provided
certs in the Routing API and offering them to routers/LBs from there. Would
we instead have to send the certs to some other proprietary system from
where the router/LB would have to pull from?

I've heard a few requests for integrating with systems that store the certs
so that the routers don't have access to the keys.

Shannon Coen
Product Manager, Cloud Foundry
Pivotal, Inc.

On Tue, Sep 20, 2016 at 5:44 PM, Carlo Alberto Ferraris <
carlo.ferraris(a)rakuten.com> wrote:

Mike,
thanks for keeping the ball rolling!
For the TLS termination part we are currently using a setup very similar
to the one described by Mike. We sit behind a bunch of SLBs that handle
termination for us. The main difference is that we're moving out of the
"one VIP per cert" model Mike describes to "one SNI VIP for all certs" - a
choice we made exactly to keep options open when it comes to automating
this process.
The biggest pain comes from the fact that the SLB in our organization is
handled by a different team and that therefore every cert add/update/delete
operation requires a manual operation spanning three teams (application
team, our team, SLB team); in the worst cases such operations can take
days. We may be different in this from other CF operators, but this
situation happens fairly frequently.
To put it simply, if CF (gorouter or a different component) had a way to
dynamically apply certificates specified by the users (and operators) we
would gladly switch away from our current setup.
We were also considering (idea stage, nothing really planned yet) using
either nginx or a custom-built TLS terminator for this very purpose (the
main reason we're considering something custom built is because it's
somewhat hard to get session ticket key rotation right with nginx when you
have multiple servers) - but if something functionally equivalent were to
appear upstream we would definitely prefer it.

I hope everything makes sense, if not I'll gladly answer any question you
may have.

Thanks for looking into this!

Carlo


Shannon Coen
 

On Tue, Sep 20, 2016 at 10:00 PM, Carlo Alberto Ferraris <
carlo.ferraris(a)rakuten.com> wrote:

While we're talking about TLS, but this is only partially related, it
would be awesome if we were to implement (or some hooks were provided to be
able to complete) either the http or tls ACME challenges. That would be the
ultimate dream. :D
Wasn't familiar with ACME until I just googled it. Do you mean some
mechanism for automated generation of certs?


Shannon Coen
 

Thank you all for your responses.

A follow up question: for the gorouter to host certs for multiple domains,
it seems only natural that it would do this via SNI. Is client support for
SNI ubiquitous among apps running on your CF deployments? Would it be
reasonable to require client SNI support for TLS termination at gorouter?

Thanks again.

Shannon Coen
Product Manager, Cloud Foundry
Pivotal, Inc.

On Wed, Sep 21, 2016 at 5:52 AM, James Leavers <james(a)cloudhelix.io> wrote:

It sounds like we are in a similar situation to Carlo, i.e.


- We have an external pair of LBs
- These are used for SSL termination
- We upload SSL certificates to the LBs for various domains, which
point to the same VIP


If something became available that would easily allow app developers /
users to upload their own certificates, I too would happily move SSL
termination from the LBs to gorouter, as it would mean one less automation
workflow for us :-)


On 21 September 2016 at 02:04:48, Shannon Coen (scoen(a)pivotal.io) wrote:

Carlo, Mike, others,

Do you store certs in the LB config itself, or federate/offload TLS
termination to some secure store? I'm thinking about storing user-provided
certs in the Routing API and offering them to routers/LBs from there. Would
we instead have to send the certs to some other proprietary system from
where the router/LB would have to pull from?

I've heard a few requests for integrating with systems that store the
certs so that the routers don't have access to the keys.

Shannon Coen
Product Manager, Cloud Foundry
Pivotal, Inc.

On Tue, Sep 20, 2016 at 5:44 PM, Carlo Alberto Ferraris <
carlo.ferraris(a)rakuten.com> wrote:

Mike,
thanks for keeping the ball rolling!
For the TLS termination part we are currently using a setup very similar
to the one described by Mike. We sit behind a bunch of SLBs that handle
termination for us. The main difference is that we're moving out of the
"one VIP per cert" model Mike describes to "one SNI VIP for all certs" - a
choice we made exactly to keep options open when it comes to automating
this process.
The biggest pain comes from the fact that the SLB in our organization is
handled by a different team and that therefore every cert add/update/delete
operation requires a manual operation spanning three teams (application
team, our team, SLB team); in the worst cases such operations can take
days. We may be different in this from other CF operators, but this
situation happens fairly frequently.
To put it simply, if CF (gorouter or a different component) had a way to
dynamically apply certificates specified by the users (and operators) we
would gladly switch away from our current setup.
We were also considering (idea stage, nothing really planned yet) using
either nginx or a custom-built TLS terminator for this very purpose (the
main reason we're considering something custom built is because it's
somewhat hard to get session ticket key rotation right with nginx when you
have multiple servers) - but if something functionally equivalent were to
appear upstream we would definitely prefer it.

I hope everything makes sense, if not I'll gladly answer any question you
may have.

Thanks for looking into this!

Carlo


Carlo Alberto Ferraris
 

Our current policy to our users is SNI by default, i.e. unless they explicitly require non-SNI TLS termination they get SNI termination. We went with this because browser support seems good[1] and because there are "easy" (albeit manual) workarounds (request a non-SNI VIP, use a CDN).

Granted, even with SNI TLS termination in gorouter we should still be able to perform TLS termination somewhere else for the few cases in which no SNI is really a requirement - but for this it's enough to not change the HTTP listener behavior.

[1]: http://caniuse.com/#feat=sni


Carlo Alberto Ferraris
 

Yes, it's the protocol[1] proposed by ISRG letsencrypt[2] (under the linux foundation umbrella) that allows automated generation and PKI signing of TLS certificates.
For the record, there's a go implementation of the acme protocol that may end up in the standard library sooner or later[3].

[1]: https://tools.ietf.org/html/draft-ietf-acme-acme-03
[2]: https://letsencrypt.org/
[3]: https://godoc.org/golang.org/x/crypto/acme


James Leavers
 

Ha, just for fun we detailed how you could, technically, request a
letsencrypt cert via a CF app :-) [1]

I would agree that a user would like the ability to auto-renew certs, if
they are currently doing this via cron on Apache or another webserver [2]

[1]
https://blog.cloudhelix.io/authorising-lets-encrypt-certificate-requests-using-a-cloud-foundry-app-71b2d0920ac9
[2]
https://community.letsencrypt.org/t/how-to-automatically-renew-certificates/4393

On 21 September 2016 at 21:47:20, Shannon Coen (scoen(a)pivotal.io) wrote:

On Tue, Sep 20, 2016 at 10:00 PM, Carlo Alberto Ferraris <
carlo.ferraris(a)rakuten.com> wrote:

While we're talking about TLS, but this is only partially related, it
would be awesome if we were to implement (or some hooks were provided to be
able to complete) either the http or tls ACME challenges. That would be the
ultimate dream. :D
Wasn't familiar with ACME until I just googled it. Do you mean some
mechanism for automated generation of certs?