community feedback on removing non-encrypted support from consul-release


Amit Kumar Gupta
 

Hey all!

The BOSH release of consul maintained by the core CF team [1] currently
supports both encrypted and unencrypted modes of operation. "encrypted"
means that all server-to-server and client-to-server is encrypted and
mutually authenticated via TLS, and all gossip traffic is encrypted using
an encryption key. "unencrypted" means none of the above.

We'd like to remove support for the non-encrypted mode of operation. All
production environments should be operating in encrypted mode, and all
production environments we know of do indeed. This should not affect the
developer workflow, as the BOSH-Lite tooling for the primary consumers of
consul-release (namely cf-release and diego-release) have built-in
self-signed certs.

We will continue to provide documentation and tooling to make it easy to
generate the right certs/keys for operating consul-release in encrypted
mode.

Does anyone have concerns about this proposal?

Thanks,
Amit, CF Infrastructure team PM


Zach Robinson <zrobinson@...>
 

I love the idea of secure by default, with simple tooling provided to
enable development workflows. If things work well on this release it's
probably a good idea to start using a similar pattern throughout.

-Zach

On Fri, Feb 5, 2016 at 4:04 PM, Amit Gupta <agupta(a)pivotal.io> wrote:

Hey all!

The BOSH release of consul maintained by the core CF team [1] currently
supports both encrypted and unencrypted modes of operation. "encrypted"
means that all server-to-server and client-to-server is encrypted and
mutually authenticated via TLS, and all gossip traffic is encrypted using
an encryption key. "unencrypted" means none of the above.

We'd like to remove support for the non-encrypted mode of operation. All
production environments should be operating in encrypted mode, and all
production environments we know of do indeed. This should not affect the
developer workflow, as the BOSH-Lite tooling for the primary consumers of
consul-release (namely cf-release and diego-release) have built-in
self-signed certs.

We will continue to provide documentation and tooling to make it easy to
generate the right certs/keys for operating consul-release in encrypted
mode.

Does anyone have concerns about this proposal?

Thanks,
Amit, CF Infrastructure team PM


Amit Kumar Gupta
 

Thanks Zach, I agree it'll be nice to move all the things to
secure-by-default as a general rule.

I realized I left the link out of my original email when I said "The BOSH
release of consul maintained by the core CF team [1]"

[1] https://github.com/cloudfoundry-incubator/consul-release

On Sat, Feb 6, 2016 at 3:19 PM, Zach Robinson <zrobinson(a)pivotal.io> wrote:

I love the idea of secure by default, with simple tooling provided to
enable development workflows. If things work well on this release it's
probably a good idea to start using a similar pattern throughout.

-Zach

On Fri, Feb 5, 2016 at 4:04 PM, Amit Gupta <agupta(a)pivotal.io> wrote:

Hey all!

The BOSH release of consul maintained by the core CF team [1] currently
supports both encrypted and unencrypted modes of operation. "encrypted"
means that all server-to-server and client-to-server is encrypted and
mutually authenticated via TLS, and all gossip traffic is encrypted using
an encryption key. "unencrypted" means none of the above.

We'd like to remove support for the non-encrypted mode of operation. All
production environments should be operating in encrypted mode, and all
production environments we know of do indeed. This should not affect the
developer workflow, as the BOSH-Lite tooling for the primary consumers of
consul-release (namely cf-release and diego-release) have built-in
self-signed certs.

We will continue to provide documentation and tooling to make it easy to
generate the right certs/keys for operating consul-release in encrypted
mode.

Does anyone have concerns about this proposal?

Thanks,
Amit, CF Infrastructure team PM


Aaron Huber
 

The only thought I have is that for some of us that are doing network level
encryption until you guys do have everything secure by default, this would
just enforce double encrypting some things. Not necessarily the end of the
world but it would slow traffic down and increase CPU load.

Our current plan was to leave TLS disabled until everything is secure and
then switch to point-to-point security at that time and turn off IPSec.
That won't be possible until you stop sending passwords in clear text over
the wire, so at least until NATS is gone and possibly a few more things.
This isn't an argument not to make the change but just a data point for you.

Aaron Huber
Intel Corporation



--
View this message in context: http://cf-bosh.70367.x6.nabble.com/cf-bosh-community-feedback-on-removing-non-encrypted-support-from-consul-release-tp1314p1322.html
Sent from the CF BOSH mailing list archive at Nabble.com.


Amit Kumar Gupta
 

Hey Aaron,

That's a good point. We have done some internal stress-testing at scale
with IPsec and with consul happening to be in encrypted mode. The data
generally showed that this had negligible impact on things involving
consul. What are the chances you could validate this in one of your
environments with IPsec and secure consul, to get another data point?

Also, do you have a summary of where sensitive information is being
transmitted over NATS? Is it just traffic involving DEA/HM9k? Are you
running Diego, and do you still see sensitive info despite being on the
Diego backend?

You may also be interested to know that the DEA & HM9k team is working on
moving a lot of traffic from NATS to HTTPS:
https://www.pivotaltracker.com/n/projects/900612

Cheers,
Amit Gupta, Pivotal
CF Infrastructure team PM

On Sat, Feb 6, 2016 at 5:26 PM, aaron_huber <aaron.m.huber(a)intel.com> wrote:

The only thought I have is that for some of us that are doing network level
encryption until you guys do have everything secure by default, this would
just enforce double encrypting some things. Not necessarily the end of the
world but it would slow traffic down and increase CPU load.

Our current plan was to leave TLS disabled until everything is secure and
then switch to point-to-point security at that time and turn off IPSec.
That won't be possible until you stop sending passwords in clear text over
the wire, so at least until NATS is gone and possibly a few more things.
This isn't an argument not to make the change but just a data point for
you.

Aaron Huber
Intel Corporation



--
View this message in context:
http://cf-bosh.70367.x6.nabble.com/cf-bosh-community-feedback-on-removing-non-encrypted-support-from-consul-release-tp1314p1322.html
Sent from the CF BOSH mailing list archive at Nabble.com.


Aaron Huber
 

It's more of a theoretical concern about double-encrypting, I'd expect with
modern CPUs it should be more or less undetectable. Again, I wouldn't hold
off implementing the change because of this, we'd just have to adjust our
plans.

The most obvious example of passwords on the wire in clear text was the
staging upload username/password being sent to the DEAs via NATS. I'm
actually not sure how those credentials flow through to Diego without
digging into the code.

There were a few others we identified the last time we looked. I recall the
VARZ credentials were in NATS also. I didn't make an extensive list at the
time - as soon as we found one we had to implement network level encryption.
:-)

Aaron



--
View this message in context: http://cf-bosh.70367.x6.nabble.com/cf-bosh-community-feedback-on-removing-non-encrypted-support-from-consul-release-tp1314p1331.html
Sent from the CF BOSH mailing list archive at Nabble.com.


Ted Young <tyoung@...>
 

The only "gotcha" I can think of is if it becomes easy to accidentally roll
out with default certs, but it sounds like you're on that.

On Sat, Feb 6, 2016 at 3:22 PM, Amit Gupta <agupta(a)pivotal.io> wrote:

Thanks Zach, I agree it'll be nice to move all the things to
secure-by-default as a general rule.

I realized I left the link out of my original email when I said "The BOSH
release of consul maintained by the core CF team [1]"

[1] https://github.com/cloudfoundry-incubator/consul-release

On Sat, Feb 6, 2016 at 3:19 PM, Zach Robinson <zrobinson(a)pivotal.io>
wrote:

I love the idea of secure by default, with simple tooling provided to
enable development workflows. If things work well on this release it's
probably a good idea to start using a similar pattern throughout.

-Zach

On Fri, Feb 5, 2016 at 4:04 PM, Amit Gupta <agupta(a)pivotal.io> wrote:

Hey all!

The BOSH release of consul maintained by the core CF team [1] currently
supports both encrypted and unencrypted modes of operation. "encrypted"
means that all server-to-server and client-to-server is encrypted and
mutually authenticated via TLS, and all gossip traffic is encrypted using
an encryption key. "unencrypted" means none of the above.

We'd like to remove support for the non-encrypted mode of operation.
All production environments should be operating in encrypted mode, and all
production environments we know of do indeed. This should not affect the
developer workflow, as the BOSH-Lite tooling for the primary consumers of
consul-release (namely cf-release and diego-release) have built-in
self-signed certs.

We will continue to provide documentation and tooling to make it easy to
generate the right certs/keys for operating consul-release in encrypted
mode.

Does anyone have concerns about this proposal?

Thanks,
Amit, CF Infrastructure team PM