Hi Cloud Foundry,

We've had a recurring issue brought to our attention regarding a Cloud
Foundry deployment using a system_domain that is in the list of
app_domains. When the system domain is in the list of app domains, a Shared
Domain is created for the system domain. This is problematic because it
allows users to create routes on the system domain, see this [1] recent
issue as an example.

[1] https://github.com/cloudfoundry/cloud_controller_ng/issues/568

I'd like to propose two solutions and get some feedback regarding which the
community would prefer. Please respond with your preferred solution and a
brief reason why.

*Solution 1 - Require a Unique System Domain*

Instead of recommending a unique system domain, we would enforce this in
the Cloud Controller. The proposed change is as follows:

1. REQUIRE system_domain to NOT be in the list of app_domains
2. REQUIRE a system_domain_organization

This will create a Private Domain for the system domain. Failure to
configure correctly would not allow the Cloud Controller to start.

If we decide to implement this, an operator should ensure their deployment
uses a unique system domain and a system_domain_organization and correct
DNS entries before upgrading.

Example for BOSH-lite

- app_domains: [bosh-lite.com]
- system_domain: system.bosh-lite.com
- system_domain_organization: system


- api endpoint: api.system.bosh-lite.com
- sample app endpoint: dora.bosh-lite.com

Example for a PaaS:

- app_domains: [yuge-paas-apps.io]
- system_domain: yuge-pass.com
- system_domain_organization: yuge-system-org


- api endpoint: api.yuge-paas.com
- sample app endpoint: dora.yuge-paas-apps.io

*Pro: *Cloud Controller now enforces what was previously the recommended
configuration for separate system and apps domains.
Con: Second SSL cert for the system domain and possibly a second DNS record
if system domain is not covered by the current wildcard record.

*Solution 2 - Cloud Controller Seeds System Routes*

To prevent a non-system app from requesting a hostname on a shared system
domain, the Cloud Controller will take a list of hostnames and seed the
database with routes. As routes are associated with a space, we will
require a system_organization and system_space. An operator could choose to
omit hostnames as desired.

cc.system_hostnames:
description: List of hostnames for which routes will be created on the
system domain.
default: [api,uaa,login,doppler,loggregator,hm9000]
cc.system_space:
description: Space where system routes will be created.
default: system

*Pro:* Significantly less change for operators running Cloud Foundry with
matching system and apps domains.
*Con: *Cloud Controller has knowledge of unrelated system components and
the list of defaults needs to be maintained as we add and remove components.


Thanks,

-Nick
--
Nicholas Calugar
CAPI Product Manager
Pivotal Software, Inc.

For option 2 would it not be simpler to have a single property such as
cc.blacklisted_system_domain_routes that contains the desired list and have
the CC deny route requests for those routes?

I don't see what storing them in the DB or creating real routes actually
buys us here. Everything would be available via config.

I'm in favour of some form of option 2 for those of us who have existing
deployments but would rather not make a change in either app or system
domains but are exposed to this issue.

John.

Hi John,

The point of seeding is so that there aren't two decisions (code paths) for
requests to create a route. Seeding the database requires no changes to
route creation logic.

Would anyone else like to comment on either of the two proposals before we
make a decision and start implementing a fix?


Thanks,

Nick

On Thu, Mar 31, 2016 at 10:52 AM john mcteague <john.mcteague(a)gmail.com>
wrote:

For option 2 would it not be simpler to have a single property such as
cc.blacklisted_system_domain_routes that contains the desired list and have
the CC deny route requests for those routes?

I don't see what storing them in the DB or creating real routes actually
buys us here. Everything would be available via config.

I'm in favour of some form of option 2 for those of us who have existing
deployments but would rather not make a change in either app or system
domains but are exposed to this issue.

John.
On 31 Mar 2016 5:47 a.m., "Nicholas Calugar" <ncalugar(a)pivotal.io> wrote:

Hi Cloud Foundry,

We've had a recurring issue brought to our attention regarding a Cloud
Foundry deployment using a system_domain that is in the list of
app_domains. When the system domain is in the list of app domains, a Shared
Domain is created for the system domain. This is problematic because it
allows users to create routes on the system domain, see this [1] recent
issue as an example.

[1] https://github.com/cloudfoundry/cloud_controller_ng/issues/568

I'd like to propose two solutions and get some feedback regarding which
the community would prefer. Please respond with your preferred solution and
a brief reason why.

*Solution 1 - Require a Unique System Domain*

Instead of recommending a unique system domain, we would enforce this in
the Cloud Controller. The proposed change is as follows:

1. REQUIRE system_domain to NOT be in the list of app_domains
2. REQUIRE a system_domain_organization

This will create a Private Domain for the system domain. Failure to
configure correctly would not allow the Cloud Controller to start.

If we decide to implement this, an operator should ensure their
deployment uses a unique system domain and a system_domain_organization and
correct DNS entries before upgrading.

Example for BOSH-lite

- app_domains: [bosh-lite.com]
- system_domain: system.bosh-lite.com
- system_domain_organization: system


- api endpoint: api.system.bosh-lite.com
- sample app endpoint: dora.bosh-lite.com

Example for a PaaS:

- app_domains: [yuge-paas-apps.io]
- system_domain: yuge-pass.com
- system_domain_organization: yuge-system-org


- api endpoint: api.yuge-paas.com
- sample app endpoint: dora.yuge-paas-apps.io

*Pro: *Cloud Controller now enforces what was previously the recommended
configuration for separate system and apps domains.
Con: Second SSL cert for the system domain and possibly a second DNS
record if system domain is not covered by the current wildcard record.

*Solution 2 - Cloud Controller Seeds System Routes*

To prevent a non-system app from requesting a hostname on a shared system
domain, the Cloud Controller will take a list of hostnames and seed the
database with routes. As routes are associated with a space, we will
require a system_organization and system_space. An operator could choose to
omit hostnames as desired.

cc.system_hostnames:
description: List of hostnames for which routes will be created on the
system domain.
default: [api,uaa,login,doppler,loggregator,hm9000]
cc.system_space:
description: Space where system routes will be created.
default: system

*Pro:* Significantly less change for operators running Cloud Foundry
with matching system and apps domains.
*Con: *Cloud Controller has knowledge of unrelated system components and
the list of defaults needs to be maintained as we add and remove components.


Thanks,

-Nick
--
Nicholas Calugar
CAPI Product Manager
Pivotal Software, Inc.

--

Nicholas Calugar
CAPI Product Manager
Pivotal Software, Inc.

Would solution 2 allow/work with a system domain?
Solution #2 means api.domain is now taken so there could be no other app
deployed to api.domain. Application developers may not be happy with the
limitation, requiring another domain. Defeats the purpose of solution #2?

Tom

On Tue, Apr 5, 2016 at 2:15 PM, Nicholas Calugar <ncalugar(a)pivotal.io>
wrote:

Hi John,

The point of seeding is so that there aren't two decisions (code paths)
for requests to create a route. Seeding the database requires no changes to
route creation logic.

Would anyone else like to comment on either of the two proposals before we
make a decision and start implementing a fix?


Thanks,

Nick

On Thu, Mar 31, 2016 at 10:52 AM john mcteague <john.mcteague(a)gmail.com>
wrote:

For option 2 would it not be simpler to have a single property such as
cc.blacklisted_system_domain_routes that contains the desired list and have
the CC deny route requests for those routes?

I don't see what storing them in the DB or creating real routes actually
buys us here. Everything would be available via config.

I'm in favour of some form of option 2 for those of us who have existing
deployments but would rather not make a change in either app or system
domains but are exposed to this issue.

John.
On 31 Mar 2016 5:47 a.m., "Nicholas Calugar" <ncalugar(a)pivotal.io> wrote:

Hi Cloud Foundry,

We've had a recurring issue brought to our attention regarding a Cloud
Foundry deployment using a system_domain that is in the list of
app_domains. When the system domain is in the list of app domains, a Shared
Domain is created for the system domain. This is problematic because it
allows users to create routes on the system domain, see this [1] recent
issue as an example.

[1] https://github.com/cloudfoundry/cloud_controller_ng/issues/568

I'd like to propose two solutions and get some feedback regarding which
the community would prefer. Please respond with your preferred solution and
a brief reason why.

*Solution 1 - Require a Unique System Domain*

Instead of recommending a unique system domain, we would enforce this in
the Cloud Controller. The proposed change is as follows:

1. REQUIRE system_domain to NOT be in the list of app_domains
2. REQUIRE a system_domain_organization

This will create a Private Domain for the system domain. Failure to
configure correctly would not allow the Cloud Controller to start.

If we decide to implement this, an operator should ensure their
deployment uses a unique system domain and a system_domain_organization and
correct DNS entries before upgrading.

Example for BOSH-lite

- app_domains: [bosh-lite.com]
- system_domain: system.bosh-lite.com
- system_domain_organization: system


- api endpoint: api.system.bosh-lite.com
- sample app endpoint: dora.bosh-lite.com

Example for a PaaS:

- app_domains: [yuge-paas-apps.io]
- system_domain: yuge-pass.com
- system_domain_organization: yuge-system-org


- api endpoint: api.yuge-paas.com
- sample app endpoint: dora.yuge-paas-apps.io

*Pro: *Cloud Controller now enforces what was previously the
recommended configuration for separate system and apps domains.
Con: Second SSL cert for the system domain and possibly a second DNS
record if system domain is not covered by the current wildcard record.

*Solution 2 - Cloud Controller Seeds System Routes*

To prevent a non-system app from requesting a hostname on a shared
system domain, the Cloud Controller will take a list of hostnames and seed
the database with routes. As routes are associated with a space, we will
require a system_organization and system_space. An operator could choose to
omit hostnames as desired.

cc.system_hostnames:
description: List of hostnames for which routes will be created on the
system domain.
default: [api,uaa,login,doppler,loggregator,hm9000]
cc.system_space:
description: Space where system routes will be created.
default: system

*Pro:* Significantly less change for operators running Cloud Foundry
with matching system and apps domains.
*Con: *Cloud Controller has knowledge of unrelated system components
and the list of defaults needs to be maintained as we add and remove
components.


Thanks,

-Nick
--
Nicholas Calugar
CAPI Product Manager
Pivotal Software, Inc.

--

Nicholas Calugar
CAPI Product Manager
Pivotal Software, Inc.

Hi Tom,

App developers can currently push an app using api.domain, but that is the
problem we are trying to solve. As this is currently allowed, the Gorouter
will balance traffic between the Cloud Controller and the user app. This is
not desired, all traffic for api.system_domain needs to go to the Cloud
Controller.

At this point we are going to move forward with Solution 2. We think this
will offer the easiest transition for operators.


Thanks,

Nick

On Wed, Apr 6, 2016 at 5:27 PM Tom Sherrod <tom.sherrod(a)gmail.com> wrote:

Would solution 2 allow/work with a system domain?
Solution #2 means api.domain is now taken so there could be no other app
deployed to api.domain. Application developers may not be happy with the
limitation, requiring another domain. Defeats the purpose of solution #2?

Tom

On Tue, Apr 5, 2016 at 2:15 PM, Nicholas Calugar <ncalugar(a)pivotal.io>
wrote:

Hi John,

The point of seeding is so that there aren't two decisions (code paths)
for requests to create a route. Seeding the database requires no changes to
route creation logic.

Would anyone else like to comment on either of the two proposals before
we make a decision and start implementing a fix?


Thanks,

Nick

On Thu, Mar 31, 2016 at 10:52 AM john mcteague <john.mcteague(a)gmail.com>
wrote:

For option 2 would it not be simpler to have a single property such as
cc.blacklisted_system_domain_routes that contains the desired list and have
the CC deny route requests for those routes?

I don't see what storing them in the DB or creating real routes actually
buys us here. Everything would be available via config.

I'm in favour of some form of option 2 for those of us who have existing
deployments but would rather not make a change in either app or system
domains but are exposed to this issue.

John.
On 31 Mar 2016 5:47 a.m., "Nicholas Calugar" <ncalugar(a)pivotal.io>
wrote:

Hi Cloud Foundry,

We've had a recurring issue brought to our attention regarding a Cloud
Foundry deployment using a system_domain that is in the list of
app_domains. When the system domain is in the list of app domains, a Shared
Domain is created for the system domain. This is problematic because it
allows users to create routes on the system domain, see this [1] recent
issue as an example.

[1] https://github.com/cloudfoundry/cloud_controller_ng/issues/568

I'd like to propose two solutions and get some feedback regarding which
the community would prefer. Please respond with your preferred solution and
a brief reason why.

*Solution 1 - Require a Unique System Domain*

Instead of recommending a unique system domain, we would enforce this
in the Cloud Controller. The proposed change is as follows:

1. REQUIRE system_domain to NOT be in the list of app_domains
2. REQUIRE a system_domain_organization

This will create a Private Domain for the system domain. Failure to
configure correctly would not allow the Cloud Controller to start.

If we decide to implement this, an operator should ensure their
deployment uses a unique system domain and a system_domain_organization and
correct DNS entries before upgrading.

Example for BOSH-lite

- app_domains: [bosh-lite.com]
- system_domain: system.bosh-lite.com
- system_domain_organization: system


- api endpoint: api.system.bosh-lite.com
- sample app endpoint: dora.bosh-lite.com

Example for a PaaS:

- app_domains: [yuge-paas-apps.io]
- system_domain: yuge-pass.com
- system_domain_organization: yuge-system-org


- api endpoint: api.yuge-paas.com
- sample app endpoint: dora.yuge-paas-apps.io

*Pro: *Cloud Controller now enforces what was previously the
recommended configuration for separate system and apps domains.
Con: Second SSL cert for the system domain and possibly a second DNS
record if system domain is not covered by the current wildcard record.

*Solution 2 - Cloud Controller Seeds System Routes*

To prevent a non-system app from requesting a hostname on a shared
system domain, the Cloud Controller will take a list of hostnames and seed
the database with routes. As routes are associated with a space, we will
require a system_organization and system_space. An operator could choose to
omit hostnames as desired.

cc.system_hostnames:
description: List of hostnames for which routes will be created on
the system domain.
default: [api,uaa,login,doppler,loggregator,hm9000]
cc.system_space:
description: Space where system routes will be created.
default: system

*Pro:* Significantly less change for operators running Cloud Foundry
with matching system and apps domains.
*Con: *Cloud Controller has knowledge of unrelated system components
and the list of defaults needs to be maintained as we add and remove
components.


Thanks,

-Nick
--
Nicholas Calugar
CAPI Product Manager
Pivotal Software, Inc.

--

Nicholas Calugar
CAPI Product Manager
Pivotal Software, Inc.

--

Nicholas Calugar
CAPI Product Manager
Pivotal Software, Inc.