|
Guillaume Berche
Hi Shannon and the routing team, Testing the route service support in v230, I observe that brokered route services are only receiving traffic for routes mapped to started apps. In other words, if a route is mapped to an app in the "crashed", "starting" or "stopped" state, then any fully-brokered route service bound to that route won't receive traffic sent to that route, instead the gorouter directly responds with a 404. I wonder whether this is a product decision (I had missed this from the design proposal [3]), or rather this is an intermediate implementation choice, and that it could be considered to forward any traffic received from a route to the associated bound route service regardless of the status of mapped apps (I did not yet find a related story in the routing backlog). I collected at the end of this email a list of use-cases where I believe route services will benefit from unconditionally receiving traffic from bound routes. In addition, the "unconditional routing of traffic to route services" would also offer a more consistent behavior between "static route services" and "fully brokered services" to developers interested in consuming route services. App developers can be guaranteed that fully-brokered services that receive traffic even if the app is unavailable (CRASHED, or during a transient Diego cell unavailability...), just like "static route services" would. Lastly, the "unconditional routing of traffic to route services" also seems more consistent with the current CLI UX: the binding of a route service to a route is independent of app mapping to the same route. The "cf bind-route-service" command does not require the route to be bound to an app. The "cf routes" commands does list routes with bound route services and not mapped to any apps, etc... Trying to imagine drawbacks/impacts of such "unconditional routing", I could so far only spot: a- slightly more traffic handled by route services that don't wish to account for/modify the default 404 response on unavailable app b- slightly more traffic for the gorouter for handling requests sent to routes mapped to unavailable apps: the request would now be proxied to route services, which will query back the gorouter c- potentially slightly larger gorouter routing table that need to kept in memory (route entries for route services but no app endpoints). I believe these impacts are acceptable. b) could potentially be reduced by passing the app status to the route service (e.g. via an additional header "X-CF-Route-Status" with value "404: route does not exist" ). Implementation wise, I'm not sure how deep/strong is the current assumption that "an active route is associated to at least one endpoint" in the different CF components (gorouter, its nats messages and routing-api, diego route emiter and BBS models) and therefore the effort required to implement the "unconditional routing to route services" behavior. Thanks in advance for your thoughts on this, Guillaume. Related use-cases: *1- returning custom response when app is unavailable (crashed, starting, stopped, or zero available app instances). * For apps returning HTML, this may be custom HTML response (rather than the default gorouter 404 response page), or specific HTTP response code such as "503 service unavailable" to suggest client some retries. For route services dealing with routes serving APIs (e.g. SOAP), a route service may return a proper SOAP-formatted fault response. Multi-site aware route services may choose to redirect users to a route hosted on a second CF instance through a "307 temporary redirect" status code. A caching service may choose to return (potentially stale) cached content when the mapped app is in the CRASHED state, rather than returning a 404. *2- Applying side effects upon unavailability of app* A SOX-compliant lossless logging service (unlike the potentially lossly loggregator-based logging), may wish to log full details of the requests sent to the route, including those that never reached the an available app instance. A api gateway route services that would maintain measurements of performance and availability of the exposed APIs that transit through its bound routes, would need to receive traffic when bound apps are crashed. The autosleep service [1] that I'm working on would be able to dynamically start a previously stopped app in order to save ram during inactivity. [1] https://docs.google.com/document/d/1tMhIBX3tw7kPEOMCzKhUgmtmr26GVxyXwUTwMO71THI/edit< https://github.com/Orange-OpenSource/autosleep> [2] http://docs.cloudfoundry.org/services/route-services.html#architecture[3] https://docs.google.com/document/d/1bGOQxiKkmaw6uaRWGd-sXpxL0Y28d3QihcluI15FiIA/edit#heading=h.8djffzes9pnb
|
|
|
Hello Guillaume, Thank you for sharing your thoughts on these use cases. I can see how having a route service field requests for an app, whether the app is up on not, could be useful. However, enabling this would significantly change how routes are registered for apps on Cloud Foundry, and how the router handles the route lookup. Routes are not currently enabled in the routing tier unless they are mapped to an app, and only when the app is determined healthy. You are proposing the router maintains routes which have no backends, and instead of a failed lookup determining whether a 404 is returned, the router should figure out whether a route has any backends or a route service. I'll chew on your use case and keep my ear out for additional use cases for maintaining routes with no backends in the routing table. Best, Shannon -- View this message in context: http://cf-dev.70369.x6.nabble.com/cf-dev-Brokered-route-services-only-receiving-traffic-for-routes-mapped-to-started-apps-tp4699p4742.htmlSent from the CF Dev mailing list archive at Nabble.com.
|
|
|
Mike Youngstrom <youngm@...>
Here is another minor use case. My users are often confused that a stopped
app returns a 404 instead of a 503. So, we implement that functionality
for the user using an app mapped to wildcard routes that constantly asks
the CC for valid routes. This works for wildcard domains but not one off
domains.
It might be better if the router returned a 503. At least for routes bound
to apps. Not sure if this should extend to routes not bound to apps.
Mike
toggle quoted messageShow quoted text
On Thu, Apr 28, 2016 at 1:32 PM, Shannon Coen <scoen(a)pivotal.io> wrote: Hello Guillaume,
Thank you for sharing your thoughts on these use cases. I can see how
having
a route service field requests for an app, whether the app is up on not,
could be useful.
However, enabling this would significantly change how routes are registered
for apps on Cloud Foundry, and how the router handles the route lookup.
Routes are not currently enabled in the routing tier unless they are mapped
to an app, and only when the app is determined healthy.
You are proposing the router maintains routes which have no backends, and
instead of a failed lookup determining whether a 404 is returned, the
router
should figure out whether a route has any backends or a route service.
I'll chew on your use case and keep my ear out for additional use cases for
maintaining routes with no backends in the routing table.
Best,
Shannon
--
View this message in context:
http://cf-dev.70369.x6.nabble.com/cf-dev-Brokered-route-services-only-receiving-traffic-for-routes-mapped-to-started-apps-tp4699p4742.html
Sent from the CF Dev mailing list archive at Nabble.com.
|
|
|
Guillaume Berche
Thanks Shannon for your reply and the accurate summary of my proposal. I'll chew on your use case and keep my ear out for additional use cases for maintaining routes with no backends in the routing table. I think some use-cases I detailed previously (reproduced and numbered at bottom of this email for convenience), could potentially be better addressed without route services, and therefore qualify as "additional use cases for maintaining routes with no backends in the routing table": - a way to recover orphan/lost traffic from a given route mapped to crashed app. Currently, the only way to get this traffic routed is to register a wildcard route that gets mapped to a "catch-all" app [3]. However, this catch-all apps collects orphaned traffic from all the domain, making it hard to distinct teams in separate spaces/orgs to apply different processing for different apps. This corresponds to the 1.1 and 1.2 scenarios previously listed, without the additional network hops implied by the use of a route service (in particular when the app is healthy). - a 1st class support for autosleep behavior within the platform (see related thread [1]) as an improvement over the service-broker based implementation we tackled. A possible workaround for our autosleep service broker implementation [2] is to register a wildcard route and map it to our autosleep proxy deployed as an app on CF. The autosleep proxy will then receive all orphan traffic, including traffic from stopped app. As this workaround has severe limitations (detailed below), we're eager to hear whether you feel that "unconditional routing of traffic to route services" is a likely future direction. This will help us decide whether we'd wait for this to happen (our current route-service-based implementation being complete), or rather tackle the wildcard route based workaround. Thanks again, Guillaume. [1] http://cf-dev.70369.x6.nabble.com/cf-dev-App-autosleep-support-tp962p989.html[2] https://github.com/Orange-OpenSource/autosleep[3] http://cf-dev.70369.x6.nabble.com/cf-dev-How-can-we-customized-quot-404-Not-Found-quot-td4501.html*Limitations of a wildcard based autosleep workaround:* - all orphaned traffic is sent to the wildcard route, regardless of whether a stopped app is mapped to the route requiring higher capacity planning for the autosleep service, and tolerance/hardening to DoS. - prevents usage of the wildcard route for other purposes such as a custom 404 pagea - unlike with traffic sent to route services url which may include path part used to pass some context (in addition to the X-CF-Forwarded-Url header containing the original request), the wildcard route only forwards original traffic to the mapped app. Our autosleep app then need to lookup apps bound to the route at runtime, implying higher load. *Related use-cases:* *1- returning custom response when app is unavailable (crashed, starting, stopped, or zero available app instances). * 1.1 For apps returning HTML, this may be custom HTML response (rather than the default gorouter 404 response page), or specific HTTP response code such as "503 service unavailable" to suggest client some retries. For route services dealing with routes serving APIs (e.g. SOAP), a route service may return a proper SOAP-formatted fault response. 1.2 Multi-site aware route services may choose to redirect users to a route hosted on a second CF instance through a "307 temporary redirect" status code. 1.3 A caching service may choose to return (potentially stale) cached content when the mapped app is in the CRASHED state, rather than returning a 404. *2- Applying side effects upon unavailability of app* 2.1 A SOX-compliant lossless logging service (unlike the potentially lossly loggregator-based logging), may wish to log full details of the requests sent to the route, including those that never reached the an available app instance. 2.2 A api gateway route services that would maintain measurements of performance and availability of the exposed APIs that transit through its bound routes, would need to receive traffic when bound apps are crashed. 2.3 The autosleep service [1] that I'm working on would be able to dynamically start a previously stopped app in order to save ram during inactivity. On Thu, Apr 28, 2016 at 9:32 PM, Shannon Coen <scoen(a)pivotal.io> wrote: Hello Guillaume,
Thank you for sharing your thoughts on these use cases. I can see how
having
a route service field requests for an app, whether the app is up on not,
could be useful.
However, enabling this would significantly change how routes are registered
for apps on Cloud Foundry, and how the router handles the route lookup.
Routes are not currently enabled in the routing tier unless they are mapped
to an app, and only when the app is determined healthy.
You are proposing the router maintains routes which have no backends, and
instead of a failed lookup determining whether a 404 is returned, the
router
should figure out whether a route has any backends or a route service.
I'll chew on your use case and keep my ear out for additional use cases for
maintaining routes with no backends in the routing table.
Best,
Shannon
--
View this message in context:
http://cf-dev.70369.x6.nabble.com/cf-dev-Brokered-route-services-only-receiving-traffic-for-routes-mapped-to-started-apps-tp4699p4742.html
Sent from the CF Dev mailing list archive at Nabble.com.
|
|
|
Hi Am 28.04.2016 um 23:08 schrieb Mike Youngstrom: Here is another minor use case. My users are often confused that a
stopped app returns a 404 instead of a 503. So, we implement that
functionality for the user using an app mapped to wildcard routes that
constantly asks the CC for valid routes. This works for wildcard
domains but not one off domains.
It might be better if the router returned a 503. At least for routes
bound to apps. Not sure if this should extend to routes not bound to apps. +1 for that proposal. A 404 also causes issues when crawler remove pages from their index. A 503 has less side effects. I would also prefer a 503 service unavailable when a route is not bound - because there is no service for this route. IMHO the meaning is much closer to what has happended. Stefan Mike
On Thu, Apr 28, 2016 at 1:32 PM, Shannon Coen <scoen(a)pivotal.io
<mailto:scoen(a)pivotal.io>> wrote:
Hello Guillaume,
Thank you for sharing your thoughts on these use cases. I can see
how having
a route service field requests for an app, whether the app is up on not,
could be useful.
However, enabling this would significantly change how routes are
registered
for apps on Cloud Foundry, and how the router handles the route lookup.
Routes are not currently enabled in the routing tier unless they are
mapped
to an app, and only when the app is determined healthy.
You are proposing the router maintains routes which have no
backends, and
instead of a failed lookup determining whether a 404 is returned,
the router
should figure out whether a route has any backends or a route service.
I'll chew on your use case and keep my ear out for additional use
cases for
maintaining routes with no backends in the routing table.
Best,
Shannon
--
View this message in context:
http://cf-dev.70369.x6.nabble.com/cf-dev-Brokered-route-services-only-receiving-traffic-for-routes-mapped-to-started-apps-tp4699p4742.html
Sent from the CF Dev mailing list archive at Nabble.com.
|
|
|
Guillaume Berche
Shannon, What are your current thoughts on "maintaining routes with no backends in the routing table" ? I quickly scanned the routing backlog few days ago without yet finding trace of it. I wish we could have used the opportunity of the cf summit "project office hours" routing session [1] to have interactive exchanges around these use cases. Unfortunately, my autosleep session [2] is scheduled at the exact same timeslot. If the cf foundation organizers were able to swap sessions that would be great. I'll send a separate email to events(a)cloudfoundry.org, is there are other community members suffering from the same conflict. Thanks in advance, Guillaume. [1] http://sched.co/71aq[2] http://sched.co/6aNpGuillaume.
On Sun, May 1, 2016 at 12:03 AM, Stefan Mayr <stefan(a)mayr-stefan.de> wrote: Hi
Am 28.04.2016 um 23:08 schrieb Mike Youngstrom:
Here is another minor use case. My users are often confused that a
stopped app returns a 404 instead of a 503. So, we implement that
functionality for the user using an app mapped to wildcard routes that
constantly asks the CC for valid routes. This works for wildcard
domains but not one off domains.
It might be better if the router returned a 503. At least for routes
bound to apps. Not sure if this should extend to routes not bound to
apps.
+1 for that proposal. A 404 also causes issues when crawler remove pages
from their index. A 503 has less side effects. I would also prefer a 503
service unavailable when a route is not bound - because there is no service
for this route. IMHO the meaning is much closer to what has happended.
Stefan
Mike
On Thu, Apr 28, 2016 at 1:32 PM, Shannon Coen <scoen(a)pivotal.io
<mailto:scoen(a)pivotal.io>> wrote:
Hello Guillaume,
Thank you for sharing your thoughts on these use cases. I can see
how having
a route service field requests for an app, whether the app is up on
not,
could be useful.
However, enabling this would significantly change how routes are
registered
for apps on Cloud Foundry, and how the router handles the route
lookup.
Routes are not currently enabled in the routing tier unless they are
mapped
to an app, and only when the app is determined healthy.
You are proposing the router maintains routes which have no
backends, and
instead of a failed lookup determining whether a 404 is returned,
the router
should figure out whether a route has any backends or a route service.
I'll chew on your use case and keep my ear out for additional use
cases for
maintaining routes with no backends in the routing table.
Best,
Shannon
--
View this message in context:
http://cf-dev.70369.x6.nabble.com/cf-dev-Brokered-route-services-only-receiving-traffic-for-routes-mapped-to-started-apps-tp4699p4742.html
Sent from the CF Dev mailing list archive at Nabble.com.
|
|
|
Hi Guillaume,
It would be great to have a chance to talk more about this at summit.
In summary, I believe supporting your use case is a large effort, and yours
in the only evidence I've heard in support of it. This makes prioritization
a challenge. However, I believe our current plan for architectural changes
to routing will eventually satisfy your requirements as well.
Currently, CC sends route registration to Diego when an app is started.
Routes do not land in the router's routing table until the app is started,
as Diego doesn't know anything about stopped apps (LRPs are deleted when a
user requests stop). Since Diego will have no information about the LRP,
the router-emitter has no way of discovering that a route should be
registered.
Our plan is to move routing info out Diego. I believe it will fulfill your
use case, and the Diego team very much wants this also.
The plan looks like this:
1. Update Routing API endpoints for HTTP route registration to be
consistent with the TCP endpoints we've been focused on
2. Update the Route-Registrar job used by system components, service
brokers, etc. to register HTTP, to point at the Routing API, instead of
NATS.
3. Update the route-emitter to register HTTP routes for apps on Diego
with the Routing API. *At this point, we believe we will have removed
the need for NATS in CF*
4. Update the Routing API to support route reservation, independent of
whether there are backends or not. *At this point, an independent client
could conceivably register a route with a route_service_url, and without
backends*
5. We may need to update Route-Registrar job to support reservation of a
route without backends, and association of backends with the route
6. Update CC to register app routes with Routing API, instead of sending
this data to Diego with createLRP, and update the route-emitter to
significantly change its behavior: instead of calculating the routing table
and sending it to Routing API, it will ask Diego for backends associated
with routes in the Routing API (linked by the process ID, most likely). *At
this point, a developer could conceivably use CLI to create a route, bind
it to a Route Service, and without mapping the route to an app, the router
would forward requests for the route to the Route Service.*
Best,
Shannon Coen
Product Manager, Cloud Foundry
Pivotal, Inc.
toggle quoted messageShow quoted text
On Fri, May 13, 2016 at 1:31 AM, Guillaume Berche <bercheg(a)gmail.com> wrote: Shannon,
What are your current thoughts on "maintaining routes with no backends in
the routing table" ? I quickly scanned the routing backlog few days ago
without yet finding trace of it.
I wish we could have used the opportunity of the cf summit "project office
hours" routing session [1] to have interactive exchanges around these use
cases. Unfortunately, my autosleep session [2] is scheduled at the exact
same timeslot.
If the cf foundation organizers were able to swap sessions that would be
great. I'll send a separate email to events(a)cloudfoundry.org, is there
are other community members suffering from the same conflict.
Thanks in advance,
Guillaume.
[1] http://sched.co/71aq
[2] http://sched.co/6aNp
Guillaume.
On Sun, May 1, 2016 at 12:03 AM, Stefan Mayr <stefan(a)mayr-stefan.de>
wrote:
Hi
Am 28.04.2016 um 23:08 schrieb Mike Youngstrom:
Here is another minor use case. My users are often confused that a
stopped app returns a 404 instead of a 503. So, we implement that
functionality for the user using an app mapped to wildcard routes that
constantly asks the CC for valid routes. This works for wildcard
domains but not one off domains.
It might be better if the router returned a 503. At least for routes
bound to apps. Not sure if this should extend to routes not bound to
apps.
+1 for that proposal. A 404 also causes issues when crawler remove pages
from their index. A 503 has less side effects. I would also prefer a 503
service unavailable when a route is not bound - because there is no service
for this route. IMHO the meaning is much closer to what has happended.
Stefan
Mike
On Thu, Apr 28, 2016 at 1:32 PM, Shannon Coen <scoen(a)pivotal.io
<mailto:scoen(a)pivotal.io>> wrote:
Hello Guillaume,
Thank you for sharing your thoughts on these use cases. I can see
how having
a route service field requests for an app, whether the app is up on
not,
could be useful.
However, enabling this would significantly change how routes are
registered
for apps on Cloud Foundry, and how the router handles the route
lookup.
Routes are not currently enabled in the routing tier unless they are
mapped
to an app, and only when the app is determined healthy.
You are proposing the router maintains routes which have no
backends, and
instead of a failed lookup determining whether a 404 is returned,
the router
should figure out whether a route has any backends or a route
service.
I'll chew on your use case and keep my ear out for additional use
cases for
maintaining routes with no backends in the routing table.
Best,
Shannon
--
View this message in context:
http://cf-dev.70369.x6.nabble.com/cf-dev-Brokered-route-services-only-receiving-traffic-for-routes-mapped-to-started-apps-tp4699p4742.html
Sent from the CF Dev mailing list archive at Nabble.com.
|
|
|
toggle quoted messageShow quoted text
On Fri, May 13, 2016 at 12:10 PM, Shannon Coen <scoen(a)pivotal.io> wrote: Hi Guillaume,
It would be great to have a chance to talk more about this at summit.
In summary, I believe supporting your use case is a large effort, and
yours in the only evidence I've heard in support of it. This makes
prioritization a challenge. However, I believe our current plan for
architectural changes to routing will eventually satisfy your requirements
as well.
Currently, CC sends route registration to Diego when an app is started.
Routes do not land in the router's routing table until the app is started,
as Diego doesn't know anything about stopped apps (LRPs are deleted when a
user requests stop). Since Diego will have no information about the LRP,
the router-emitter has no way of discovering that a route should be
registered.
Our plan is to move routing info out Diego. I believe it will fulfill your
use case, and the Diego team very much wants this also.
The plan looks like this:
1. Update Routing API endpoints for HTTP route registration to be
consistent with the TCP endpoints we've been focused on
2. Update the Route-Registrar job used by system components, service
brokers, etc. to register HTTP, to point at the Routing API, instead of
NATS.
3. Update the route-emitter to register HTTP routes for apps on Diego
with the Routing API. *At this point, we believe we will have removed
the need for NATS in CF*
4. Update the Routing API to support route reservation, independent of
whether there are backends or not. *At this point, an independent
client could conceivably register a route with a route_service_url, and
without backends*
5. We may need to update Route-Registrar job to support reservation of
a route without backends, and association of backends with the route
6. Update CC to register app routes with Routing API, instead of
sending this data to Diego with createLRP, and update the route-emitter to
significantly change its behavior: instead of calculating the routing table
and sending it to Routing API, it will ask Diego for backends associated
with routes in the Routing API (linked by the process ID, most likely). *At
this point, a developer could conceivably use CLI to create a route, bind
it to a Route Service, and without mapping the route to an app, the router
would forward requests for the route to the Route Service.*
Best,
Shannon Coen
Product Manager, Cloud Foundry
Pivotal, Inc.
On Fri, May 13, 2016 at 1:31 AM, Guillaume Berche <bercheg(a)gmail.com>
wrote:
Shannon,
What are your current thoughts on "maintaining routes with no backends in
the routing table" ? I quickly scanned the routing backlog few days ago
without yet finding trace of it.
I wish we could have used the opportunity of the cf summit "project
office hours" routing session [1] to have interactive exchanges around
these use cases. Unfortunately, my autosleep session [2] is scheduled at
the exact same timeslot.
If the cf foundation organizers were able to swap sessions that would be
great. I'll send a separate email to events(a)cloudfoundry.org, is there
are other community members suffering from the same conflict.
Thanks in advance,
Guillaume.
[1] http://sched.co/71aq
[2] http://sched.co/6aNp
Guillaume.
On Sun, May 1, 2016 at 12:03 AM, Stefan Mayr <stefan(a)mayr-stefan.de>
wrote:
Hi
Am 28.04.2016 um 23:08 schrieb Mike Youngstrom:
Here is another minor use case. My users are often confused that a
stopped app returns a 404 instead of a 503. So, we implement that
functionality for the user using an app mapped to wildcard routes that
constantly asks the CC for valid routes. This works for wildcard
domains but not one off domains.
It might be better if the router returned a 503. At least for routes
bound to apps. Not sure if this should extend to routes not bound to
apps.
+1 for that proposal. A 404 also causes issues when crawler remove pages
from their index. A 503 has less side effects. I would also prefer a 503
service unavailable when a route is not bound - because there is no service
for this route. IMHO the meaning is much closer to what has happended.
Stefan
Mike
On Thu, Apr 28, 2016 at 1:32 PM, Shannon Coen <scoen(a)pivotal.io
<mailto:scoen(a)pivotal.io>> wrote:
Hello Guillaume,
Thank you for sharing your thoughts on these use cases. I can see
how having
a route service field requests for an app, whether the app is up on
not,
could be useful.
However, enabling this would significantly change how routes are
registered
for apps on Cloud Foundry, and how the router handles the route
lookup.
Routes are not currently enabled in the routing tier unless they are
mapped
to an app, and only when the app is determined healthy.
You are proposing the router maintains routes which have no
backends, and
instead of a failed lookup determining whether a 404 is returned,
the router
should figure out whether a route has any backends or a route
service.
I'll chew on your use case and keep my ear out for additional use
cases for
maintaining routes with no backends in the routing table.
Best,
Shannon
--
View this message in context:
http://cf-dev.70369.x6.nabble.com/cf-dev-Brokered-route-services-only-receiving-traffic-for-routes-mapped-to-started-apps-tp4699p4742.html
Sent from the CF Dev mailing list archive at Nabble.com.
|
|
|
Guillaume Berche
Thanks a lot Shannon for your detailed response and sharing the routing architecture plans. I realize the priorization of such effort remains a challenge. Out of curiosity, in step #6, how would CC be notified of LRP current state change, as to perform route unregister/updates ? Would CC be registering to BBS external event client [1] through server-side-events, or rather diego notifying CC through HTTP callbacks ? I wonder whether this architecture could also enable fully-brokered route services implementations to fetch the routing table from the routing-api, and perform direct routing to apps (an alternative discussed in [2]), enabling more advanced features (such as custom load balancing). I understand this currently would require granting routing.routes.read scope to route services. Granting them a routing.route.<bound_route_guid>.read oauth scope at SB route binding time would remove such a need for "admin creds". Thanks again, [1] https://godoc.org/github.com/cloudfoundry-incubator/bbs#ExternalEventClient[2] https://docs.google.com/document/d/1bGOQxiKkmaw6uaRWGd-sXpxL0Y28d3QihcluI15FiIA/edit#section "Other proposals that we considered" Guillaume.
On Fri, May 13, 2016 at 9:10 PM, Shannon Coen <scoen(a)pivotal.io> wrote: Hi Guillaume,
It would be great to have a chance to talk more about this at summit.
In summary, I believe supporting your use case is a large effort, and
yours in the only evidence I've heard in support of it. This makes
prioritization a challenge. However, I believe our current plan for
architectural changes to routing will eventually satisfy your requirements
as well.
Currently, CC sends route registration to Diego when an app is started.
Routes do not land in the router's routing table until the app is started,
as Diego doesn't know anything about stopped apps (LRPs are deleted when a
user requests stop). Since Diego will have no information about the LRP,
the router-emitter has no way of discovering that a route should be
registered.
Our plan is to move routing info out Diego. I believe it will fulfill your
use case, and the Diego team very much wants this also.
The plan looks like this:
1. Update Routing API endpoints for HTTP route registration to be
consistent with the TCP endpoints we've been focused on
2. Update the Route-Registrar job used by system components, service
brokers, etc. to register HTTP, to point at the Routing API, instead of
NATS.
3. Update the route-emitter to register HTTP routes for apps on Diego
with the Routing API. *At this point, we believe we will have removed
the need for NATS in CF*
4. Update the Routing API to support route reservation, independent of
whether there are backends or not. *At this point, an independent
client could conceivably register a route with a route_service_url, and
without backends*
5. We may need to update Route-Registrar job to support reservation of
a route without backends, and association of backends with the route
6. Update CC to register app routes with Routing API, instead of
sending this data to Diego with createLRP, and update the route-emitter to
significantly change its behavior: instead of calculating the routing table
and sending it to Routing API, it will ask Diego for backends associated
with routes in the Routing API (linked by the process ID, most likely). *At
this point, a developer could conceivably use CLI to create a route, bind
it to a Route Service, and without mapping the route to an app, the router
would forward requests for the route to the Route Service.*
Best,
Shannon Coen
Product Manager, Cloud Foundry
Pivotal, Inc.
On Fri, May 13, 2016 at 1:31 AM, Guillaume Berche <bercheg(a)gmail.com>
wrote:
Shannon,
What are your current thoughts on "maintaining routes with no backends in
the routing table" ? I quickly scanned the routing backlog few days ago
without yet finding trace of it.
I wish we could have used the opportunity of the cf summit "project
office hours" routing session [1] to have interactive exchanges around
these use cases. Unfortunately, my autosleep session [2] is scheduled at
the exact same timeslot.
If the cf foundation organizers were able to swap sessions that would be
great. I'll send a separate email to events(a)cloudfoundry.org, is there
are other community members suffering from the same conflict.
Thanks in advance,
Guillaume.
[1] http://sched.co/71aq
[2] http://sched.co/6aNp
Guillaume.
On Sun, May 1, 2016 at 12:03 AM, Stefan Mayr <stefan(a)mayr-stefan.de>
wrote:
Hi
Am 28.04.2016 um 23:08 schrieb Mike Youngstrom:
Here is another minor use case. My users are often confused that a
stopped app returns a 404 instead of a 503. So, we implement that
functionality for the user using an app mapped to wildcard routes that
constantly asks the CC for valid routes. This works for wildcard
domains but not one off domains.
It might be better if the router returned a 503. At least for routes
bound to apps. Not sure if this should extend to routes not bound to
apps.
+1 for that proposal. A 404 also causes issues when crawler remove pages
from their index. A 503 has less side effects. I would also prefer a 503
service unavailable when a route is not bound - because there is no service
for this route. IMHO the meaning is much closer to what has happended.
Stefan
Mike
On Thu, Apr 28, 2016 at 1:32 PM, Shannon Coen <scoen(a)pivotal.io
<mailto:scoen(a)pivotal.io>> wrote:
Hello Guillaume,
Thank you for sharing your thoughts on these use cases. I can see
how having
a route service field requests for an app, whether the app is up on
not,
could be useful.
However, enabling this would significantly change how routes are
registered
for apps on Cloud Foundry, and how the router handles the route
lookup.
Routes are not currently enabled in the routing tier unless they are
mapped
to an app, and only when the app is determined healthy.
You are proposing the router maintains routes which have no
backends, and
instead of a failed lookup determining whether a 404 is returned,
the router
should figure out whether a route has any backends or a route
service.
I'll chew on your use case and keep my ear out for additional use
cases for
maintaining routes with no backends in the routing table.
Best,
Shannon
--
View this message in context:
http://cf-dev.70369.x6.nabble.com/cf-dev-Brokered-route-services-only-receiving-traffic-for-routes-mapped-to-started-apps-tp4699p4742.html
Sent from the CF Dev mailing list archive at Nabble.com.
|
|
|
Inline Shannon Coen Product Manager, Cloud Foundry Pivotal, Inc. On Mon, May 16, 2016 at 3:29 PM, Guillaume Berche <bercheg(a)gmail.com> wrote: Thanks a lot Shannon for your detailed response and sharing the routing
architecture plans. I realize the priorization of such effort remains a
challenge.
Out of curiosity, in step #6, how would CC be notified of LRP current
state change, as to perform route unregister/updates ? Would CC be
registering to BBS external event client [1] through server-side-events, or
rather diego notifying CC through HTTP callbacks ?
CC would not be involved with route registration. CC would send the route and the app process ID (received from Diego) to the Routing API. The route-emitter would adds/remove backends for this process ID based on Diego server-sent-events, and periodic batch fetches (as it does now). I wonder whether this architecture could also enable fully-brokered route
services implementations to fetch the routing table from the routing-api,
and perform direct routing to apps (an alternative discussed in [2]),
enabling more advanced features (such as custom load balancing). I
understand this currently would require granting routing.routes.read
scope to route services. Granting them a routing.route.<bound_route_guid>.read
oauth scope at SB route binding time would remove such a need for "admin
creds".
Yes, we imagine the Routing API providing the route/backend mapping as a service, which will enable bring-your-own router, as well as direct routing from Route Services. This assumes your route service has access to the same private network as the interfaces for the Cell VMs. We may to consider how to partition this data so that your route service or router only receives routing data it should know about. Thanks again,
[1]
https://godoc.org/github.com/cloudfoundry-incubator/bbs#ExternalEventClient
[2]
https://docs.google.com/document/d/1bGOQxiKkmaw6uaRWGd-sXpxL0Y28d3QihcluI15FiIA/edit#
section "Other proposals that we considered"
Guillaume.
On Fri, May 13, 2016 at 9:10 PM, Shannon Coen <scoen(a)pivotal.io> wrote:
Hi Guillaume,
It would be great to have a chance to talk more about this at summit.
In summary, I believe supporting your use case is a large effort, and
yours in the only evidence I've heard in support of it. This makes
prioritization a challenge. However, I believe our current plan for
architectural changes to routing will eventually satisfy your requirements
as well.
Currently, CC sends route registration to Diego when an app is started.
Routes do not land in the router's routing table until the app is started,
as Diego doesn't know anything about stopped apps (LRPs are deleted when a
user requests stop). Since Diego will have no information about the LRP,
the router-emitter has no way of discovering that a route should be
registered.
Our plan is to move routing info out Diego. I believe it will fulfill
your use case, and the Diego team very much wants this also.
The plan looks like this:
1. Update Routing API endpoints for HTTP route registration to be
consistent with the TCP endpoints we've been focused on
2. Update the Route-Registrar job used by system components, service
brokers, etc. to register HTTP, to point at the Routing API, instead of
NATS.
3. Update the route-emitter to register HTTP routes for apps on Diego
with the Routing API. *At this point, we believe we will have
removed the need for NATS in CF*
4. Update the Routing API to support route reservation, independent
of whether there are backends or not. *At this point, an independent
client could conceivably register a route with a route_service_url, and
without backends*
5. We may need to update Route-Registrar job to support reservation
of a route without backends, and association of backends with the route
6. Update CC to register app routes with Routing API, instead of
sending this data to Diego with createLRP, and update the route-emitter to
significantly change its behavior: instead of calculating the routing table
and sending it to Routing API, it will ask Diego for backends associated
with routes in the Routing API (linked by the process ID, most likely). *At
this point, a developer could conceivably use CLI to create a route, bind
it to a Route Service, and without mapping the route to an app, the router
would forward requests for the route to the Route Service.*
Best,
Shannon Coen
Product Manager, Cloud Foundry
Pivotal, Inc.
On Fri, May 13, 2016 at 1:31 AM, Guillaume Berche <bercheg(a)gmail.com>
wrote:
Shannon,
What are your current thoughts on "maintaining routes with no backends
in the routing table" ? I quickly scanned the routing backlog few days ago
without yet finding trace of it.
I wish we could have used the opportunity of the cf summit "project
office hours" routing session [1] to have interactive exchanges around
these use cases. Unfortunately, my autosleep session [2] is scheduled at
the exact same timeslot.
If the cf foundation organizers were able to swap sessions that would be
great. I'll send a separate email to events(a)cloudfoundry.org, is there
are other community members suffering from the same conflict.
Thanks in advance,
Guillaume.
[1] http://sched.co/71aq
[2] http://sched.co/6aNp
Guillaume.
On Sun, May 1, 2016 at 12:03 AM, Stefan Mayr <stefan(a)mayr-stefan.de>
wrote:
Hi
Am 28.04.2016 um 23:08 schrieb Mike Youngstrom:
Here is another minor use case. My users are often confused that a
stopped app returns a 404 instead of a 503. So, we implement that
functionality for the user using an app mapped to wildcard routes that
constantly asks the CC for valid routes. This works for wildcard
domains but not one off domains.
It might be better if the router returned a 503. At least for routes
bound to apps. Not sure if this should extend to routes not bound to
apps.
+1 for that proposal. A 404 also causes issues when crawler remove
pages from their index. A 503 has less side effects. I would also prefer a
503 service unavailable when a route is not bound - because there is no
service for this route. IMHO the meaning is much closer to what has
happended.
Stefan
Mike
On Thu, Apr 28, 2016 at 1:32 PM, Shannon Coen <scoen(a)pivotal.io
<mailto:scoen(a)pivotal.io>> wrote:
Hello Guillaume,
Thank you for sharing your thoughts on these use cases. I can see
how having
a route service field requests for an app, whether the app is up
on not,
could be useful.
However, enabling this would significantly change how routes are
registered
for apps on Cloud Foundry, and how the router handles the route
lookup.
Routes are not currently enabled in the routing tier unless they
are
mapped
to an app, and only when the app is determined healthy.
You are proposing the router maintains routes which have no
backends, and
instead of a failed lookup determining whether a 404 is returned,
the router
should figure out whether a route has any backends or a route
service.
I'll chew on your use case and keep my ear out for additional use
cases for
maintaining routes with no backends in the routing table.
Best,
Shannon
--
View this message in context:
http://cf-dev.70369.x6.nabble.com/cf-dev-Brokered-route-services-only-receiving-traffic-for-routes-mapped-to-started-apps-tp4699p4742.html
Sent from the CF Dev mailing list archive at Nabble.com.
|
|
|
