Hello cf-dev,

I work on a team at Pivotal that builds lots of service brokers. We are currently working on broker that backs onto BOSH and want to move towards making it stateless. We have written a proposal to enhance the CF Service Broker API to allow us to achieve this. We believe this will help other service broker authors as the principal transfers to most asynchronous backing systems.

You can read the full proposal here:

https://docs.google.com/document/d/1QzrG3d9-RgB7v5W44jnwgDuQWgqqPosASyCunLwfYF0/edit?usp=sharing

Open to comments on the document and on this thread.

Alex

Sorry for the name misunderstanding... We use the word "marketplace", but not the term in CF.

The background is, firstly it is a public cloud, people use CF and the services provided by the CF; We built a framework (not like service broker) to help people contribute different service, then sell the services in it to the developers using Cloud Foundry. (The services can be used in CF, by the service broker we build; but it also in K8S and other PaaS. But mainly it is used in CF currently...)

So the problem is, when a new contribution, such as MySQL based on Brooklyn added to the marketplace, we need to register it to the CF.

I am not sure if it is clear enough, if you interested in it, I can share some introduction videos.

Hi,
Marketplace is a CF things for CF so what is exaclty the problem you try to
solve ?

marketplace is CF ....

Actually, we are creating the a service marketplace, and meet the same problem, too: when we add a new service from the service marketplace, it need to be registered in the CF side.

But with the service broker, we can manage the services just like managing the instances. You can set a usage restrictions on the service, make it only be consumed by a given team/org (defined in CF).

How to reconcile these use cases:

Given one shared domain mycorp.com
And a wildcard route from that domain, *.mycorp.com
And an app mapped to the wildcard route that returns a 503
I expect that a request to any subdomain of mycorp.com, at any path, to
receive a 503.

Given smoke test that maps a route to an app
Then deletes the app or unmaps the route
Then make a request to the route
and expects a 404 in response
I expect smoke tests to pass.

Smoke tests need a domain that does not have a wildcard route mapped to this
503 app.
- The smoke tests could create a shared domain and delete it. The drawback
is users may see a domain come and go while tests are run.
- In addition to shared domains you may have wildcard routes for, keep a
spare shared domain that does not have this wildcard route. Call it a
sandbox. Use it for smoke tests.




--
View this message in context: http://cf-dev.70369.x6.nabble.com/cf-dev-How-can-we-customized-404-Not-Found-tp4501p4913.html
Sent from the CF Dev mailing list archive at Nabble.com.

Inline

Shannon Coen
Product Manager, Cloud Foundry
Pivotal, Inc.

On Mon, May 16, 2016 at 3:29 PM, Guillaume Berche <bercheg(a)gmail.com> wrote:

Thanks a lot Shannon for your detailed response and sharing the routing
architecture plans. I realize the priorization of such effort remains a
challenge.

Out of curiosity, in step #6, how would CC be notified of LRP current
state change, as to perform route unregister/updates ? Would CC be
registering to BBS external event client [1] through server-side-events, or
rather diego notifying CC through HTTP callbacks ?

CC would not be involved with route registration. CC would send the route
and the app process ID (received from Diego) to the Routing API. The
route-emitter would adds/remove backends for this process ID based on Diego
server-sent-events, and periodic batch fetches (as it does now).

I wonder whether this architecture could also enable fully-brokered route
services implementations to fetch the routing table from the routing-api,
and perform direct routing to apps (an alternative discussed in [2]),
enabling more advanced features (such as custom load balancing). I
understand this currently would require granting routing.routes.read
scope to route services. Granting them a routing.route.<bound_route_guid>.read
oauth scope at SB route binding time would remove such a need for "admin
creds".

Yes, we imagine the Routing API providing the route/backend mapping as a
service, which will enable bring-your-own router, as well as direct routing
from Route Services. This assumes your route service has access to the same
private network as the interfaces for the Cell VMs. We may to consider how
to partition this data so that your route service or router only receives
routing data it should know about.

Thanks again,

[1]
https://godoc.org/github.com/cloudfoundry-incubator/bbs#ExternalEventClient
[2]
https://docs.google.com/document/d/1bGOQxiKkmaw6uaRWGd-sXpxL0Y28d3QihcluI15FiIA/edit#
section "Other proposals that we considered"

Guillaume.


On Fri, May 13, 2016 at 9:10 PM, Shannon Coen <scoen(a)pivotal.io> wrote:

Hi Guillaume,

It would be great to have a chance to talk more about this at summit.

In summary, I believe supporting your use case is a large effort, and
yours in the only evidence I've heard in support of it. This makes
prioritization a challenge. However, I believe our current plan for
architectural changes to routing will eventually satisfy your requirements
as well.

Currently, CC sends route registration to Diego when an app is started.
Routes do not land in the router's routing table until the app is started,
as Diego doesn't know anything about stopped apps (LRPs are deleted when a
user requests stop). Since Diego will have no information about the LRP,
the router-emitter has no way of discovering that a route should be
registered.

Our plan is to move routing info out Diego. I believe it will fulfill
your use case, and the Diego team very much wants this also.

The plan looks like this:

1. Update Routing API endpoints for HTTP route registration to be
consistent with the TCP endpoints we've been focused on
2. Update the Route-Registrar job used by system components, service
brokers, etc. to register HTTP, to point at the Routing API, instead of
NATS.
3. Update the route-emitter to register HTTP routes for apps on Diego
with the Routing API. *At this point, we believe we will have
removed the need for NATS in CF*
4. Update the Routing API to support route reservation, independent
of whether there are backends or not. *At this point, an independent
client could conceivably register a route with a route_service_url, and
without backends*
5. We may need to update Route-Registrar job to support reservation
of a route without backends, and association of backends with the route
6. Update CC to register app routes with Routing API, instead of
sending this data to Diego with createLRP, and update the route-emitter to
significantly change its behavior: instead of calculating the routing table
and sending it to Routing API, it will ask Diego for backends associated
with routes in the Routing API (linked by the process ID, most likely). *At
this point, a developer could conceivably use CLI to create a route, bind
it to a Route Service, and without mapping the route to an app, the router
would forward requests for the route to the Route Service.*


Best,

Shannon Coen
Product Manager, Cloud Foundry
Pivotal, Inc.

On Fri, May 13, 2016 at 1:31 AM, Guillaume Berche <bercheg(a)gmail.com>
wrote:

Shannon,

What are your current thoughts on "maintaining routes with no backends
in the routing table" ? I quickly scanned the routing backlog few days ago
without yet finding trace of it.

I wish we could have used the opportunity of the cf summit "project
office hours" routing session [1] to have interactive exchanges around
these use cases. Unfortunately, my autosleep session [2] is scheduled at
the exact same timeslot.
If the cf foundation organizers were able to swap sessions that would be
great. I'll send a separate email to events(a)cloudfoundry.org, is there
are other community members suffering from the same conflict.


Thanks in advance,

Guillaume.

[1] http://sched.co/71aq
[2] http://sched.co/6aNp


Guillaume.

On Sun, May 1, 2016 at 12:03 AM, Stefan Mayr <stefan(a)mayr-stefan.de>
wrote:

Hi

Am 28.04.2016 um 23:08 schrieb Mike Youngstrom:

Here is another minor use case. My users are often confused that a
stopped app returns a 404 instead of a 503. So, we implement that
functionality for the user using an app mapped to wildcard routes that
constantly asks the CC for valid routes. This works for wildcard
domains but not one off domains.

It might be better if the router returned a 503. At least for routes
bound to apps. Not sure if this should extend to routes not bound to
apps.

+1 for that proposal. A 404 also causes issues when crawler remove
pages from their index. A 503 has less side effects. I would also prefer a
503 service unavailable when a route is not bound - because there is no
service for this route. IMHO the meaning is much closer to what has
happended.

Stefan

Mike

On Thu, Apr 28, 2016 at 1:32 PM, Shannon Coen <scoen(a)pivotal.io
<mailto:scoen(a)pivotal.io>> wrote:

Hello Guillaume,

Thank you for sharing your thoughts on these use cases. I can see
how having
a route service field requests for an app, whether the app is up
on not,
could be useful.

However, enabling this would significantly change how routes are
registered
for apps on Cloud Foundry, and how the router handles the route
lookup.
Routes are not currently enabled in the routing tier unless they
are
mapped
to an app, and only when the app is determined healthy.

You are proposing the router maintains routes which have no
backends, and
instead of a failed lookup determining whether a 404 is returned,
the router
should figure out whether a route has any backends or a route
service.

I'll chew on your use case and keep my ear out for additional use
cases for
maintaining routes with no backends in the routing table.

Best,
Shannon



--
View this message in context:

http://cf-dev.70369.x6.nabble.com/cf-dev-Brokered-route-services-only-receiving-traffic-for-routes-mapped-to-started-apps-tp4699p4742.html
Sent from the CF Dev mailing list archive at Nabble.com.

Hi Amulya,

Using cups for service binding has no down side because that is how
services are consumed if we ignore service broker.

Whole point of using service broker with cf is to consume service api's in
best possible way , managing 1 or 2 services with cups sounds good but when
you have large number of services then service broker plays important role
specially in service provisioning automation .

We have different teams exposing their api's and it is very easy for us to
consume those services and provide them to users on platform, increases
value of platform itself on business side .

It also helps our users to get information about their provisioned services
and manage them without maintaining any other information documents.

Thanks
Ronak Banka

On Tue, May 17, 2016 at 6:59 AM, Amulya Sharma <amulya.sharma(a)gmail.com>
wrote:

Thanks Dan,

but Services have nothing to do with cloud foundry .. the only job broker
is doing here is publishing themselves in marketplace .. wondering why it
need to be this way why not totally independent from cloud foundry..



On Mon, May 16, 2016 at 12:40 PM Dan Wendorf <dwendorf(a)pivotal.io> wrote:

Hi Amulya,

Using user-provided-services can be a good way to manage your service
instances if it is challenging to automate provisioning, but it sounds like
one of your main hurdles is needing to coordinate with an CF admin to
update a broker.

There's a relatively new option to create a private service broker that
is scoped to a particular space. Dr. Nic has a good blog post
<https://blog.starkandwayne.com/2015/11/18/register-your-own-service-broker-with-any-cloud-foundry/> explaining
this. I think the advantages of using the CF services workflow are nice,
especially not needing to learn an external system and having
dynamically-generated and secure credentials.


Cheers,
Dan

On Mon, May 16, 2016 at 11:20 AM, Amulya Sharma <amulya.sharma(a)gmail.com>
wrote:

Question on not using cf Service Broker or only use Private Broker,

We have Service Broker in CF its kind of hassel to maintain service
brokers as it need admin privileges to register also unique names and all
upgrade path is complex.

What if we do not use service brokers at all instead we just create a
portal to create service instance and at the end of creation just show VCAP
variables of service and a CUPS Command to create that service instance in
CF CLI

my question is

Q What are the downside of this approach of not using service broker at
all?

pro side is decoupling of service offering from CF ?

Hey all,

*TL;DR: we have deprecated the runtime-ci Docker image, and will soon
delete it from Docker Hub.*

Thanks to Concourse, it's a cinch to build, maintain, and use Docker images
in your CI pipelines. And it's easy to have clean, slim images with the
minimum requirements for your builds. The release integration team no
longer uses the runtime-ci Dockerfile that lives in cf-release. Some fun
facts about that image:

- it has multiple versions of ruby in it
- it has a .bashrc, so many builds using it would source ~/.bashrc
- it was building spiff from source (master of spiff is unrecognizable
compared to the last official release of spiff, though still magically
backwards compatible)

We'd like to delete the image from Docker Hub. Please let us know if there
are any objections.

Thanks
CF Release Integration Team
Pivotal Software

Thanks a lot Shannon for your detailed response and sharing the routing
architecture plans. I realize the priorization of such effort remains a
challenge.

Out of curiosity, in step #6, how would CC be notified of LRP current state
change, as to perform route unregister/updates ? Would CC be registering to
BBS external event client [1] through server-side-events, or rather diego
notifying CC through HTTP callbacks ?

I wonder whether this architecture could also enable fully-brokered route
services implementations to fetch the routing table from the routing-api,
and perform direct routing to apps (an alternative discussed in [2]),
enabling more advanced features (such as custom load balancing). I
understand this currently would require granting routing.routes.read scope
to route services. Granting them a routing.route.<bound_route_guid>.read
oauth scope at SB route binding time would remove such a need for "admin
creds".

Thanks again,

[1]
https://godoc.org/github.com/cloudfoundry-incubator/bbs#ExternalEventClient
[2]
https://docs.google.com/document/d/1bGOQxiKkmaw6uaRWGd-sXpxL0Y28d3QihcluI15FiIA/edit#
section "Other proposals that we considered"

Guillaume.

Thanks Dan,

but Services have nothing to do with cloud foundry .. the only job broker
is doing here is publishing themselves in marketplace .. wondering why it
need to be this way why not totally independent from cloud foundry..

Hi Amulya,

Using user-provided-services can be a good way to manage your service
instances if it is challenging to automate provisioning, but it sounds like
one of your main hurdles is needing to coordinate with an CF admin to
update a broker.

There's a relatively new option to create a private service broker that is
scoped to a particular space. Dr. Nic has a good blog post
<https://blog.starkandwayne.com/2015/11/18/register-your-own-service-broker-with-any-cloud-foundry/>
explaining
this. I think the advantages of using the CF services workflow are nice,
especially not needing to learn an external system and having
dynamically-generated and secure credentials.


Cheers,
Dan

On Mon, May 16, 2016 at 11:20 AM, Amulya Sharma <amulya.sharma(a)gmail.com>
wrote:

Question on not using cf Service Broker or only use Private Broker,

We have Service Broker in CF its kind of hassel to maintain service
brokers as it need admin privileges to register also unique names and all
upgrade path is complex.

What if we do not use service brokers at all instead we just create a
portal to create service instance and at the end of creation just show VCAP
variables of service and a CUPS Command to create that service instance in
CF CLI

my question is

Q What are the downside of this approach of not using service broker at
all?

pro side is decoupling of service offering from CF ?

It was late Friday afternoon. I just deleted the VM and rebuilt it over the
weekend, but I'll retry that next time I see it.

Thanks,
Eric

Question on not using cf Service Broker or only use Private Broker,

We have Service Broker in CF its kind of hassel to maintain service brokers
as it need admin privileges to register also unique names and all upgrade
path is complex.

What if we do not use service brokers at all instead we just create a
portal to create service instance and at the end of creation just show VCAP
variables of service and a CUPS Command to create that service instance in
CF CLI

my question is

Q What are the downside of this approach of not using service broker at
all?

pro side is decoupling of service offering from CF ?

Hi Nick,

Thanks for the update!
Are there any plans of bringing it to V2 APIs as well?

Kind Regards,
Padma

Hi Nick, Eric,

I have confirmed that with the current CC release we could change the default behaviour on the CLI side without making additional API calls:
In the body of POST /v2/apps?async=true we simply add "health_check_type":"none" if the user specified “--no-route” and that seems to be accepted regardless whether the app defaults to DEA or Diego.
Can you confirm CC does not return an error on older CC versions (before the health_check_type attribute was introduced) if we add this attribute to the API request?

Also, can you confirm you definitely don’t want CC to change its default behaviour to align with what the user (and CLI) sees compared with Diego?

The current API sequence the CLI uses is:

1. POST /v2/apps?async=true – create app

2. POST /v2/routes?async=true&inline-relations-depth=1 – create route(s)

3. PUT /v2/apps/guid/routes/guid – map route(s)

4. PUT /v2/resource_match & PUT /v2/apps/guid/bits?async=true – upload app

5. GET /v2/jobs/guid – check its status

Although at step 1 CC would not know whether the app will get any routes (i.e. whether the user specified “--no-route”), at step 4 CC would, so it could toggle the health-check-type accordingly. Not sure how CC does this for apps deployed to DEA – is it the DEA backend that makes this decision at the timing of 4?

Regards,
Dies Koper
Cloud Foundry CLI PM


From: Nicholas Calugar [mailto:ncalugar(a)pivotal.io]
Sent: Thursday, May 12, 2016 4:34 AM
To: Discussions about Cloud Foundry projects and the system overall.
Subject: [cf-dev] Re: Re: Re: aligning cf push health-check default value

Hi Dies,

I spoke with Eric and he explained that this could be the desired UX for a majority of the apps pushed with --no-route. There are more advanced deployment strategies that might set --no-route and bind to routes later, but I think we can expect these users to be explicit with their health check as well. I think my discomfort with this arose when you mentioned to me that we might want to do this in the Cloud Controller. As long as this continues to be explicit from the API perspective, I'm fine with changing the UX of the CLI per your above proposal.


Thanks,

Nick

On Wed, May 4, 2016 at 1:13 PM, Shannon Coen <scoen(a)pivotal.io<mailto:scoen(a)pivotal.io>> wrote:
Hi Dies,

IMO the healthcheck of the app should be determined independently of whether a developer wants their app routable.

My understanding of the implications for you proposal are that a developer could not have a port-based healthcheck without mapping a route. This seems unnecessarily restrictive. Soon developers will be able to specify http healthchecks. Would these be prevented also?

Best,

Shannon Coen
Product Manager, Cloud Foundry
Pivotal, Inc.

On Wed, May 4, 2016 at 12:26 PM, Nicholas Calugar <ncalugar(a)pivotal.io<mailto:ncalugar(a)pivotal.io>> wrote:
Hi Dies,

I considered this a bit more after we chatted yesterday and I don't think we should try to create parity between DEAs and Diego in this case. My personal opinion is that behavior should be explicit and these two flags provide a more correct experience with the Diego backend.


Thanks,

Nick

On Mon, Apr 25, 2016 at 6:09 PM, Koper, Dies <diesk(a)fast.au.fujitsu.com<mailto:diesk(a)fast.au.fujitsu.com>> wrote:
Hi CLI users,

With apps deployed to DEAs, a health check is performed at application start-up targeting the app’s port, unless you specified `--no-route`, in which case the process is monitored.
With Diego, the health check is performed continuously and the type of check was exposed through an option to the `cf push` command.
This option defaults to `port`, which isn't always appropriate for apps pushed without a route, such as worker apps.

We propose fixing the `--health-check-type` option’s default value to align with the behaviour seen for DEAs, i.e. to use “none” if option `--no-route` is used:

--health-check-type, -u Application health check type (Default: 'none' if '--no-route' is set, otherwise 'port')`
Would anyone object to such change?

Cheers,
Dies Koper
Cloud Foundry CLI PM

Hi Ponraj / Padma,

Apologies for the delay. I've added a story to do environment variable
validation for our upcoming V3 api:
https://www.pivotaltracker.com/story/show/119590729

Thanks,

Nick

Have you tried `cf delete-buildpack`? or `cf update-buildpack` with a new,
working binary?

Dan

On Fri, May 13, 2016 at 8:32 PM, Eric Promislow <eric.promislow(a)gmail.com>
wrote:

I tried to upload a simple app to a local cf build, and it failed
because the go-buildpack download failed.

It turns out that the buildpack
/var/vcap/store/shared/cc-buildpacks/75/21/ \

75210aea-160c-4438-a9f9-3710734f35cd_a25a466217b64d5e4d47a6796be8ab23e7b7eeaf

had lost its pkzip header (but the end of the file looked like it was
part of a zip file).

When I moved it out of the way ('mv 75 hide-75') I got a stranger
error message, which unfortunately scrolled off my window before
I thought to report it.

What's the best way to push a faulty buildpack out of the cache?

- Eric

I tried to upload a simple app to a local cf build, and it failed
because the go-buildpack download failed.

It turns out that the buildpack
/var/vcap/store/shared/cc-buildpacks/75/21/ \
75210aea-160c-4438-a9f9-3710734f35cd_a25a466217b64d5e4d47a6796be8ab23e7b7eeaf

had lost its pkzip header (but the end of the file looked like it was
part of a zip file).

When I moved it out of the way ('mv 75 hide-75') I got a stranger
error message, which unfortunately scrolled off my window before
I thought to report it.

What's the best way to push a faulty buildpack out of the cache?

- Eric

Recently we made some changes to the dropsonde library. We changed some
interface types to be concrete structs. This has the possibility of
breaking some users of this library that specify types directly and not use
type inference.

The following are the types affected with their full import paths for easy
grepping:

github.com/cloudfoundry/dropsonde/dropsonde_marshaller
DropsondeMarshaller
github.com/cloudfoundry/dropsonde/dropsonde_unmarshaller
DropsondeUnmarshaller
DropsondeUnmarshallerCollection
github.com/cloudfoundry/dropsonde/emitter
EventEmitter
github.com/cloudfoundry/dropsonde/envelope_sender
EnvelopeSender
github.com/cloudfoundry/dropsonde/log_sender
LogSender
github.com/cloudfoundry/dropsonde/metric_sender
MetricSender


If you declare any of these types in your code you will just need to change
them to be pointers. Alternatively you can create your own local interfaces
in your packages that only specify the methods you require.

Jason
CF Loggregator