Date   

Re: problem with running uaa locally

Filip Hanik
 

there is a file that should have your error in it.

./uaa/build/reports/tests/uaa-server.log

I just tested this on a fresh clone on a mac, not yet running Sierra. and
it worked fine.
There were a few versions of UAA that didn't have the
required_configuration.yml enabled in gradle
https://github.com/cloudfoundry/uaa/blob/754f2716d3d8f2e227952f74adcfde5378b17b96/build.gradle#L271


On Sun, Oct 22, 2017 at 2:31 PM, chaitra hegde <chaitrahegde08(a)gmail.com>
wrote:

Hi Filip, Hi Alex,

I have the same problem.
I have macOS Sierra.
I just cloned the https://github.com/cloudfoundry/uaa.git .

"./gradlew run" just stuck on "Building 97% > :cargoRunLocal",
"./gradlew run --info" and "./gradlew clean assemble run --info" both
provide an output that "Tomcat 8.x started on port [8080]" , but:
"curl -H "Accept: application/json" localhost:8080/uaa/login"
always returns:
FAILURE

Alex, how did you manage that to work for you?
I am facing the same issue. If someone provides some help i'll be really
grateful.


Re: Customization of the "CF push" process

Dieu Cao <dcao@...>
 

Hey Mao,

I'd suggest the first approach with the cf CLI plugin as being the simplest
thing to experiment/start with.
Also, perhaps the multi-buildpack proposal might be worth looking into as
well.

-Dieu

On Thu, Oct 19, 2017 at 1:14 AM, Mao, Yi <yi.mao(a)siemens.com> wrote:

Dear Experts,



This is Mao Yi from Siemens CT China. And now our team has some
requirements to do some customizations to the ”CF push” process. The idea
is the application is still be deployment by CF CLI, but we want to do some
validation(by validate the MD5 of the binary file or signature) before
being deployed to Diego cell. We have thought different approaches to
implement.



Approach 1: by implement a customized CF CLI plugin and implement the
validation logic in the plugin, the disadvantages is we need to use another
syntax to do the validated push other than “CF push”

Approach 2: We plans to develop an interceptor before calling CF API, the
interceptor will listen to events to indicate some milestones of the push
process such as the app files has been stored completely in Blobstore. But
the question is that we do not know whether this kind of interceptor is
reasonable. Whether Cloud Foundry exposes the events we need?

Approach 3: I am not sure whether the CF API calls go through some
proxy/gateway or not, Is it possible to implement it by customizing the
gateway?

Approach 4: We are not sure whether there are some other way to implement
our requirement. Please give us some suggestions about other options.



Many thanks!



Regards,

Mao Yi



Siemens Ltd., China
CT RTC ITP CMI-CN
7, Wangjing Zhonghuan Nanlu
100102 BEIJING, China
Tel.: +86 10 64766397 <+86%2010%206476%206397>
Fax: +86 10 64764814 <+86%2010%206476%204814>
Mobile: +86 13811650584 <+86%20138%201165%200584>
mailto:yi.mao(a)siemens.com <yi.mao(a)siemens.com>



Re: Block internet access during staging

Daniel Mikusa
 

Peter,

It's a little more work, but what you could do is block all outbound
traffic in staging with an ASG then configure an HTTP/HTTPS proxy for the
apps that are allowed to talk out (don't forget to add an ASG for the
proxy). This would give you a lot of control over outbound connections
since each app could have its own proxy credentials and you can apply rules
on a per request basis.

Hope that helps!

Dan

On Mon, Oct 23, 2017 at 3:18 AM, Peter Dotchev <dotchev(a)gmail.com> wrote:

Hi Stephen,

It seems Application Security Groups could address some of our security
concerns.
Still ASGs have some limitations.

We don't want to restrict internet access in platform-wide ASGs as this
would break existing applications. Still we would like to allow some users
to restrict internet access in their spaces.
But ASGs can define only *allow* rules, so a space-scoped ASG can only
relax the rules defined in platform-wide ASGs.

Are there any plans to support also *deny* rules or some other approach
to restrict network access per space?

Again, our goal is to block downloading untrusted code during application
staging.

Best regards,
Peter


On Tue, Oct 17, 2017 at 9:57 AM, Peter Dotchev <dotchev(a)gmail.com> wrote:

Hi Stephen,

Thank you again for your thorough explanation.
We will explore further those CF features and check if they meet our
requirements.

Best regards,
Peter


On Mon, Oct 16, 2017 at 7:50 PM, Stephen Levine <slevine(a)pivotal.io>
wrote:

Hi Peter,

Apologies for the delay. I was at CF Summit EU last week.

First, thanks for following up with me here! While I believe that most
of your concerns are addressed by the platform, I truly appreciate this
type of feedback from the community.

I think this is relevant for most environments as any changes in
production invalidate the tests in staging. Especially changes in
application runtime carry high risk of regression for the app.

I agree! In my experience, Cloud Foundry users almost always leverage
these techniques in production.

Where can I find more info about these in particular?
- *don't permit internet access during staging at all*
- *stage vendored apps in offline containers with the same buildpack
version*

Staging offline can be accomplished with Application Security Groups
(ASGs). Specifically, you can create an ASG that only applies during
staging using the `cf create-security-group` and `cf
bind-staging-security-group` commands. Check out the docs[1] for more info.

Each buildpack provides very explicit documentation for proper app
vendoring. For example, the Node.js buildpack provides methods for both
yarn and npm[2].

I think it's worth pointing out that delivering secure, consistent
application bits to productions is actually more difficult with Docker
images. On CF, I only need to restage if my application code changes. With
Docker images, I often have to build the image from scratch just to patch
my (ABI-compatible) OS-level dependencies. Aside from that, you can vendor
your app dependencies and build/stage in offline containers very easily
with both Docker and CF. And it's good practice to make your Dockerfile
re-builds reproducible, just like it's good practice to make your CF app
stages reproducible.

As I understand CAPI v2 is the current official API. When is CAPI v3
planned to become productive?

Greg answered this, but I'd also like to add that the v2 endpoints will
let you download/upload droplets as well. The UX is just less friendly. I
have a tool called CF Local[3] that makes this easier.

Regarding OS-level patches, we should consider that CF by default
uses the same rootfs for all applications. In order to support many
application types, this image probably contains many packages that are not
used by every application. cloudfoundry/cflinuxfs2 image in Docker Hub is
about 300MB. For comparison node:alpine image is about 20MB. So using an
app-specific trimmed down image can reduce the attack surface significantly.

This is an important point. I definitely understand your concerns about
attack surface area. However, consider that when you use an Alpine based
image, you have to re-build your image every time any additional packages
that you install are updated. As long as those additional packages are in
cflinuxfs2, Cloud Foundry takes care of this automatically, without any
need for developer intervention. Also, unless Alpine keeps all their
packages patched as quickly and meticulously as Canonical does for Ubuntu,
the OS-level dependencies that your application actually uses are likely
going to be further behind in security patches, compared the equivalent app
on CF.

While I believe Cloud Foundry is more secure in this respect, I also
believe that nothing prevents us from adding platform features that give CF
the best security properties of both models. We're working on a smaller
footprint rootfs that still works with the buildpacks, and I'm hoping that
future platform features will let us provide user-selected Ubuntu packages
that are automatically always patched (unlike the apt-buildpack, which
requires re-staging to update the user-selected packages).

I hope these answers are helpful! I'm currently writing a blog post
(partially inspired by your questions) that should address more of this
in-depth.

[1] https://docs.cloudfoundry.org/concepts/asg.html
[2] https://docs.cloudfoundry.org/buildpacks/node/index.html#vendoring
[3] https://github.com/sclevine/cflocal

Thanks!
Stephen

PS: There was a misunderstanding about that cf CLI issue you brought up
(#903) that (combined with the fact that the Node.js buildpack would
re-build those symlinks in many cases) resulted in it sitting around for
far too long. The cf CLI team is looking into this again. Thanks for
bringing it up again!


On Mon, Oct 9, 2017 at 2:12 PM, Greg Cobb <gcobb(a)pivotal.io> wrote:

Regarding CAPI v3: The v3 API now has several endpoints that are
production-ready. All endpoints listed under the "Resources" section of the
v3 docs are ready for general use: http://v3-apidocs.cloudfo
undry.org/version/3.32.0/index.html#resources

Both the v2 and v3 apis will be simultaneously supported for the
foreseeable future.

On Sun, Oct 8, 2017 at 1:11 PM, Peter Dotchev <dotchev(a)gmail.com>
wrote:

Hi Stephen,

Thank you for the useful insights.
Still I have some further questions and comments.

*relevant for production environments where any risk of application
dependencies changing between staging and production is unacceptable*

I think this is relevant for most environments as any changes in
production invalidate the tests in staging. Especially changes in
application runtime carry high risk of regression for the app.

I was not aware that CF provides some of these features. Where can I
find more info about these in particular?
- *don't permit internet access during staging at all*
- *stage vendored apps in offline containers with the same buildpack
version*

*You can promote droplets between staging environments and production
environments, instead of re-staging. The CAPI v3 API makes this somewhat
straightforward.*

As I understand CAPI v2 is the current official API. When is CAPI v3
planned to become productive?

Regarding OS-level patches, we should consider that CF by default uses
the same rootfs for all applications. In order to support many application
types, this image probably contains many packages that are not used by
every application. cloudfoundry/cflinuxfs2 image in Docker Hub is about
300MB. For comparison node:alpine image is about 20MB. So using an
app-specific trimmed down image can reduce the attack surface significantly.

Best regards,
Peter



On Thu, Oct 5, 2017 at 6:00 PM, Stephen Levine <slevine(a)pivotal.io>
wrote:

Hi Peter,

Many of your criticisms about buildpack dependencies stem from using
online (uncached) buildpacks with apps that aren't vendored. Most
enterprise CF users use offline (cached) buildpacks, and many vendor their
apps and don't permit internet access during staging at all. None of the
downloading you describe (including the downloading of the buildpack)
happens on an optimized deployment of CF with a vendored app.

Your criticisms about "re-deployment" of already deployed apps are
perhaps relevant for production environments where any risk of application
dependencies changing between staging and production is unacceptable.
There are plenty of options for these environments though:
1. You can promote droplets between staging environments and
production environments, instead of re-staging. The CAPI v3 API makes this
somewhat straightforward. Nothing stops you from treating staging like an
image build.
2. You can stage vendored apps in offline containers with the same
buildpack version. This should always produce the same result.

I actually see the "always the same" property of OCI images as a
disadvantage for application security. With droplets in CF, you can patch
the OS-level dependencies for every app on a foundation within hours of a
USN announcement with zero downtime. The droplet model takes advantage of
the ABI compatibility that Canonical ensures for patches to Ubuntu 14.04
packages. If you try to do the same thing with OCI-layer builders like
Docker, you end up reinventing droplets.

As for the flexibility of the buildpack model: many of your
criticisms are addressed by the new multi-buildpack feature. We're about to
release a new (experimental) multi-buildpack-compatible apt-buildpack[1]
that will let you install Ubuntu packages into droplets (and longer term,
I'm hoping we can add additional platform features that will allow
user-specified Ubuntu packages to be patched when Canonical announces a USN
that affects them).

And finally, for single-buildpack use cases, Heroku buildpacks are
compatible with Cloud Foundry, and Cloud Foundry buildpacks are compatible
with Heroku. We have no plans to change that. And if you need Heroku-style
multi-buildpack support in CF, you can use Heroku's multi-buildpack shim
buildpack. Similarly, if you need Cloud Foundry's multi-buildpack support
in Heroku, you can use Cloud Foundry's multi-buildpack shim buildpack.
Buildpacks are not proprietary.

-Stephen


[1] https://github.com/cloudfoundry/apt-buildpack/tree/develop




On Thu, Oct 5, 2017 at 10:02 AM, Peter Dotchev <dotchev(a)gmail.com>
wrote:

Hi,

For a long time I felt there is something wrong with the buildpack
concept. When Lattice <http://lattice.cf/> was announced I thought
that CF is moving away from buildpacks. Then it turned out this is not the
case.
So I finally took the time and summarized all the issues that bug me.
You can find it here https://medium.com/@dotch
ev/whats-wrong-with-buildpacks-10ca6789ef80

Do you still believe that buildpacks are better for application
deployment than containers?

Best regards,
Peter


Re: Recommendations for Service parameters

Aniruddha Kulkarni
 

Thanks, that makes sense. Whether it is the create service bind call or
create service key call, the broker will expect the same/similar "-c"
parameters.

Regards,
Ani

On Mon, Oct 23, 2017 at 2:24 PM, Matt McNeeney <mmcneeney(a)pivotal.io> wrote:

Hey Aniruddha,

From a service broker's perspective, there isn't really a difference
between a service binding and a service key (and so the same schema that is
exposed via *service_binding.create *should be used).

The only difference that a service broker sees is that the *app_guid* field
is only sent to service brokers from CC when running *cf bind-service *(the
field is not set when running *cf create-service-key*).

Let me know if that doesn't make sense,
Matt


On Mon, Oct 23, 2017 at 5:57 AM Aniruddha Kulkarni <aquila.25(a)gmail.com>
wrote:

Hi Matt,
I have one more follow-up question regarding the "schemas"
object in the service metadata. As mentioned in the link above, we can have
"schemas" specified for Service instance "create/edit" and Service binding
"create". CF also supports specifying parameters when creating Service
Keys.

My question is whether it makes sense to use the Service binding
"schemas" object during service key creation as well, since logically
service key and service binding are similar?

Thanks,
Ani

On Wed, Sep 13, 2017 at 7:14 PM, Aniruddha Kulkarni <aquila.25(a)gmail.com>
wrote:

Thank you, this looks like exactly what I was looking for.

Regards,
Ani

On Tue, Sep 12, 2017 at 11:08 AM, Matt McNeeney <mmcneeney(a)pivotal.io>
wrote:

Hi Aniruddha,

A new feature has just been merged into the Open Service Broker API
specification that allows service brokers to use JSON schemes in their
catalogs to define the configuration parameters they accept for creating a
service instance, updating a service instance and creating a service
binding: https://github.com/openservicebrokerapi/
servicebroker/blob/master/spec.md#schema-object

If you are new to JSON schema, this excellent guide should help you:
https://spacetelescope.github.io/understanding-json-schema/

This feature will be supported very soon in Cloud Foundry when the next
version of the spec is released.

Let me know if you have any further questions,
Matt


On Tue, 12 Sep 2017 at 06:20, Aniruddha Kulkarni <aquila.25(a)gmail.com>
wrote:

Hello,
CF provides support for specifying service parameters during:

1. Service instance creation (cf cli: -c option)
2. Service instance bind to an application.
3. Service key creation.

The format of these parameters is owned by the individual services.

My question is what is the recommendation regarding how to "publish"
the service parameter's format to the end-developers who will be using a
Service? i.e. how will an end developer know about what parameters are to
be provided?

Is there any guidance w.r.t to using the service/service plan metadata
for providing this information?

Regards,
--
-Aniruddha Kulkarni

--
-Ani


--
-Ani
--
-Ani


Re: Recommendations for Service parameters

Matt McNeeney
 

Hey Aniruddha,

From a service broker's perspective, there isn't really a difference
between a service binding and a service key (and so the same schema that is
exposed via *service_binding.create *should be used).

The only difference that a service broker sees is that the *app_guid* field
is only sent to service brokers from CC when running *cf bind-service *(the
field is not set when running *cf create-service-key*).

Let me know if that doesn't make sense,
Matt


On Mon, Oct 23, 2017 at 5:57 AM Aniruddha Kulkarni <aquila.25(a)gmail.com>
wrote:

Hi Matt,
I have one more follow-up question regarding the "schemas"
object in the service metadata. As mentioned in the link above, we can have
"schemas" specified for Service instance "create/edit" and Service binding
"create". CF also supports specifying parameters when creating Service
Keys.

My question is whether it makes sense to use the Service binding "schemas"
object during service key creation as well, since logically service key and
service binding are similar?

Thanks,
Ani

On Wed, Sep 13, 2017 at 7:14 PM, Aniruddha Kulkarni <aquila.25(a)gmail.com>
wrote:

Thank you, this looks like exactly what I was looking for.

Regards,
Ani

On Tue, Sep 12, 2017 at 11:08 AM, Matt McNeeney <mmcneeney(a)pivotal.io>
wrote:

Hi Aniruddha,

A new feature has just been merged into the Open Service Broker API
specification that allows service brokers to use JSON schemes in their
catalogs to define the configuration parameters they accept for creating a
service instance, updating a service instance and creating a service
binding:
https://github.com/openservicebrokerapi/servicebroker/blob/master/spec.md#schema-object

If you are new to JSON schema, this excellent guide should help you:
https://spacetelescope.github.io/understanding-json-schema/

This feature will be supported very soon in Cloud Foundry when the next
version of the spec is released.

Let me know if you have any further questions,
Matt


On Tue, 12 Sep 2017 at 06:20, Aniruddha Kulkarni <aquila.25(a)gmail.com>
wrote:

Hello,
CF provides support for specifying service parameters during:

1. Service instance creation (cf cli: -c option)
2. Service instance bind to an application.
3. Service key creation.

The format of these parameters is owned by the individual services.

My question is what is the recommendation regarding how to "publish"
the service parameter's format to the end-developers who will be using a
Service? i.e. how will an end developer know about what parameters are to
be provided?

Is there any guidance w.r.t to using the service/service plan metadata
for providing this information?

Regards,
--
-Aniruddha Kulkarni

--
-Ani


--
-Ani


Re: Block internet access during staging

Peter Dotchev <dotchev@...>
 

Hi Stephen,

It seems Application Security Groups could address some of our security
concerns.
Still ASGs have some limitations.

We don't want to restrict internet access in platform-wide ASGs as this
would break existing applications. Still we would like to allow some users
to restrict internet access in their spaces.
But ASGs can define only *allow* rules, so a space-scoped ASG can only
relax the rules defined in platform-wide ASGs.

Are there any plans to support also *deny* rules or some other approach to
restrict network access per space?

Again, our goal is to block downloading untrusted code during application
staging.

Best regards,
Peter

On Tue, Oct 17, 2017 at 9:57 AM, Peter Dotchev <dotchev(a)gmail.com> wrote:

Hi Stephen,

Thank you again for your thorough explanation.
We will explore further those CF features and check if they meet our
requirements.

Best regards,
Peter


On Mon, Oct 16, 2017 at 7:50 PM, Stephen Levine <slevine(a)pivotal.io>
wrote:

Hi Peter,

Apologies for the delay. I was at CF Summit EU last week.

First, thanks for following up with me here! While I believe that most of
your concerns are addressed by the platform, I truly appreciate this type
of feedback from the community.

I think this is relevant for most environments as any changes in
production invalidate the tests in staging. Especially changes in
application runtime carry high risk of regression for the app.

I agree! In my experience, Cloud Foundry users almost always leverage
these techniques in production.

Where can I find more info about these in particular?
- *don't permit internet access during staging at all*
- *stage vendored apps in offline containers with the same buildpack
version*

Staging offline can be accomplished with Application Security Groups
(ASGs). Specifically, you can create an ASG that only applies during
staging using the `cf create-security-group` and `cf
bind-staging-security-group` commands. Check out the docs[1] for more info.

Each buildpack provides very explicit documentation for proper app
vendoring. For example, the Node.js buildpack provides methods for both
yarn and npm[2].

I think it's worth pointing out that delivering secure, consistent
application bits to productions is actually more difficult with Docker
images. On CF, I only need to restage if my application code changes. With
Docker images, I often have to build the image from scratch just to patch
my (ABI-compatible) OS-level dependencies. Aside from that, you can vendor
your app dependencies and build/stage in offline containers very easily
with both Docker and CF. And it's good practice to make your Dockerfile
re-builds reproducible, just like it's good practice to make your CF app
stages reproducible.

As I understand CAPI v2 is the current official API. When is CAPI v3
planned to become productive?

Greg answered this, but I'd also like to add that the v2 endpoints will
let you download/upload droplets as well. The UX is just less friendly. I
have a tool called CF Local[3] that makes this easier.

Regarding OS-level patches, we should consider that CF by default uses
the same rootfs for all applications. In order to support many application
types, this image probably contains many packages that are not used by
every application. cloudfoundry/cflinuxfs2 image in Docker Hub is about
300MB. For comparison node:alpine image is about 20MB. So using an
app-specific trimmed down image can reduce the attack surface significantly.

This is an important point. I definitely understand your concerns about
attack surface area. However, consider that when you use an Alpine based
image, you have to re-build your image every time any additional packages
that you install are updated. As long as those additional packages are in
cflinuxfs2, Cloud Foundry takes care of this automatically, without any
need for developer intervention. Also, unless Alpine keeps all their
packages patched as quickly and meticulously as Canonical does for Ubuntu,
the OS-level dependencies that your application actually uses are likely
going to be further behind in security patches, compared the equivalent app
on CF.

While I believe Cloud Foundry is more secure in this respect, I also
believe that nothing prevents us from adding platform features that give CF
the best security properties of both models. We're working on a smaller
footprint rootfs that still works with the buildpacks, and I'm hoping that
future platform features will let us provide user-selected Ubuntu packages
that are automatically always patched (unlike the apt-buildpack, which
requires re-staging to update the user-selected packages).

I hope these answers are helpful! I'm currently writing a blog post
(partially inspired by your questions) that should address more of this
in-depth.

[1] https://docs.cloudfoundry.org/concepts/asg.html
[2] https://docs.cloudfoundry.org/buildpacks/node/index.html#vendoring
[3] https://github.com/sclevine/cflocal

Thanks!
Stephen

PS: There was a misunderstanding about that cf CLI issue you brought up
(#903) that (combined with the fact that the Node.js buildpack would
re-build those symlinks in many cases) resulted in it sitting around for
far too long. The cf CLI team is looking into this again. Thanks for
bringing it up again!


On Mon, Oct 9, 2017 at 2:12 PM, Greg Cobb <gcobb(a)pivotal.io> wrote:

Regarding CAPI v3: The v3 API now has several endpoints that are
production-ready. All endpoints listed under the "Resources" section of the
v3 docs are ready for general use: http://v3-apidocs.cloudfo
undry.org/version/3.32.0/index.html#resources

Both the v2 and v3 apis will be simultaneously supported for the
foreseeable future.

On Sun, Oct 8, 2017 at 1:11 PM, Peter Dotchev <dotchev(a)gmail.com> wrote:

Hi Stephen,

Thank you for the useful insights.
Still I have some further questions and comments.

*relevant for production environments where any risk of application
dependencies changing between staging and production is unacceptable*

I think this is relevant for most environments as any changes in
production invalidate the tests in staging. Especially changes in
application runtime carry high risk of regression for the app.

I was not aware that CF provides some of these features. Where can I
find more info about these in particular?
- *don't permit internet access during staging at all*
- *stage vendored apps in offline containers with the same buildpack
version*

*You can promote droplets between staging environments and production
environments, instead of re-staging. The CAPI v3 API makes this somewhat
straightforward.*

As I understand CAPI v2 is the current official API. When is CAPI v3
planned to become productive?

Regarding OS-level patches, we should consider that CF by default uses
the same rootfs for all applications. In order to support many application
types, this image probably contains many packages that are not used by
every application. cloudfoundry/cflinuxfs2 image in Docker Hub is about
300MB. For comparison node:alpine image is about 20MB. So using an
app-specific trimmed down image can reduce the attack surface significantly.

Best regards,
Peter



On Thu, Oct 5, 2017 at 6:00 PM, Stephen Levine <slevine(a)pivotal.io>
wrote:

Hi Peter,

Many of your criticisms about buildpack dependencies stem from using
online (uncached) buildpacks with apps that aren't vendored. Most
enterprise CF users use offline (cached) buildpacks, and many vendor their
apps and don't permit internet access during staging at all. None of the
downloading you describe (including the downloading of the buildpack)
happens on an optimized deployment of CF with a vendored app.

Your criticisms about "re-deployment" of already deployed apps are
perhaps relevant for production environments where any risk of application
dependencies changing between staging and production is unacceptable.
There are plenty of options for these environments though:
1. You can promote droplets between staging environments and
production environments, instead of re-staging. The CAPI v3 API makes this
somewhat straightforward. Nothing stops you from treating staging like an
image build.
2. You can stage vendored apps in offline containers with the same
buildpack version. This should always produce the same result.

I actually see the "always the same" property of OCI images as a
disadvantage for application security. With droplets in CF, you can patch
the OS-level dependencies for every app on a foundation within hours of a
USN announcement with zero downtime. The droplet model takes advantage of
the ABI compatibility that Canonical ensures for patches to Ubuntu 14.04
packages. If you try to do the same thing with OCI-layer builders like
Docker, you end up reinventing droplets.

As for the flexibility of the buildpack model: many of your criticisms
are addressed by the new multi-buildpack feature. We're about to release a
new (experimental) multi-buildpack-compatible apt-buildpack[1] that will
let you install Ubuntu packages into droplets (and longer term, I'm hoping
we can add additional platform features that will allow user-specified
Ubuntu packages to be patched when Canonical announces a USN that affects
them).

And finally, for single-buildpack use cases, Heroku buildpacks are
compatible with Cloud Foundry, and Cloud Foundry buildpacks are compatible
with Heroku. We have no plans to change that. And if you need Heroku-style
multi-buildpack support in CF, you can use Heroku's multi-buildpack shim
buildpack. Similarly, if you need Cloud Foundry's multi-buildpack support
in Heroku, you can use Cloud Foundry's multi-buildpack shim buildpack.
Buildpacks are not proprietary.

-Stephen


[1] https://github.com/cloudfoundry/apt-buildpack/tree/develop




On Thu, Oct 5, 2017 at 10:02 AM, Peter Dotchev <dotchev(a)gmail.com>
wrote:

Hi,

For a long time I felt there is something wrong with the buildpack
concept. When Lattice <http://lattice.cf/> was announced I thought
that CF is moving away from buildpacks. Then it turned out this is not the
case.
So I finally took the time and summarized all the issues that bug me.
You can find it here https://medium.com/@dotch
ev/whats-wrong-with-buildpacks-10ca6789ef80

Do you still believe that buildpacks are better for application
deployment than containers?

Best regards,
Peter


Re: Recommendations for Service parameters

Aniruddha Kulkarni
 

Hi Matt,
I have one more follow-up question regarding the "schemas"
object in the service metadata. As mentioned in the link above, we can have
"schemas" specified for Service instance "create/edit" and Service binding
"create". CF also supports specifying parameters when creating Service
Keys.

My question is whether it makes sense to use the Service binding "schemas"
object during service key creation as well, since logically service key and
service binding are similar?

Thanks,
Ani

On Wed, Sep 13, 2017 at 7:14 PM, Aniruddha Kulkarni <aquila.25(a)gmail.com>
wrote:

Thank you, this looks like exactly what I was looking for.

Regards,
Ani

On Tue, Sep 12, 2017 at 11:08 AM, Matt McNeeney <mmcneeney(a)pivotal.io>
wrote:

Hi Aniruddha,

A new feature has just been merged into the Open Service Broker API
specification that allows service brokers to use JSON schemes in their
catalogs to define the configuration parameters they accept for creating a
service instance, updating a service instance and creating a service
binding: https://github.com/openservicebrokerapi/servicebroker/blob/
master/spec.md#schema-object

If you are new to JSON schema, this excellent guide should help you:
https://spacetelescope.github.io/understanding-json-schema/

This feature will be supported very soon in Cloud Foundry when the next
version of the spec is released.

Let me know if you have any further questions,
Matt


On Tue, 12 Sep 2017 at 06:20, Aniruddha Kulkarni <aquila.25(a)gmail.com>
wrote:

Hello,
CF provides support for specifying service parameters during:

1. Service instance creation (cf cli: -c option)
2. Service instance bind to an application.
3. Service key creation.

The format of these parameters is owned by the individual services.

My question is what is the recommendation regarding how to "publish" the
service parameter's format to the end-developers who will be using a
Service? i.e. how will an end developer know about what parameters are to
be provided?

Is there any guidance w.r.t to using the service/service plan metadata
for providing this information?

Regards,
--
-Aniruddha Kulkarni

--
-Ani


--
-Ani


Application monitoring on Cloud Foundry

XJ Z
 

Hello.
I'd like to know what the community provides/uses for application monitoring of apps deployed on Cloud Foundry.
If someone could share how you implement application (on cf) monitoring system or some ideas, I will deeply appreciate it
Thank you.


Re: How stateless are service brokers?

Matt McNeeney
 

Thanks for the feedback all. It sounds like there are a mix of brokers out
there:
- some are totally stateless
- some manage their own state
- some can rely on other components (like BOSH) or things like names and
tags to hold state

Knowing this, I will try to ensure to ensure than the Open Service Broker
API specification continues to support all of these setups and that any new
features that require state will be optional for brokers to support.

Best wishes,
Matt

On Fri, 20 Oct 2017 at 11:22, Sascha Matzke <sascha.matzke(a)didolo.org>
wrote:

Hi,

we have several internally used brokers.

While we try to avoid keeping state in the broker (by encoding the state
in backend instances names, tags or everything else that can hold
information and allows to query it), we couldn't avoid in all cases.

So yes, our brokers partially keep state and most of them also support
async operations (but that's not directly coupled to whether or not they
keep state).

Best,

Sascha



On Thu, Oct 19, 2017 at 2:58 PM, Matt McNeeney <mmcneeney(a)pivotal.io>
wrote:

Hi all,

Members of the Open Service Broker API
<https://www.openservicebrokerapi.org/> group are currently asking about *how
stateless service brokers really are*. There are features in the spec (
asynchronous
<https://github.com/openservicebrokerapi/servicebroker/blob/v2.13/spec.md#asynchronous-operations> instance
provisioning) and features that we want to get into the spec (allowing
platforms to GET
<https://github.com/openservicebrokerapi/servicebroker/pull/333> instances
and bindings) that we believe require some degree of statefulness, and so
we want to understand this space better before making these changes.

I'm keen to understand from Service Broker authors whether or not the
service brokers you have created keep any state (either on their own or by
relying on other platform components), and if not, whether or not you
currently support or plan to support asynchronous operations (provisioning
and binding)?

Many thanks,
Matt
*CF Services API PM*



--
Through the darkness of future past
the magician longs to see
One chants out between two worlds
*Fire walk with me*.


Re: Is there a way to extract config parameters for a service?

Matt McNeeney
 

Hey Ankit,

This isn’t possible right now, but is actively being worked on by be Open
Service Broker API community:
https://github.com/openservicebrokerapi/servicebroker/pull/333

We are hoping to integrate this feature into the specification soon, at
which point service brokers can start exposing the new GET endpoints, and
then Cloud Foundry can expose this through the Cloud Controller API. Stay
tuned!

Matt

On Fri, 20 Oct 2017 at 14:14, Ankit Chaudhari <ankitjc(a)gmail.com> wrote:

In the process of deployment - I want to delete an app, delete all
services binded to it. And then recreate an app, recreate all services and
bind them back.

From the list of apis, I figured I can get list of binded services for
apps (
https://apidocs.cloudfoundry.org/272/apps/list_all_service_bindings_for_the_app.html
)

I am searching if there is a way to get the configuration parameters that
were passed when creating the service.
for eg. I have a config server, initiated with git, it has params like -c
"{'git':{...}}". Any way to get this info?


Re: How stateless are service brokers?

Robert Moss <robert.moss@...>
 

Hi,

The apache-brooklyn-service-broker
<https://github.com/cloudfoundry-incubator/apache-brooklyn-service-broker>
keeps
the brokers state in Apache Brooklyn: it create a mapping between the
Brooklyn entities and the service instances or bindings. It makes heavy
use of async operations too and has an internal state machine to make
mappings between the Brooklyn lifecycle events and CF.

Best,
Robert

On 20 October 2017 at 11:22, Sascha Matzke <sascha.matzke(a)didolo.org> wrote:

Hi,

we have several internally used brokers.

While we try to avoid keeping state in the broker (by encoding the state
in backend instances names, tags or everything else that can hold
information and allows to query it), we couldn't avoid in all cases.

So yes, our brokers partially keep state and most of them also support
async operations (but that's not directly coupled to whether or not they
keep state).

Best,

Sascha



On Thu, Oct 19, 2017 at 2:58 PM, Matt McNeeney <mmcneeney(a)pivotal.io>
wrote:

Hi all,

Members of the Open Service Broker API
<https://www.openservicebrokerapi.org/> group are currently asking about *how
stateless service brokers really are*. There are features in the spec (
asynchronous
<https://github.com/openservicebrokerapi/servicebroker/blob/v2.13/spec.md#asynchronous-operations> instance
provisioning) and features that we want to get into the spec (allowing
platforms to GET
<https://github.com/openservicebrokerapi/servicebroker/pull/333> instances
and bindings) that we believe require some degree of statefulness, and so
we want to understand this space better before making these changes.

I'm keen to understand from Service Broker authors whether or not the
service brokers you have created keep any state (either on their own or by
relying on other platform components), and if not, whether or not you
currently support or plan to support asynchronous operations (provisioning
and binding)?

Many thanks,
Matt
*CF Services API PM*



--
Through the darkness of future past
the magician longs to see
One chants out between two worlds
*Fire walk with me*.


Is there a way to extract config parameters for a service?

Ankit Chaudhari
 

In the process of deployment - I want to delete an app, delete all services binded to it. And then recreate an app, recreate all services and bind them back.

From the list of apis, I figured I can get list of binded services for apps (https://apidocs.cloudfoundry.org/272/apps/list_all_service_bindings_for_the_app.html)

I am searching if there is a way to get the configuration parameters that were passed when creating the service.
for eg. I have a config server, initiated with git, it has params like -c "{'git':{...}}". Any way to get this info?


Re: How stateless are service brokers?

Sascha Matzke
 

Hi,

we have several internally used brokers.

While we try to avoid keeping state in the broker (by encoding the state in
backend instances names, tags or everything else that can hold information
and allows to query it), we couldn't avoid in all cases.

So yes, our brokers partially keep state and most of them also support
async operations (but that's not directly coupled to whether or not they
keep state).

Best,

Sascha

On Thu, Oct 19, 2017 at 2:58 PM, Matt McNeeney <mmcneeney(a)pivotal.io> wrote:

Hi all,

Members of the Open Service Broker API
<https://www.openservicebrokerapi.org/> group are currently asking about *how
stateless service brokers really are*. There are features in the spec (
asynchronous
<https://github.com/openservicebrokerapi/servicebroker/blob/v2.13/spec.md#asynchronous-operations> instance
provisioning) and features that we want to get into the spec (allowing
platforms to GET
<https://github.com/openservicebrokerapi/servicebroker/pull/333> instances
and bindings) that we believe require some degree of statefulness, and so
we want to understand this space better before making these changes.

I'm keen to understand from Service Broker authors whether or not the
service brokers you have created keep any state (either on their own or by
relying on other platform components), and if not, whether or not you
currently support or plan to support asynchronous operations (provisioning
and binding)?

Many thanks,
Matt
*CF Services API PM*


--
Through the darkness of future past
the magician longs to see
One chants out between two worlds
*Fire walk with me*.


Re: How stateless are service brokers?

Dr Nic Williams <drnicwilliams@...>
 

Dmitriy's On Demand Broker looks to be stateless https://github.com/cppforlife/bosh-generic-sb-release/blob/master/manifests/broker.yml#L24

________________________________
From: Gareth Smith <gsmith(a)pivotal.io>
Sent: Friday, October 20, 2017 5:11:40 PM
To: Discussions about Cloud Foundry projects and the system overall.
Subject: [cf-dev] Re: Re: How stateless are service brokers?

I think both the cf rabbitmq broker<https://github.com/pivotal-cf/cf-rabbitmq-multitenant-broker-release> and the on demand broker<https://github.com/pivotal-cf/cf-rabbitmq-multitenant-broker-release> are currently stateless.

G

On Fri, Oct 20, 2017 at 1:12 AM, Marco Nicosia <mnicosia(a)pivotal.io<mailto:mnicosia(a)pivotal.io>> wrote:
Matt,

cf-mysql-broker is stateful.

We have a feature of the service plans which defines how much storage a plan may use; this isn't a core feature of MySQL. We leverage the fact that we have a DB at our disposal to remember how much space each service-instance is allowed, and monitor that.

On Thu, Oct 19, 2017 at 5:59 AM Matt McNeeney <mmcneeney(a)pivotal.io<mailto:mmcneeney(a)pivotal.io>> wrote:
Hi all,

Members of the Open Service Broker API<https://www.openservicebrokerapi.org/> group are currently asking about how stateless service brokers really are. There are features in the spec (asynchronous<https://github.com/openservicebrokerapi/servicebroker/blob/v2.13/spec.md#asynchronous-operations> instance provisioning) and features that we want to get into the spec (allowing platforms to GET<https://github.com/openservicebrokerapi/servicebroker/pull/333> instances and bindings) that we believe require some degree of statefulness, and so we want to understand this space better before making these changes.

I'm keen to understand from Service Broker authors whether or not the service brokers you have created keep any state (either on their own or by relying on other platform components), and if not, whether or not you currently support or plan to support asynchronous operations (provisioning and binding)?

Many thanks,
Matt
CF Services API PM


--
--
Marco Nicosia
Product Manager
Pivotal Software, Inc.
mnicosia(a)pivotal.io<mailto:mnicosia(a)pivotal.io>
c: 650-796-2948<tel:(650)%20796-2948>


Re: How stateless are service brokers?

Gareth Smith
 

On Fri, Oct 20, 2017 at 1:12 AM, Marco Nicosia <mnicosia(a)pivotal.io> wrote:

Matt,

cf-mysql-broker is stateful.

We have a feature of the service plans which defines how much storage a
plan may use; this isn't a core feature of MySQL. We leverage the fact that
we have a DB at our disposal to remember how much space each
service-instance is allowed, and monitor that.

On Thu, Oct 19, 2017 at 5:59 AM Matt McNeeney <mmcneeney(a)pivotal.io>
wrote:

Hi all,

Members of the Open Service Broker API
<https://www.openservicebrokerapi.org/> group are currently asking about *how
stateless service brokers really are*. There are features in the spec (
asynchronous
<https://github.com/openservicebrokerapi/servicebroker/blob/v2.13/spec.md#asynchronous-operations> instance
provisioning) and features that we want to get into the spec (allowing
platforms to GET
<https://github.com/openservicebrokerapi/servicebroker/pull/333> instances
and bindings) that we believe require some degree of statefulness, and so
we want to understand this space better before making these changes.

I'm keen to understand from Service Broker authors whether or not the
service brokers you have created keep any state (either on their own or by
relying on other platform components), and if not, whether or not you
currently support or plan to support asynchronous operations (provisioning
and binding)?

Many thanks,
Matt
*CF Services API PM*


--
--
Marco Nicosia
Product Manager
Pivotal Software, Inc.
mnicosia(a)pivotal.io
c: 650-796-2948 <(650)%20796-2948>


[INFO] Bi-Weekly Runtime PMC Notes 2017-10-18

Dieu Cao <dcao@...>
 

Hello all,

Notes from the Runtime PMC meeting from earlier this week available here [1]

Questions? Reach out here or on slack!

-Dieu Cao
Application Runtime PMC Lead

[1]
https://github.com/cloudfoundry/pmc-notes/blob/master/Runtime/2017/2017-10-18-runtime.md


Re: How stateless are service brokers?

Marco Nicosia
 

Matt,

cf-mysql-broker is stateful.

We have a feature of the service plans which defines how much storage a
plan may use; this isn't a core feature of MySQL. We leverage the fact that
we have a DB at our disposal to remember how much space each
service-instance is allowed, and monitor that.

On Thu, Oct 19, 2017 at 5:59 AM Matt McNeeney <mmcneeney(a)pivotal.io> wrote:

Hi all,

Members of the Open Service Broker API
<https://www.openservicebrokerapi.org/> group are currently asking about *how
stateless service brokers really are*. There are features in the spec (
asynchronous
<https://github.com/openservicebrokerapi/servicebroker/blob/v2.13/spec.md#asynchronous-operations> instance
provisioning) and features that we want to get into the spec (allowing
platforms to GET
<https://github.com/openservicebrokerapi/servicebroker/pull/333> instances
and bindings) that we believe require some degree of statefulness, and so
we want to understand this space better before making these changes.

I'm keen to understand from Service Broker authors whether or not the
service brokers you have created keep any state (either on their own or by
relying on other platform components), and if not, whether or not you
currently support or plan to support asynchronous operations (provisioning
and binding)?

Many thanks,
Matt
*CF Services API PM*


--
--
Marco Nicosia
Product Manager
Pivotal Software, Inc.
mnicosia(a)pivotal.io
c: 650-796-2948


Re: How stateless are service brokers?

Mike Youngstrom
 

We start with the goal of making every broker stateless. But end up making
almost everyone stateful. Most of the system we integrate with using a
service broker don't provide the api mechanisms necessary for us to make
the broker stateless. So we almost always end up storing some information
mapping CF details with service provider details in a db for the broker.

None of our services today support asynchronous operations but we know
we're going to eventually need to do so as we integrate with services that
take longer to provision and/or bind than we like. We've simply gotten
lucky to this point. Our rule of thumb is if it takes longer than 2
minutes we need to make it async. We haven't hit that yet with any of our
services.

Does that help answer your question?

Mike

On Thu, Oct 19, 2017 at 6:58 AM, Matt McNeeney <mmcneeney(a)pivotal.io> wrote:

Hi all,

Members of the Open Service Broker API
<https://www.openservicebrokerapi.org/> group are currently asking about *how
stateless service brokers really are*. There are features in the spec (
asynchronous
<https://github.com/openservicebrokerapi/servicebroker/blob/v2.13/spec.md#asynchronous-operations> instance
provisioning) and features that we want to get into the spec (allowing
platforms to GET
<https://github.com/openservicebrokerapi/servicebroker/pull/333> instances
and bindings) that we believe require some degree of statefulness, and so
we want to understand this space better before making these changes.

I'm keen to understand from Service Broker authors whether or not the
service brokers you have created keep any state (either on their own or by
relying on other platform components), and if not, whether or not you
currently support or plan to support asynchronous operations (provisioning
and binding)?

Many thanks,
Matt
*CF Services API PM*



Re: How stateless are service brokers?

Dr Nic Williams <drnicwilliams@...>
 

Two stateless brokers I can think of are:
* cf-containers-broker - all its state is moved into Docker container metadata (broker API guids become Docker container names; parameters become env vars for containers)
* cf-subway - it allocates incoming requests to a random backend; one day it might have an in memory cache of what it remembers about its backend service broker APIs

The former could probably look up its state from Docker API. The latter could delegate its lack of local state to the backend APIs that would have the new stateful APIs (GET).

I would like to see the OSBAPI expanded to add GET commands.


________________________________
From: Matt McNeeney <mmcneeney(a)pivotal.io>
Sent: Thursday, October 19, 2017 10:58:31 PM
To: Discussions about Cloud Foundry projects and the system overall.
Subject: [cf-dev] How stateless are service brokers?

Hi all,

Members of the Open Service Broker API<https://www.openservicebrokerapi.org/> group are currently asking about how stateless service brokers really are. There are features in the spec (asynchronous<https://github.com/openservicebrokerapi/servicebroker/blob/v2.13/spec.md#asynchronous-operations> instance provisioning) and features that we want to get into the spec (allowing platforms to GET<https://github.com/openservicebrokerapi/servicebroker/pull/333> instances and bindings) that we believe require some degree of statefulness, and so we want to understand this space better before making these changes.

I'm keen to understand from Service Broker authors whether or not the service brokers you have created keep any state (either on their own or by relying on other platform components), and if not, whether or not you currently support or plan to support asynchronous operations (provisioning and binding)?

Many thanks,
Matt
CF Services API PM


Re: Droplet upload failing

Eric Malm <emalm@...>
 

Hi, Nitin,

The cell rep logs should be under /var/vcap/sys/log/rep/rep.stdout.log, or
possibly the related log-rotated files in that directory. From the
timestamp, it looks like that task ran at about 10:40 AM PDT today, so I
would expect the related logs still to be on that cell VM.

Also, the droplet upload goes through the cc-uploader component on the
cc_bridge VM, so it may be worthwhile to look at the logs for those
component instnaces as well. That component is stateless and resolved via
Consul DNS (cc-uploader.service.cf.internal) so the cell will communicate
with any one of them for the upload.

Best,
Eric

On Thu, Oct 19, 2017 at 10:48 AM, nitin padalia <padalia.nitin(a)gmail.com>
wrote:

Thanks! Eric,
I found out the cell guid for CF push event from auctioneer logs:
{"timestamp":"1508434820.493932724","source":"auctioneer","message":"
auctioneer.auction.task-added-to-cell","log_level":1,"data":
{"cell-guid":"21a6ce4c-09c1-1332-a83e-ab2ea4f4d6ea","
session":"1718","task-guid":"c1c5fbda-19a7-4da8-b710-16e1b4f1c4fb"}}
{"timestamp":"1508434820.493979216","source":"auctioneer","message":"
auctioneer.auction.scheduled","log_level":1,"data":{"failed-
lrp-start-auctions":0,"failed-task-auctions":0,"session":"
1718","successful-lrp-start-auctions":0,"successful-task-auctions":1}}

However when I checked the logs of rep in that cell
(21a6ce4c-09c1-1332-a83e-ab2ea4f4d6ea), I don't see any recent logs.
Stager logs in cc_bridge doen't point to anything except saying staging
error.

1981 - 2000 of 9387