Date   

Re: "application_version" is changed although source code is not changed.

Matthew Sykes <matthew.sykes@...>
 

The application version really represents the versioned metadata for the
application. It changes regularly. [1] The only guarantee for version is
that it will change if the app restarts (could be due to package changes or
explicit operation), the health check changes, or the memory changes.

The droplet hash is not an appropriate representation of "application"
version since it doesn't take enough into account. It is something that
could be added to VCAP_APPLICATION but I think people would want to
understand why it's needed there.

Out of curiosity, are people acting on the application version in a way
that breaks things if it changes when the app bits or droplet did not? If
so, how?

Thanks.

[1]:
https://github.com/cloudfoundry/cloud_controller_ng/blob/master/app/models/runtime/app.rb#L164-L178

On Fri, Oct 16, 2015 at 3:15 AM, Gerhard Lazu <gerhard(a)cloudcredo.com>
wrote:

I was also assuming that app version would not change between app restarts
(when the same droplet is used). I was wrong.

Would it make more sense if the app version was the droplet shasum?

On Thursday, 15 October 2015, CF Runtime <cfruntime(a)gmail.com> wrote:

application_version is mostly an internal cloud foundry concern. The DEAs
broadcast the application version they are running, and the health manager
uses that with what version it expects to be running to converge the system
on the desired state.

Restarting the app is supposed to terminate the old instances and start
the new ones, so they new ones get a new application version so they are
separate from the old.

Joseph
CF Release Integration Team

On Thu, Oct 15, 2015 at 2:00 AM, Hiroaki Ukaji <dt3snow.w(a)gmail.com>
wrote:

Hi.
I have a question about a json field "application_version" in an
environment
variable "VCAP_APPLICATION".

By intuition, "application_version" should be only changed when there is
some update for an application.
Actually, when an application is terminated for some reasons and
restarted
automatically, "application_version" remain unchanged.

As far as I see it, however, "application_version" is changed when I "cf
restart" my application i.e. CCNG should use the same droplet so there
are
no differences from the one before deployment.

If it is possible, could someone please tell me the intention about this
implementation?


Thanks.

Hiroaki UKAJI



--
View this message in context:
http://cf-dev.70369.x6.nabble.com/cf-dev-application-version-is-changed-although-source-code-is-not-changed-tp2262.html
Sent from the CF Dev mailing list archive at Nabble.com.

--
Matthew Sykes
matthew.sykes(a)gmail.com


Re: "application_version" is changed although source code is not changed.

Gerhard
 

I was also assuming that app version would not change between app restarts
(when the same droplet is used). I was wrong.

Would it make more sense if the app version was the droplet shasum?

On Thursday, 15 October 2015, CF Runtime <cfruntime(a)gmail.com> wrote:

application_version is mostly an internal cloud foundry concern. The DEAs
broadcast the application version they are running, and the health manager
uses that with what version it expects to be running to converge the system
on the desired state.

Restarting the app is supposed to terminate the old instances and start
the new ones, so they new ones get a new application version so they are
separate from the old.

Joseph
CF Release Integration Team

On Thu, Oct 15, 2015 at 2:00 AM, Hiroaki Ukaji <dt3snow.w(a)gmail.com
<javascript:_e(%7B%7D,'cvml','dt3snow.w(a)gmail.com');>> wrote:

Hi.
I have a question about a json field "application_version" in an
environment
variable "VCAP_APPLICATION".

By intuition, "application_version" should be only changed when there is
some update for an application.
Actually, when an application is terminated for some reasons and restarted
automatically, "application_version" remain unchanged.

As far as I see it, however, "application_version" is changed when I "cf
restart" my application i.e. CCNG should use the same droplet so there are
no differences from the one before deployment.

If it is possible, could someone please tell me the intention about this
implementation?


Thanks.

Hiroaki UKAJI



--
View this message in context:
http://cf-dev.70369.x6.nabble.com/cf-dev-application-version-is-changed-although-source-code-is-not-changed-tp2262.html
Sent from the CF Dev mailing list archive at Nabble.com.


Re: diego -- apps usage/view container placement?

Eric Malm <emalm@...>
 

Hi, Tom,

Thanks for asking. The Diego team has a semi-supported tool called
'veritas' (at https://github.com/pivotal-cf-experimental/veritas) that can
give similar sorts of diagnostics on the command line, especially with its
'distribution' and 'dump-store' commands. It's easiest to run from inside
the BOSH deployment, as it makes requests directly to the BBS API. It is
possible, though, to make requests from outside the deployment by setting
up an SSH tunnel through, say, your BOSH or microBOSH VM to the BBS leader.
We've also recently made veritas capable of communicating with the BBS even
if it's protected by mutual SSL authentication, although in those cases
you'll also need to provide a suitable SSL certificate/key pair for veritas
to present as the client.

The Diego team will soon remake veritas as a more thoroughly supported and
tested operational tool that we could package with diego-release. Operators
will then be able to use it to get a high-level overview of the entire
deployment, to extract LRPs and Tasks as JSON for machine processing, to
inspect the state reported by the cell reps or the Garden containers on
each cell, to discover which instances of the Diego services hold their
distributed locks, and to inspect the Golang runtime state of components,
among other things. Many of these operations are already present in veritas
today, but they're not always well-documented or as polished as they could
be. If you have particular suggestions for other features you'd find
useful, we'd like to hear them!

Best,
Eric, CF Runtime Diego PM

On Thu, Oct 15, 2015 at 7:59 AM, Tom Sherrod <tom.sherrod(a)gmail.com> wrote:

Is there an equivalent of xray or ltc visualize for diego enable CF
environment?

Thanks,
Tom


Re: [abacus] Usage submission authorization

Saravanakumar A. Srinivasan
 

> what will be the scope for securing internal Abacus pipeline that Assk describes as system token ? 

It is 'abacus.usage.write'.

Updated my previous statements to make it more specific:

We have enabled scope based authorization for REST endpoints at usage collector and usage reporting service. While we are working on using system OAuth bearer access token at internal Abacus pipeline, Submitting usage to a secured Abacus needs a OAuth bearer access token with 'abacus.usage.write' system scope in addition to the resource provider specific scope(s) - 'abacus.usage.<resource_id>.write'.

Thanks,
Saravanakumar Srinivasan (Assk),


-----Piotr Przybylski/Burlingame/IBM@IBMUS wrote: -----
To: cf-dev@...
From: Piotr Przybylski/Burlingame/IBM@IBMUS
Date: 10/15/2015 09:50PM
Subject: [cf-dev] Re: Re: Re: Re: Re: Re: [cf-dev][abacus] Usage submission authorization

Makes sense, and just to complete - what will be the scope for securing internal Abacus pipeline that Assk describes as system token ? 

Piotr
 
 

----- Original message -----
From: Jean-Sebastien Delfino <jsdelfino@...>
To: "Discussions about Cloud Foundry projects and the system overall." <cf-dev@...>
Cc:
Subject: [cf-dev] Re: Re: Re: Re: Re: [cf-dev][abacus] Usage submission authorization
Date: Thu, Oct 15, 2015 9:11 PM
 
Hey Piotr,
 
To read usage I believe you'll need 'abacus.usage.read', as 'abacus.usage.write' is for, well... writing.
 
P.S. That reminds me of a period of my life long time ago when I was a contractor for some big company and they had hired me to write code for them but had not given me the authorization to read the confidential code I was writing :)
 
- Jean-Sebastien
 
On Thu, Oct 15, 2015 at 7:28 PM, Piotr Przybylski <piotrp@...> wrote:
Assk,
can you confirm that the same scope (abacus.usage.write) is sufficient to retrieve usage ? 

Piotr
 
 
----- Original message -----
From: Saravanakumar A Srinivasan/Burlingame/IBM@IBMUS
To: <cf-dev@...>
Cc:
Subject: [cf-dev] Re: Re: Re: [cf-dev][abacus] Usage submission authorization
Date: Thu, Oct 15, 2015 7:06 PM
 
We have enabled scope based authorization for REST endpoints at usage collector and usage reporting service. While we are working on using system OAuth bearer access token at internal Abacus pipeline, Submitting usage to a secured Abacus needs a OAuth bearer access token with 'abacus.usage.write' scope.

Thanks,
Saravanakumar Srinivasan (Assk),

Bay Area Lab, 1001, E Hillsdale Blvd, Ste 400, Foster City, CA - 94404.
E-mail: sasrin@...
Phone: 650 645 8251 (T/L 367-8251)


-----Jean-Sebastien Delfino <jsdelfino@...> wrote: -----
To: "Discussions about Cloud Foundry projects and the system overall." <cf-dev@...>
From: Jean-Sebastien Delfino <jsdelfino@...>
Date: 10/12/2015 07:50PM
Subject: [cf-dev] Re: Re: [cf-dev][abacus] Usage submission authorization

 
> Also, resource id is an arbitrary identifier, making it part of the scope may create quite complex names e.g. 'abacus.runtimes/node/v12-07.revision-2-buildpack-guid-a3d7ff4d-3cb1-4cc3-a855-fae98e20cf57.write.
 
Do you have a specific issue in mind with putting the resource uuid in the scope name? We have uuids all over the place in CF, in most of the APIs, the usage docs etc so I'm not sure why it'd be a problem to have one here.
 
> Any naming convention may not be generic enough, for example for my UAA instance requires the scope names to start with component using it, followed by proper name - 'bss.runtimes.abacus.<resource id>.write'.
 
Like I said before, if you can't or don't want to use a specific scope per resource, then you can use abacus.usage.write (with the same disclaimers/warnings I gave in my previous post.)
 
I must be missing something though :) ... aren't you happily using cloud_controller.write for example (or similar other CF scopes) without renaming it to <your client component>.cloud_controller.write? Why would you treat abacus.usage.write different?
 
Also, I must admit to find a bit surprising a naming convention that will tie the scope name to the client that presents it. Isn't the scope typically defined by the owner of the resource it protects instead of the client? In that case the owner of the resource is not the client component... it is the CF abacus project, hence <abacus>.usage.write. Wouldn't that make more sense?
 
Finally, I'm also not quite sure how this will work at all if for example Abacus needs to authorize resource access from multiple clients. That would have to be really dynamic then, as each new client would require Abacus to know about a new client specific naming convention (or client component name prefix in the example you gave...)
 
Now, all that being said, looks like I'm not really following how you're envisioning this to work, so do you think you could maybe submit a pull request with how you concretely propose to make that dynamic scope naming work when it includes client component names, or follows client component specific naming conventions?
 
Thanks!
 
- Jean-Sebastien
 
On Mon, Oct 12, 2015 at 5:22 PM, Piotr Przybylski <piotrp@...> wrote:

Hi Sebastien,
I am not sure why allowing resource provider to explicitly specify scope with which particular resource usage will be submitted is a problem. Just allowing to pick a name would not compromise submission security in any way. It could be done for example by adding scope name to the resource definition.

Any naming convention may not be generic enough, for example for my UAA instance requires the scope names to start with component using it, followed by proper name - 'bss.runtimes.abacus.<resource id>.write'. Also, resource id is an arbitrary identifier, making it part of the scope may create quite complex names e.g.
'abacus.runtimes/node/v12-07.revision-2-buildpack-guid-a3d7ff4d-3cb1-4cc3-a855-fae98e20cf57.write.

Piotr



Jean-Sebastien Delfino ---10/09/2015 09:38:09 PM---Hey Piotr, >>> In some cases it may not be possible or viable to create new scope for

From: Jean-Sebastien Delfino <jsdelfino@...>
To: "Discussions about Cloud Foundry projects and the system overall." <cf-dev@...>

Date: 10/09/2015 09:38 PM
Subject: [cf-dev] Re: Re: Re: Re: Re: [abacus] Usage submission authorization

 



Hey Piotr,

>>> In some cases it may not be possible or viable to create new scope for each resource id e.g. short lived resources.

>> Why wouldn't that be possible? What type of short-lived resources did you have in mind?

> For example experimental service version (beta) replaced by release version, usage of which may be reported and metered but not necessarily billed. 

OK, that use case makes sense to me. So, your resource is going to be available for a few hours or days. I'm assuming that to get it on board CF and meter it with Abacus you're going to run a cf create-service-broker command or cf update-service-broker, define the resource config specifying how to meter it, and store that config where your Abacus provisioning endpoint implementation can retrieve it.

To secure the submission of usage for it, if I understand correctly how UAA works, you'll then need to do this:
uaac client update <your service provider's client id> --authorities "... existing permissions... abacus.<your resource id>.write"

That's all...

If that's really too much of a burden (really?) compared to the other steps, you're basically looking to do *nothing* to secure that resource. You could just submit usage with the abacus.usage.write scope, but that's the equivalent of the CF cloud_controller.write scope for Abacus, close to all powers... I'd probably advise against it as that's a serious risk but that may be what you're looking for.

> The scope names may need to follow adopter specific conventions so creating scope with predefined name 'abacus.usage....' may not fit that scheme. Abacus should offer ability to adjust the scope names, otherwise submission may not be possible.

These are simple names that we expect in the token used to submit usage. They're just constants like the names of our APIs, parameters, options, fields in our JSON schemas... basically the contract/interface between the Abacus user and its implementation. Not sure if there's a specific issue with that abacus naming convention or if it's just a theoretical question, but I'll be happy to discuss alternate naming conventions:

Do you have another naming convention in mind that you'd like to use?

Is there a specific issue with abacus.usage.write? Is the 'abacus' part in the name a problem?

Would you prefer to submit usage with an existing CF scope like cloud_controller.write or another of these high power scopes?
(again, I'd advise against it though...)

- Jean-Sebastien

- Jean-Sebastien

On Thu, Oct 8, 2015 at 5:24 PM, Piotr Przybylski <piotrp@...> wrote:
  • Hi Sebastien,

    >> In some cases it may not be possible or viable to create new scope for each resource id e.g. short lived resources.

    >Why wouldn't that be possible? What type of short-lived resources did you have in mind?

    For example experimental service version (beta) replaced by release version, usage of which may be reported and metered but not necessarily billed.
    The scope names may need to follow adopter specific conventions so creating scope with predefined name '
    abacus.usage....' may not fit that scheme. Abacus should offer ability to adjust the scope names, otherwise submission may not be possible.


    > Another reason why I'm not sure about short lived resources, is that although you may decide to stop offering a type a resource at some point, once you've metered it, and sent a bill for it >to a customer, I don't think you can really 'forget' about its existence anymore... So in that sense I'm not sure how it can be 'short lived'.
    The short lived resource is only for submission, once it is not offered, its specific scope is not needed. Thad does not mean erasing history of usage.


    Piotr




    Jean-Sebastien Delfino ---10/08/2015 11:10:16 AM---Hi Piotr, > In some cases it may not be possible or viable to create new scope for

    From: Jean-Sebastien Delfino <jsdelfino@...>
    To: "Discussions about Cloud Foundry projects and the system overall." <cf-dev@...>
    Date: 10/08/2015 11:10 AM
    Subject: [cf-dev] Re: Re: Re: [abacus] Usage submission authorization

     





    Hi Piotr,

    > In some cases it may not be possible or viable to create new scope for each resource id e.g. short lived resources.

    Why wouldn't that be possible? What type of short-lived resources did you have in mind?

    The typical use case I've seen is for a Cloud platform to decide to offer a new type of database or analytics or messaging service, or a new type of runtime for example. Before that new resource is offered on the platform, their resource provider needs to get on board, get a user id, auth credentials defined in UAA etc... You probably also need to define how you're going to meter that new resource and the pricing for it.

    Couldn't a scope be created in UAA at that time along all these other on boarding steps?

    Another reason why I'm not sure about short lived resources, is that although you may decide to stop offering a type a resource at some point, once you've metered it, and sent a bill for it to a customer, I don't think you can really 'forget' about its existence anymore... So in that sense I'm not sure how it can be 'short lived'.

    > Some flexibility would also help to accommodate changes related to grouping resources by type as discussed in [1].

    We discussed two options in [1]:
    a) support a resource_type in addition to resource_id for grouping many resource_ids under a single type
    b) a common resource_id for several resources (something like 'node' for all your versions of Node.js build packs for example) 

    Since option (a) is not implemented at this point and Issue #38 is actually assigned to a 'future' milestone, AIUI resource providers need to use option (b) with a common resource_id for multiple resources. Is creating a scope for that common id still too much of a burden then?

    [1] - https://github.com/cloudfoundry-incubator/cf-abacus/issues/38

    Thoughts?

    - Jean-Sebastien

    On Wed, Oct 7, 2015 at 5:51 PM, Piotr Przybylski <piotrp@...> wrote:
      • Hi Sebastien,

        > That OAuth token should include:
        > - a user id uniquely identifying that resource provider;
        > - an OAuth scope named like abacus.usage.<resource_id>.write

        What kind of customization of the above do you plan to expose? In some cases it may not be possible or viable to create new scope for each resource id e.g. short lived resources. The ability to either configure scope to use for validation or provide scope 'mapping' would help to adapt it to existing deployments. Some flexibility would also help to accommodate changes related to grouping resources by type as discussed in [1].

        [1] - https://github.com/cloudfoundry-incubator/cf-abacus/issues/38


        Piotr



        Jean-Sebastien Delfino ---10/07/2015 12:30:05 AM---Hi Piotr, > what kind of authorization is required to submit usage to Abacus ?

        From: Jean-Sebastien Delfino <jsdelfino@...>
        To: "Discussions about Cloud Foundry projects and the system overall." <cf-dev@...>
        Date: 10/07/2015 12:30 AM
        Subject: [cf-dev] Re: [abacus] Usage submission authorization





        Hi Piotr,

        > what kind of authorization is required to submit usage to Abacus ? 
        > Is the oauth token used for submission [1] required to have particular scope, specific to resource or resource provider ?

        A resource provider is expected to present an OAuth token with the usage it submits for a (service or runtime) resource.

        That OAuth token should include:
        - a user id uniquely identifying that resource provider;
        - an OAuth scope named like abacus.usage.<resource_id>.write.

        The precise naming syntax for that scope may still evolve in the next few days as we progress with the implementation of user story 101703426 [1].

        > Is there a different scope required to submit runtimes usage (like cf bridge) versus other services or its possible to use single scope for all the submissions

        I'd like to handle runtimes and services consistently as they're basically just different types of resources, i.e. one scope per 'service' resource, one scope per 'runtime' resource.

        We're still working on the detailed design and implementation, but I'm not sure we'd want to share scopes across (service and runtime) resource providers as that'd allow a resource provider to submit usage for resources owned by another...

        @assk / @sasrin, anything I missed? Thoughts?

        -- Jean-Sebastien


        On Tue, Oct 6, 2015 at 6:29 PM, Piotr Przybylski <piotrp@...> wrote:
              • Hi,
                what kind of authorization is required to submit usage to Abacus ?
                Is the oauth token used for submission [1] required to have particular scope, specific to resource or resource provider ? Is there a different scope required to submit runtimes usage (like cf bridge) versus other services or its possible to use single scope for all the submissions ?



                [1] - https://www.pivotaltracker.com/story/show/101703426

                Piotr

     

     

 

 

 




Re: [abacus] Usage submission authorization

Piotr Przybylski <piotrp@...>
 

Makes sense, and just to complete - what will be the scope for securing internal Abacus pipeline that Assk describes as system token ? 

Piotr
 
 

----- Original message -----
From: Jean-Sebastien Delfino <jsdelfino@...>
To: "Discussions about Cloud Foundry projects and the system overall." <cf-dev@...>
Cc:
Subject: [cf-dev] Re: Re: Re: Re: Re: [cf-dev][abacus] Usage submission authorization
Date: Thu, Oct 15, 2015 9:11 PM
 
Hey Piotr,
 
To read usage I believe you'll need 'abacus.usage.read', as 'abacus.usage.write' is for, well... writing.
 
P.S. That reminds me of a period of my life long time ago when I was a contractor for some big company and they had hired me to write code for them but had not given me the authorization to read the confidential code I was writing :)
 
- Jean-Sebastien
 
On Thu, Oct 15, 2015 at 7:28 PM, Piotr Przybylski <piotrp@...> wrote:
Assk,
can you confirm that the same scope (abacus.usage.write) is sufficient to retrieve usage ? 

Piotr
 
 
----- Original message -----
From: Saravanakumar A Srinivasan/Burlingame/IBM@IBMUS
To: <cf-dev@...>
Cc:
Subject: [cf-dev] Re: Re: Re: [cf-dev][abacus] Usage submission authorization
Date: Thu, Oct 15, 2015 7:06 PM
 
We have enabled scope based authorization for REST endpoints at usage collector and usage reporting service. While we are working on using system OAuth bearer access token at internal Abacus pipeline, Submitting usage to a secured Abacus needs a OAuth bearer access token with 'abacus.usage.write' scope.

Thanks,
Saravanakumar Srinivasan (Assk),

Bay Area Lab, 1001, E Hillsdale Blvd, Ste 400, Foster City, CA - 94404.
E-mail: sasrin@...
Phone: 650 645 8251 (T/L 367-8251)


-----Jean-Sebastien Delfino <jsdelfino@...> wrote: -----
To: "Discussions about Cloud Foundry projects and the system overall." <cf-dev@...>
From: Jean-Sebastien Delfino <jsdelfino@...>
Date: 10/12/2015 07:50PM
Subject: [cf-dev] Re: Re: [cf-dev][abacus] Usage submission authorization

 
> Also, resource id is an arbitrary identifier, making it part of the scope may create quite complex names e.g. 'abacus.runtimes/node/v12-07.revision-2-buildpack-guid-a3d7ff4d-3cb1-4cc3-a855-fae98e20cf57.write.
 
Do you have a specific issue in mind with putting the resource uuid in the scope name? We have uuids all over the place in CF, in most of the APIs, the usage docs etc so I'm not sure why it'd be a problem to have one here.
 
> Any naming convention may not be generic enough, for example for my UAA instance requires the scope names to start with component using it, followed by proper name - 'bss.runtimes.abacus.<resource id>.write'.
 
Like I said before, if you can't or don't want to use a specific scope per resource, then you can use abacus.usage.write (with the same disclaimers/warnings I gave in my previous post.)
 
I must be missing something though :) ... aren't you happily using cloud_controller.write for example (or similar other CF scopes) without renaming it to <your client component>.cloud_controller.write? Why would you treat abacus.usage.write different?
 
Also, I must admit to find a bit surprising a naming convention that will tie the scope name to the client that presents it. Isn't the scope typically defined by the owner of the resource it protects instead of the client? In that case the owner of the resource is not the client component... it is the CF abacus project, hence <abacus>.usage.write. Wouldn't that make more sense?
 
Finally, I'm also not quite sure how this will work at all if for example Abacus needs to authorize resource access from multiple clients. That would have to be really dynamic then, as each new client would require Abacus to know about a new client specific naming convention (or client component name prefix in the example you gave...)
 
Now, all that being said, looks like I'm not really following how you're envisioning this to work, so do you think you could maybe submit a pull request with how you concretely propose to make that dynamic scope naming work when it includes client component names, or follows client component specific naming conventions?
 
Thanks!
 
- Jean-Sebastien
 
On Mon, Oct 12, 2015 at 5:22 PM, Piotr Przybylski <piotrp@...> wrote:

Hi Sebastien,
I am not sure why allowing resource provider to explicitly specify scope with which particular resource usage will be submitted is a problem. Just allowing to pick a name would not compromise submission security in any way. It could be done for example by adding scope name to the resource definition.

Any naming convention may not be generic enough, for example for my UAA instance requires the scope names to start with component using it, followed by proper name - 'bss.runtimes.abacus.<resource id>.write'. Also, resource id is an arbitrary identifier, making it part of the scope may create quite complex names e.g.
'abacus.runtimes/node/v12-07.revision-2-buildpack-guid-a3d7ff4d-3cb1-4cc3-a855-fae98e20cf57.write.

Piotr



Jean-Sebastien Delfino ---10/09/2015 09:38:09 PM---Hey Piotr, >>> In some cases it may not be possible or viable to create new scope for

From: Jean-Sebastien Delfino <jsdelfino@...>
To: "Discussions about Cloud Foundry projects and the system overall." <cf-dev@...>

Date: 10/09/2015 09:38 PM
Subject: [cf-dev] Re: Re: Re: Re: Re: [abacus] Usage submission authorization

 



Hey Piotr,

>>> In some cases it may not be possible or viable to create new scope for each resource id e.g. short lived resources.

>> Why wouldn't that be possible? What type of short-lived resources did you have in mind?

> For example experimental service version (beta) replaced by release version, usage of which may be reported and metered but not necessarily billed. 

OK, that use case makes sense to me. So, your resource is going to be available for a few hours or days. I'm assuming that to get it on board CF and meter it with Abacus you're going to run a cf create-service-broker command or cf update-service-broker, define the resource config specifying how to meter it, and store that config where your Abacus provisioning endpoint implementation can retrieve it.

To secure the submission of usage for it, if I understand correctly how UAA works, you'll then need to do this:
uaac client update <your service provider's client id> --authorities "... existing permissions... abacus.<your resource id>.write"

That's all...

If that's really too much of a burden (really?) compared to the other steps, you're basically looking to do *nothing* to secure that resource. You could just submit usage with the abacus.usage.write scope, but that's the equivalent of the CF cloud_controller.write scope for Abacus, close to all powers... I'd probably advise against it as that's a serious risk but that may be what you're looking for.

> The scope names may need to follow adopter specific conventions so creating scope with predefined name 'abacus.usage....' may not fit that scheme. Abacus should offer ability to adjust the scope names, otherwise submission may not be possible.

These are simple names that we expect in the token used to submit usage. They're just constants like the names of our APIs, parameters, options, fields in our JSON schemas... basically the contract/interface between the Abacus user and its implementation. Not sure if there's a specific issue with that abacus naming convention or if it's just a theoretical question, but I'll be happy to discuss alternate naming conventions:

Do you have another naming convention in mind that you'd like to use?

Is there a specific issue with abacus.usage.write? Is the 'abacus' part in the name a problem?

Would you prefer to submit usage with an existing CF scope like cloud_controller.write or another of these high power scopes?
(again, I'd advise against it though...)

- Jean-Sebastien

- Jean-Sebastien

On Thu, Oct 8, 2015 at 5:24 PM, Piotr Przybylski <piotrp@...> wrote:
  • Hi Sebastien,

    >> In some cases it may not be possible or viable to create new scope for each resource id e.g. short lived resources.

    >Why wouldn't that be possible? What type of short-lived resources did you have in mind?

    For example experimental service version (beta) replaced by release version, usage of which may be reported and metered but not necessarily billed.
    The scope names may need to follow adopter specific conventions so creating scope with predefined name '
    abacus.usage....' may not fit that scheme. Abacus should offer ability to adjust the scope names, otherwise submission may not be possible.


    > Another reason why I'm not sure about short lived resources, is that although you may decide to stop offering a type a resource at some point, once you've metered it, and sent a bill for it >to a customer, I don't think you can really 'forget' about its existence anymore... So in that sense I'm not sure how it can be 'short lived'.
    The short lived resource is only for submission, once it is not offered, its specific scope is not needed. Thad does not mean erasing history of usage.


    Piotr




    Jean-Sebastien Delfino ---10/08/2015 11:10:16 AM---Hi Piotr, > In some cases it may not be possible or viable to create new scope for

    From: Jean-Sebastien Delfino <jsdelfino@...>
    To: "Discussions about Cloud Foundry projects and the system overall." <cf-dev@...>
    Date: 10/08/2015 11:10 AM
    Subject: [cf-dev] Re: Re: Re: [abacus] Usage submission authorization

     





    Hi Piotr,

    > In some cases it may not be possible or viable to create new scope for each resource id e.g. short lived resources.

    Why wouldn't that be possible? What type of short-lived resources did you have in mind?

    The typical use case I've seen is for a Cloud platform to decide to offer a new type of database or analytics or messaging service, or a new type of runtime for example. Before that new resource is offered on the platform, their resource provider needs to get on board, get a user id, auth credentials defined in UAA etc... You probably also need to define how you're going to meter that new resource and the pricing for it.

    Couldn't a scope be created in UAA at that time along all these other on boarding steps?

    Another reason why I'm not sure about short lived resources, is that although you may decide to stop offering a type a resource at some point, once you've metered it, and sent a bill for it to a customer, I don't think you can really 'forget' about its existence anymore... So in that sense I'm not sure how it can be 'short lived'.

    > Some flexibility would also help to accommodate changes related to grouping resources by type as discussed in [1].

    We discussed two options in [1]:
    a) support a resource_type in addition to resource_id for grouping many resource_ids under a single type
    b) a common resource_id for several resources (something like 'node' for all your versions of Node.js build packs for example) 

    Since option (a) is not implemented at this point and Issue #38 is actually assigned to a 'future' milestone, AIUI resource providers need to use option (b) with a common resource_id for multiple resources. Is creating a scope for that common id still too much of a burden then?

    [1] - https://github.com/cloudfoundry-incubator/cf-abacus/issues/38

    Thoughts?

    - Jean-Sebastien

    On Wed, Oct 7, 2015 at 5:51 PM, Piotr Przybylski <piotrp@...> wrote:
      • Hi Sebastien,

        > That OAuth token should include:
        > - a user id uniquely identifying that resource provider;
        > - an OAuth scope named like abacus.usage.<resource_id>.write

        What kind of customization of the above do you plan to expose? In some cases it may not be possible or viable to create new scope for each resource id e.g. short lived resources. The ability to either configure scope to use for validation or provide scope 'mapping' would help to adapt it to existing deployments. Some flexibility would also help to accommodate changes related to grouping resources by type as discussed in [1].

        [1] - https://github.com/cloudfoundry-incubator/cf-abacus/issues/38


        Piotr



        Jean-Sebastien Delfino ---10/07/2015 12:30:05 AM---Hi Piotr, > what kind of authorization is required to submit usage to Abacus ?

        From: Jean-Sebastien Delfino <jsdelfino@...>
        To: "Discussions about Cloud Foundry projects and the system overall." <cf-dev@...>
        Date: 10/07/2015 12:30 AM
        Subject: [cf-dev] Re: [abacus] Usage submission authorization





        Hi Piotr,

        > what kind of authorization is required to submit usage to Abacus ? 
        > Is the oauth token used for submission [1] required to have particular scope, specific to resource or resource provider ?

        A resource provider is expected to present an OAuth token with the usage it submits for a (service or runtime) resource.

        That OAuth token should include:
        - a user id uniquely identifying that resource provider;
        - an OAuth scope named like abacus.usage.<resource_id>.write.

        The precise naming syntax for that scope may still evolve in the next few days as we progress with the implementation of user story 101703426 [1].

        > Is there a different scope required to submit runtimes usage (like cf bridge) versus other services or its possible to use single scope for all the submissions

        I'd like to handle runtimes and services consistently as they're basically just different types of resources, i.e. one scope per 'service' resource, one scope per 'runtime' resource.

        We're still working on the detailed design and implementation, but I'm not sure we'd want to share scopes across (service and runtime) resource providers as that'd allow a resource provider to submit usage for resources owned by another...

        @assk / @sasrin, anything I missed? Thoughts?

        -- Jean-Sebastien


        On Tue, Oct 6, 2015 at 6:29 PM, Piotr Przybylski <piotrp@...> wrote:
              • Hi,
                what kind of authorization is required to submit usage to Abacus ?
                Is the oauth token used for submission [1] required to have particular scope, specific to resource or resource provider ? Is there a different scope required to submit runtimes usage (like cf bridge) versus other services or its possible to use single scope for all the submissions ?



                [1] - https://www.pivotaltracker.com/story/show/101703426

                Piotr

     

     

 

 

 



Re: [abacus] Usage submission authorization

Saravanakumar A. Srinivasan
 


Since usage reporting service is reporting usage for an account, for an organization or for a set of organizations, Abacus delegates the authorization to account and organization information resource server - account stub. For more details, refer to the discussion at [1].


Thanks,
Saravanakumar Srinivasan (Assk),


-----Piotr Przybylski/Burlingame/IBM@IBMUS wrote: -----
To: cf-dev@...
From: Piotr Przybylski/Burlingame/IBM@IBMUS
Date: 10/15/2015 07:28PM
Subject: [cf-dev] Re: Re: Re: Re: [cf-dev][abacus] Usage submission authorization

Assk,
can you confirm that the same scope (abacus.usage.write) is sufficient to retrieve usage ? 

Piotr
 
 

----- Original message -----
From: Saravanakumar A Srinivasan/Burlingame/IBM@IBMUS
To: <cf-dev@...>
Cc:
Subject: [cf-dev] Re: Re: Re: [cf-dev][abacus] Usage submission authorization
Date: Thu, Oct 15, 2015 7:06 PM
 
We have enabled scope based authorization for REST endpoints at usage collector and usage reporting service. While we are working on using system OAuth bearer access token at internal Abacus pipeline, Submitting usage to a secured Abacus needs a OAuth bearer access token with 'abacus.usage.write' scope.

Thanks,
Saravanakumar Srinivasan (Assk),

Bay Area Lab, 1001, E Hillsdale Blvd, Ste 400, Foster City, CA - 94404.
E-mail: sasrin@...
Phone: 650 645 8251 (T/L 367-8251)


-----Jean-Sebastien Delfino <jsdelfino@...> wrote: -----
To: "Discussions about Cloud Foundry projects and the system overall." <cf-dev@...>
From: Jean-Sebastien Delfino <jsdelfino@...>
Date: 10/12/2015 07:50PM
Subject: [cf-dev] Re: Re: [cf-dev][abacus] Usage submission authorization

 
> Also, resource id is an arbitrary identifier, making it part of the scope may create quite complex names e.g. 'abacus.runtimes/node/v12-07.revision-2-buildpack-guid-a3d7ff4d-3cb1-4cc3-a855-fae98e20cf57.write.
 
Do you have a specific issue in mind with putting the resource uuid in the scope name? We have uuids all over the place in CF, in most of the APIs, the usage docs etc so I'm not sure why it'd be a problem to have one here.
 
> Any naming convention may not be generic enough, for example for my UAA instance requires the scope names to start with component using it, followed by proper name - 'bss.runtimes.abacus.<resource id>.write'.
 
Like I said before, if you can't or don't want to use a specific scope per resource, then you can use abacus.usage.write (with the same disclaimers/warnings I gave in my previous post.)
 
I must be missing something though :) ... aren't you happily using cloud_controller.write for example (or similar other CF scopes) without renaming it to <your client component>.cloud_controller.write? Why would you treat abacus.usage.write different?
 
Also, I must admit to find a bit surprising a naming convention that will tie the scope name to the client that presents it. Isn't the scope typically defined by the owner of the resource it protects instead of the client? In that case the owner of the resource is not the client component... it is the CF abacus project, hence <abacus>.usage.write. Wouldn't that make more sense?
 
Finally, I'm also not quite sure how this will work at all if for example Abacus needs to authorize resource access from multiple clients. That would have to be really dynamic then, as each new client would require Abacus to know about a new client specific naming convention (or client component name prefix in the example you gave...)
 
Now, all that being said, looks like I'm not really following how you're envisioning this to work, so do you think you could maybe submit a pull request with how you concretely propose to make that dynamic scope naming work when it includes client component names, or follows client component specific naming conventions?
 
Thanks!
 
- Jean-Sebastien
 
On Mon, Oct 12, 2015 at 5:22 PM, Piotr Przybylski <piotrp@...> wrote:

Hi Sebastien,
I am not sure why allowing resource provider to explicitly specify scope with which particular resource usage will be submitted is a problem. Just allowing to pick a name would not compromise submission security in any way. It could be done for example by adding scope name to the resource definition.

Any naming convention may not be generic enough, for example for my UAA instance requires the scope names to start with component using it, followed by proper name - 'bss.runtimes.abacus.<resource id>.write'. Also, resource id is an arbitrary identifier, making it part of the scope may create quite complex names e.g.
'abacus.runtimes/node/v12-07.revision-2-buildpack-guid-a3d7ff4d-3cb1-4cc3-a855-fae98e20cf57.write.

Piotr



Jean-Sebastien Delfino ---10/09/2015 09:38:09 PM---Hey Piotr, >>> In some cases it may not be possible or viable to create new scope for

From: Jean-Sebastien Delfino <jsdelfino@...>
To: "Discussions about Cloud Foundry projects and the system overall." <cf-dev@...>

Date: 10/09/2015 09:38 PM
Subject: [cf-dev] Re: Re: Re: Re: Re: [abacus] Usage submission authorization

 



Hey Piotr,

>>> In some cases it may not be possible or viable to create new scope for each resource id e.g. short lived resources.

>> Why wouldn't that be possible? What type of short-lived resources did you have in mind?

> For example experimental service version (beta) replaced by release version, usage of which may be reported and metered but not necessarily billed. 

OK, that use case makes sense to me. So, your resource is going to be available for a few hours or days. I'm assuming that to get it on board CF and meter it with Abacus you're going to run a cf create-service-broker command or cf update-service-broker, define the resource config specifying how to meter it, and store that config where your Abacus provisioning endpoint implementation can retrieve it.

To secure the submission of usage for it, if I understand correctly how UAA works, you'll then need to do this:
uaac client update <your service provider's client id> --authorities "... existing permissions... abacus.<your resource id>.write"

That's all...

If that's really too much of a burden (really?) compared to the other steps, you're basically looking to do *nothing* to secure that resource. You could just submit usage with the abacus.usage.write scope, but that's the equivalent of the CF cloud_controller.write scope for Abacus, close to all powers... I'd probably advise against it as that's a serious risk but that may be what you're looking for.

> The scope names may need to follow adopter specific conventions so creating scope with predefined name 'abacus.usage....' may not fit that scheme. Abacus should offer ability to adjust the scope names, otherwise submission may not be possible.

These are simple names that we expect in the token used to submit usage. They're just constants like the names of our APIs, parameters, options, fields in our JSON schemas... basically the contract/interface between the Abacus user and its implementation. Not sure if there's a specific issue with that abacus naming convention or if it's just a theoretical question, but I'll be happy to discuss alternate naming conventions:

Do you have another naming convention in mind that you'd like to use?

Is there a specific issue with abacus.usage.write? Is the 'abacus' part in the name a problem?

Would you prefer to submit usage with an existing CF scope like cloud_controller.write or another of these high power scopes?
(again, I'd advise against it though...)

- Jean-Sebastien

- Jean-Sebastien

On Thu, Oct 8, 2015 at 5:24 PM, Piotr Przybylski <piotrp@...> wrote:
  • Hi Sebastien,

    >> In some cases it may not be possible or viable to create new scope for each resource id e.g. short lived resources.

    >Why wouldn't that be possible? What type of short-lived resources did you have in mind?

    For example experimental service version (beta) replaced by release version, usage of which may be reported and metered but not necessarily billed.
    The scope names may need to follow adopter specific conventions so creating scope with predefined name '
    abacus.usage....' may not fit that scheme. Abacus should offer ability to adjust the scope names, otherwise submission may not be possible.


    > Another reason why I'm not sure about short lived resources, is that although you may decide to stop offering a type a resource at some point, once you've metered it, and sent a bill for it >to a customer, I don't think you can really 'forget' about its existence anymore... So in that sense I'm not sure how it can be 'short lived'.
    The short lived resource is only for submission, once it is not offered, its specific scope is not needed. Thad does not mean erasing history of usage.


    Piotr




    Jean-Sebastien Delfino ---10/08/2015 11:10:16 AM---Hi Piotr, > In some cases it may not be possible or viable to create new scope for

    From: Jean-Sebastien Delfino <jsdelfino@...>
    To: "Discussions about Cloud Foundry projects and the system overall." <cf-dev@...>
    Date: 10/08/2015 11:10 AM
    Subject: [cf-dev] Re: Re: Re: [abacus] Usage submission authorization

     





    Hi Piotr,

    > In some cases it may not be possible or viable to create new scope for each resource id e.g. short lived resources.

    Why wouldn't that be possible? What type of short-lived resources did you have in mind?

    The typical use case I've seen is for a Cloud platform to decide to offer a new type of database or analytics or messaging service, or a new type of runtime for example. Before that new resource is offered on the platform, their resource provider needs to get on board, get a user id, auth credentials defined in UAA etc... You probably also need to define how you're going to meter that new resource and the pricing for it.

    Couldn't a scope be created in UAA at that time along all these other on boarding steps?

    Another reason why I'm not sure about short lived resources, is that although you may decide to stop offering a type a resource at some point, once you've metered it, and sent a bill for it to a customer, I don't think you can really 'forget' about its existence anymore... So in that sense I'm not sure how it can be 'short lived'.

    > Some flexibility would also help to accommodate changes related to grouping resources by type as discussed in [1].

    We discussed two options in [1]:
    a) support a resource_type in addition to resource_id for grouping many resource_ids under a single type
    b) a common resource_id for several resources (something like 'node' for all your versions of Node.js build packs for example) 

    Since option (a) is not implemented at this point and Issue #38 is actually assigned to a 'future' milestone, AIUI resource providers need to use option (b) with a common resource_id for multiple resources. Is creating a scope for that common id still too much of a burden then?

    [1] - https://github.com/cloudfoundry-incubator/cf-abacus/issues/38

    Thoughts?

    - Jean-Sebastien

    On Wed, Oct 7, 2015 at 5:51 PM, Piotr Przybylski <piotrp@...> wrote:
      • Hi Sebastien,

        > That OAuth token should include:
        > - a user id uniquely identifying that resource provider;
        > - an OAuth scope named like abacus.usage.<resource_id>.write

        What kind of customization of the above do you plan to expose? In some cases it may not be possible or viable to create new scope for each resource id e.g. short lived resources. The ability to either configure scope to use for validation or provide scope 'mapping' would help to adapt it to existing deployments. Some flexibility would also help to accommodate changes related to grouping resources by type as discussed in [1].

        [1] - https://github.com/cloudfoundry-incubator/cf-abacus/issues/38


        Piotr



        Jean-Sebastien Delfino ---10/07/2015 12:30:05 AM---Hi Piotr, > what kind of authorization is required to submit usage to Abacus ?

        From: Jean-Sebastien Delfino <jsdelfino@...>
        To: "Discussions about Cloud Foundry projects and the system overall." <cf-dev@...>
        Date: 10/07/2015 12:30 AM
        Subject: [cf-dev] Re: [abacus] Usage submission authorization





        Hi Piotr,

        > what kind of authorization is required to submit usage to Abacus ? 
        > Is the oauth token used for submission [1] required to have particular scope, specific to resource or resource provider ?

        A resource provider is expected to present an OAuth token with the usage it submits for a (service or runtime) resource.

        That OAuth token should include:
        - a user id uniquely identifying that resource provider;
        - an OAuth scope named like abacus.usage.<resource_id>.write.

        The precise naming syntax for that scope may still evolve in the next few days as we progress with the implementation of user story 101703426 [1].

        > Is there a different scope required to submit runtimes usage (like cf bridge) versus other services or its possible to use single scope for all the submissions

        I'd like to handle runtimes and services consistently as they're basically just different types of resources, i.e. one scope per 'service' resource, one scope per 'runtime' resource.

        We're still working on the detailed design and implementation, but I'm not sure we'd want to share scopes across (service and runtime) resource providers as that'd allow a resource provider to submit usage for resources owned by another...

        @assk / @sasrin, anything I missed? Thoughts?

        -- Jean-Sebastien


        On Tue, Oct 6, 2015 at 6:29 PM, Piotr Przybylski <piotrp@...> wrote:
              • Hi,
                what kind of authorization is required to submit usage to Abacus ?
                Is the oauth token used for submission [1] required to have particular scope, specific to resource or resource provider ? Is there a different scope required to submit runtimes usage (like cf bridge) versus other services or its possible to use single scope for all the submissions ?



                [1] - https://www.pivotaltracker.com/story/show/101703426

                Piotr

     

     

 

 

 




Re: Open sourcing our RDS service broker

Gwenn Etourneau
 

Thanks that's a really nice job !

On Fri, Oct 16, 2015 at 1:15 PM, Eric Poelke <epoelke(a)gmail.com> wrote:

Throwing this out there for anyone who might find it useful.

https://github.com/epoelke/rds-service-broker


Open sourcing our RDS service broker

Eric Poelke
 

Throwing this out there for anyone who might find it useful.

https://github.com/epoelke/rds-service-broker


Re: "application_version" is changed although source code is not changed.

Hiroaki Ukaji <dt3snow.w@...>
 

Hi, thanks for your reply.

I understood that "application_version" is mainly used by CF components so I
cannot used it as an actual version of application.


Thanks.

Hiroaki UKAJI



--
View this message in context: http://cf-dev.70369.x6.nabble.com/cf-dev-application-version-is-changed-although-source-code-is-not-changed-tp2262p2284.html
Sent from the CF Dev mailing list archive at Nabble.com.


Re: [abacus] Usage submission authorization

Jean-Sebastien Delfino
 

Hey Piotr,

To read usage I believe you'll need 'abacus.usage.read', as
'abacus.usage.write' is for, well... writing.

P.S. That reminds me of a period of my life long time ago when I was a
contractor for some big company and they had hired me to write code for
them but had not given me the authorization to read the confidential code I
was writing :)

- Jean-Sebastien

On Thu, Oct 15, 2015 at 7:28 PM, Piotr Przybylski <piotrp(a)us.ibm.com> wrote:

Assk,
can you confirm that the same scope (abacus.usage.write) is sufficient to
retrieve usage ?

Piotr




----- Original message -----
From: Saravanakumar A Srinivasan/Burlingame/IBM(a)IBMUS
To: <cf-dev(a)lists.cloudfoundry.org>
Cc:
Subject: [cf-dev] Re: Re: Re: [cf-dev][abacus] Usage submission
authorization
Date: Thu, Oct 15, 2015 7:06 PM

We have enabled scope based authorization for REST endpoints at usage
collector and usage reporting service. While we are working on using system
OAuth bearer access token at internal Abacus pipeline, Submitting usage to
a secured Abacus needs a OAuth bearer access token with
'abacus.usage.write' scope.

Thanks,
Saravanakumar Srinivasan (Assk),

Bay Area Lab, 1001, E Hillsdale Blvd, Ste 400, Foster City, CA - 94404.
E-mail: sasrin(a)us.ibm.com
Phone: 650 645 8251 (T/L 367-8251)


-----Jean-Sebastien Delfino <jsdelfino(a)gmail.com> wrote: -----
To: "Discussions about Cloud Foundry projects and the system overall." <
cf-dev(a)lists.cloudfoundry.org>
From: Jean-Sebastien Delfino <jsdelfino(a)gmail.com>
Date: 10/12/2015 07:50PM
Subject: [cf-dev] Re: Re: [cf-dev][abacus] Usage submission authorization

Also, resource id is an arbitrary identifier, making it part of the
scope may create quite complex names e.g.
'abacus.runtimes/node/v12-07.revision-2-buildpack-guid-a3d7ff4d-3cb1-4cc3-a855-fae98e20cf57.write.

Do you have a specific issue in mind with putting the resource uuid in the
scope name? We have uuids all over the place in CF, in most of the APIs,
the usage docs etc so I'm not sure why it'd be a problem to have one here.

Any naming convention may not be generic enough, for example for my UAA
instance requires the scope names to start with component using it,
followed by proper name - 'bss.runtimes.abacus.<resource id>.write'.

Like I said before, if you can't or don't want to use a specific scope per
resource, then you can use abacus.usage.write (with the same
disclaimers/warnings I gave in my previous post.)

I must be missing something though :) ... aren't you happily using
cloud_controller.write for example (or similar other CF scopes) without
renaming it to <your client component>.cloud_controller.write? Why would
you treat abacus.usage.write different?

Also, I must admit to find a bit surprising a naming convention that will
tie the scope name to the client that presents it. Isn't the scope
typically defined by the owner of the resource it protects instead of the
client? In that case the owner of the resource is not the client
component... it is the CF abacus project, hence <abacus>.usage.write.
Wouldn't that make more sense?

Finally, I'm also not quite sure how this will work at all if for example
Abacus needs to authorize resource access from multiple clients. That would
have to be really dynamic then, as each new client would require Abacus to
know about a new client specific naming convention (or client component
name prefix in the example you gave...)

Now, all that being said, looks like I'm not really following how you're
envisioning this to work, so do you think you could maybe submit a pull
request with how you concretely propose to make that dynamic scope naming
work when it includes client component names, or follows client component
specific naming conventions?

Thanks!

- Jean-Sebastien

On Mon, Oct 12, 2015 at 5:22 PM, Piotr Przybylski <piotrp(a)us.ibm.com>
wrote:

Hi Sebastien,
I am not sure why allowing resource provider to explicitly specify scope
with which particular resource usage will be submitted is a problem. Just
allowing to pick a name would not compromise submission security in any
way. It could be done for example by adding scope name to the resource
definition.

Any naming convention may not be generic enough, for example for my UAA
instance requires the scope names to start with component using it,
followed by proper name - 'bss.runtimes.abacus.<resource id>.write'. Also,
resource id is an arbitrary identifier, making it part of the scope may
create quite complex names e.g.
'abacus.runtimes/node/v12-07.revision-2-buildpack-guid-a3d7ff4d-3cb1-4cc3-a855-fae98e20cf57.write.


Piotr



[image: Inactive hide details for Jean-Sebastien Delfino ---10/09/2015
09:38:09 PM---Hey Piotr, >>> In some cases it may not be possibl]Jean-Sebastien
Delfino ---10/09/2015 09:38:09 PM---Hey Piotr, >>> In some cases it may not
be possible or viable to create new scope for

From: Jean-Sebastien Delfino <jsdelfino(a)gmail.com>
To: "Discussions about Cloud Foundry projects and the system overall." <
cf-dev(a)lists.cloudfoundry.org>
Date: 10/09/2015 09:38 PM
Subject: [cf-dev] Re: Re: Re: Re: Re: [abacus] Usage submission
authorization

------------------------------



Hey Piotr,

In some cases it may not be possible or viable to create new scope for
each resource id e.g. short lived resources.

Why wouldn't that be possible? What type of short-lived resources did
you have in mind?

For example experimental service version (beta) replaced by release
version, usage of which may be reported and metered but not necessarily
billed.

OK, that use case makes sense to me. So, your resource is going to be
available for a few hours or days. I'm assuming that to get it on board CF
and meter it with Abacus you're going to run a cf create-service-broker
command or cf update-service-broker, define the resource config specifying
how to meter it, and store that config where your Abacus provisioning
endpoint implementation can retrieve it.

To secure the submission of usage for it, if I understand correctly how
UAA works, you'll then need to do this:
uaac client update <your service provider's client id> --authorities "...
existing permissions... abacus.<your resource id>.write"

That's all...

If that's really too much of a burden (really?) compared to the other
steps, you're basically looking to do *nothing* to secure that resource.
You could just submit usage with the abacus.usage.write scope, but that's
the equivalent of the CF cloud_controller.write scope for Abacus, close to
all powers... I'd probably advise against it as that's a serious risk but
that may be what you're looking for.

The scope names may need to follow adopter specific conventions so
creating scope with predefined name 'abacus.usage....' may not fit that
scheme. Abacus should offer ability to adjust the scope names, otherwise
submission may not be possible.

These are simple names that we expect in the token used to submit usage.
They're just constants like the names of our APIs, parameters, options,
fields in our JSON schemas... basically the contract/interface between the
Abacus user and its implementation. Not sure if there's a specific issue
with that abacus naming convention or if it's just a theoretical question,
but I'll be happy to discuss alternate naming conventions:

Do you have another naming convention in mind that you'd like to use?

Is there a specific issue with abacus.usage.write? Is the 'abacus' part in
the name a problem?

Would you prefer to submit usage with an existing CF scope like
cloud_controller.write or another of these high power scopes?
(again, I'd advise against it though...)

- Jean-Sebastien

- Jean-Sebastien

On Thu, Oct 8, 2015 at 5:24 PM, Piotr Przybylski <*piotrp(a)us.ibm.com*
<piotrp(a)us.ibm.com>> wrote:

- Hi Sebastien,

>> In some cases it may not be possible or viable to create new scope
for each resource id e.g. short lived resources.

>Why wouldn't that be possible? What type of short-lived resources did
you have in mind?

For example experimental service version (beta) replaced by release
version, usage of which may be reported and metered but not necessarily
billed.
The scope names may need to follow adopter specific conventions so
creating scope with predefined name 'abacus.usage....' may not fit
that scheme. Abacus should offer ability to adjust the scope names,
otherwise submission may not be possible.


> Another reason why I'm not sure about short lived resources, is that
although you may decide to stop offering a type a resource at some point,
once you've metered it, and sent a bill for it >to a customer, I don't
think you can really 'forget' about its existence anymore... So in that
sense I'm not sure how it can be 'short lived'.
The short lived resource is only for submission, once it is not
offered, its specific scope is not needed. Thad does not mean erasing
history of usage.


Piotr




[image: Inactive hide details for Jean-Sebastien Delfino ---10/08/2015
11:10:16 AM---Hi Piotr, > In some cases it may not be possible o]Jean-Sebastien
Delfino ---10/08/2015 11:10:16 AM---Hi Piotr, > In some cases it may not be
possible or viable to create new scope for

From: Jean-Sebastien Delfino <*jsdelfino(a)gmail.com*
<jsdelfino(a)gmail.com>>
To: "Discussions about Cloud Foundry projects and the system overall."
<*cf-dev(a)lists.cloudfoundry.org* <cf-dev(a)lists.cloudfoundry.org>>
Date: 10/08/2015 11:10 AM
Subject: [cf-dev] Re: Re: Re: [abacus] Usage submission authorization


------------------------------



Hi Piotr,

> In some cases it may not be possible or viable to create new scope
for each resource id e.g. short lived resources.

Why wouldn't that be possible? What type of short-lived resources did
you have in mind?

The typical use case I've seen is for a Cloud platform to decide to
offer a new type of database or analytics or messaging service, or a new
type of runtime for example. Before that new resource is offered on the
platform, their resource provider needs to get on board, get a user id,
auth credentials defined in UAA etc... You probably also need to define how
you're going to meter that new resource and the pricing for it.

Couldn't a scope be created in UAA at that time along all these other
on boarding steps?

Another reason why I'm not sure about short lived resources, is that
although you may decide to stop offering a type a resource at some point,
once you've metered it, and sent a bill for it to a customer, I don't think
you can really 'forget' about its existence anymore... So in that sense I'm
not sure how it can be 'short lived'.

> Some flexibility would also help to accommodate changes related to
grouping resources by type as discussed in [1].

We discussed two options in [1]:
a) support a resource_type in addition to resource_id for grouping
many resource_ids under a single type
b) a common resource_id for several resources (something like 'node'
for all your versions of Node.js build packs for example)

Since option (a) is not implemented at this point and Issue #38 is
actually assigned to a 'future' milestone, AIUI resource providers need to
use option (b) with a common resource_id for multiple resources. Is
creating a scope for that common id still too much of a burden then?

[1] - *https://github.com/cloudfoundry-incubator/cf-abacus/issues/38*
<https://github.com/cloudfoundry-incubator/cf-abacus/issues/38>

Thoughts?

- Jean-Sebastien

On Wed, Oct 7, 2015 at 5:51 PM, Piotr Przybylski <*piotrp(a)us.ibm.com*
<piotrp(a)us.ibm.com>> wrote:
-
- Hi Sebastien,

> That OAuth token should include:
> - a user id uniquely identifying that resource provider;
> - an OAuth scope named like abacus.usage.<resource_id>.write

What kind of customization of the above do you plan to expose?
In some cases it may not be possible or viable to create new scope for each
resource id e.g. short lived resources. The ability to either configure
scope to use for validation or provide scope 'mapping' would help to adapt
it to existing deployments. Some flexibility would also help to accommodate
changes related to grouping resources by type as discussed in [1].

[1] -
*https://github.com/cloudfoundry-incubator/cf-abacus/issues/38*
<https://github.com/cloudfoundry-incubator/cf-abacus/issues/38>


Piotr



[image: Inactive hide details for Jean-Sebastien Delfino
---10/07/2015 12:30:05 AM---Hi Piotr, > what kind of authorization is
required]Jean-Sebastien Delfino ---10/07/2015 12:30:05 AM---Hi
Piotr, > what kind of authorization is required to submit usage to Abacus ?

From: Jean-Sebastien Delfino <*jsdelfino(a)gmail.com*
<jsdelfino(a)gmail.com>>
To: "Discussions about Cloud Foundry projects and the system
overall." <*cf-dev(a)lists.cloudfoundry.org*
<cf-dev(a)lists.cloudfoundry.org>>
Date: 10/07/2015 12:30 AM
Subject: [cf-dev] Re: [abacus] Usage submission authorization
------------------------------




Hi Piotr,

> what kind of authorization is required to submit usage to
Abacus ?
> Is the oauth token used for submission [1] required to have
particular scope, specific to resource or resource provider ?

A resource provider is expected to present an OAuth token with
the usage it submits for a (service or runtime) resource.

That OAuth token should include:
- a user id uniquely identifying that resource provider;
- an OAuth scope named like abacus.usage.<resource_id>.write.

The precise naming syntax for that scope may still evolve in the
next few days as we progress with the implementation of user story
101703426 [1].

> Is there a different scope required to submit runtimes usage
(like cf bridge) versus other services or its possible to use single scope
for all the submissions

I'd like to handle runtimes and services consistently as they're
basically just different types of resources, i.e. one scope per 'service'
resource, one scope per 'runtime' resource.

We're still working on the detailed design and implementation,
but I'm not sure we'd want to share scopes across (service and runtime)
resource providers as that'd allow a resource provider to submit usage for
resources owned by another...

@assk / @sasrin, anything I missed? Thoughts?

-- Jean-Sebastien


On Tue, Oct 6, 2015 at 6:29 PM, Piotr Przybylski <
*piotrp(a)us.ibm.com* <piotrp(a)us.ibm.com>> wrote:
-
-
-
- Hi,
what kind of authorization is required to submit
usage to Abacus ?
Is the oauth token used for submission [1] required
to have particular scope, specific to resource or resource provider ? Is
there a different scope required to submit runtimes usage (like cf bridge)
versus other services or its possible to use single scope for all the
submissions ?


[1] -
*https://www.pivotaltracker.com/story/show/101703426*
<https://www.pivotaltracker.com/story/show/101703426>

Piotr













Re: REST API endpoint for accessing application logs

Warren Fernandes
 

Hi Ponraj,

It could be that your token may have expired. Also, silly question but you may want to check that you have access to the app in question. A good way to verify this is by running `cf logs <your_app_name> --recent`

Hope this helps.
Warren


Re: [abacus] Usage submission authorization

Piotr Przybylski <piotrp@...>
 

Assk,
can you confirm that the same scope (abacus.usage.write) is sufficient to retrieve usage ? 

Piotr
 
 

----- Original message -----
From: Saravanakumar A Srinivasan/Burlingame/IBM@IBMUS
To: <cf-dev@...>
Cc:
Subject: [cf-dev] Re: Re: Re: [cf-dev][abacus] Usage submission authorization
Date: Thu, Oct 15, 2015 7:06 PM
 
We have enabled scope based authorization for REST endpoints at usage collector and usage reporting service. While we are working on using system OAuth bearer access token at internal Abacus pipeline, Submitting usage to a secured Abacus needs a OAuth bearer access token with 'abacus.usage.write' scope.

Thanks,
Saravanakumar Srinivasan (Assk),

Bay Area Lab, 1001, E Hillsdale Blvd, Ste 400, Foster City, CA - 94404.
E-mail: sasrin@...
Phone: 650 645 8251 (T/L 367-8251)


-----Jean-Sebastien Delfino <jsdelfino@...> wrote: -----
To: "Discussions about Cloud Foundry projects and the system overall." <cf-dev@...>
From: Jean-Sebastien Delfino <jsdelfino@...>
Date: 10/12/2015 07:50PM
Subject: [cf-dev] Re: Re: [cf-dev][abacus] Usage submission authorization

 
> Also, resource id is an arbitrary identifier, making it part of the scope may create quite complex names e.g. 'abacus.runtimes/node/v12-07.revision-2-buildpack-guid-a3d7ff4d-3cb1-4cc3-a855-fae98e20cf57.write.
 
Do you have a specific issue in mind with putting the resource uuid in the scope name? We have uuids all over the place in CF, in most of the APIs, the usage docs etc so I'm not sure why it'd be a problem to have one here.
 
> Any naming convention may not be generic enough, for example for my UAA instance requires the scope names to start with component using it, followed by proper name - 'bss.runtimes.abacus.<resource id>.write'.
 
Like I said before, if you can't or don't want to use a specific scope per resource, then you can use abacus.usage.write (with the same disclaimers/warnings I gave in my previous post.)
 
I must be missing something though :) ... aren't you happily using cloud_controller.write for example (or similar other CF scopes) without renaming it to <your client component>.cloud_controller.write? Why would you treat abacus.usage.write different?
 
Also, I must admit to find a bit surprising a naming convention that will tie the scope name to the client that presents it. Isn't the scope typically defined by the owner of the resource it protects instead of the client? In that case the owner of the resource is not the client component... it is the CF abacus project, hence <abacus>.usage.write. Wouldn't that make more sense?
 
Finally, I'm also not quite sure how this will work at all if for example Abacus needs to authorize resource access from multiple clients. That would have to be really dynamic then, as each new client would require Abacus to know about a new client specific naming convention (or client component name prefix in the example you gave...)
 
Now, all that being said, looks like I'm not really following how you're envisioning this to work, so do you think you could maybe submit a pull request with how you concretely propose to make that dynamic scope naming work when it includes client component names, or follows client component specific naming conventions?
 
Thanks!
 
- Jean-Sebastien
 
On Mon, Oct 12, 2015 at 5:22 PM, Piotr Przybylski <piotrp@...> wrote:

Hi Sebastien,
I am not sure why allowing resource provider to explicitly specify scope with which particular resource usage will be submitted is a problem. Just allowing to pick a name would not compromise submission security in any way. It could be done for example by adding scope name to the resource definition.

Any naming convention may not be generic enough, for example for my UAA instance requires the scope names to start with component using it, followed by proper name - 'bss.runtimes.abacus.<resource id>.write'. Also, resource id is an arbitrary identifier, making it part of the scope may create quite complex names e.g.
'abacus.runtimes/node/v12-07.revision-2-buildpack-guid-a3d7ff4d-3cb1-4cc3-a855-fae98e20cf57.write.

Piotr



Jean-Sebastien Delfino ---10/09/2015 09:38:09 PM---Hey Piotr, >>> In some cases it may not be possible or viable to create new scope for

From: Jean-Sebastien Delfino <jsdelfino@...>
To: "Discussions about Cloud Foundry projects and the system overall." <cf-dev@...>

Date: 10/09/2015 09:38 PM
Subject: [cf-dev] Re: Re: Re: Re: Re: [abacus] Usage submission authorization

 



Hey Piotr,

>>> In some cases it may not be possible or viable to create new scope for each resource id e.g. short lived resources.

>> Why wouldn't that be possible? What type of short-lived resources did you have in mind?

> For example experimental service version (beta) replaced by release version, usage of which may be reported and metered but not necessarily billed. 

OK, that use case makes sense to me. So, your resource is going to be available for a few hours or days. I'm assuming that to get it on board CF and meter it with Abacus you're going to run a cf create-service-broker command or cf update-service-broker, define the resource config specifying how to meter it, and store that config where your Abacus provisioning endpoint implementation can retrieve it.

To secure the submission of usage for it, if I understand correctly how UAA works, you'll then need to do this:
uaac client update <your service provider's client id> --authorities "... existing permissions... abacus.<your resource id>.write"

That's all...

If that's really too much of a burden (really?) compared to the other steps, you're basically looking to do *nothing* to secure that resource. You could just submit usage with the abacus.usage.write scope, but that's the equivalent of the CF cloud_controller.write scope for Abacus, close to all powers... I'd probably advise against it as that's a serious risk but that may be what you're looking for.

> The scope names may need to follow adopter specific conventions so creating scope with predefined name 'abacus.usage....' may not fit that scheme. Abacus should offer ability to adjust the scope names, otherwise submission may not be possible.

These are simple names that we expect in the token used to submit usage. They're just constants like the names of our APIs, parameters, options, fields in our JSON schemas... basically the contract/interface between the Abacus user and its implementation. Not sure if there's a specific issue with that abacus naming convention or if it's just a theoretical question, but I'll be happy to discuss alternate naming conventions:

Do you have another naming convention in mind that you'd like to use?

Is there a specific issue with abacus.usage.write? Is the 'abacus' part in the name a problem?

Would you prefer to submit usage with an existing CF scope like cloud_controller.write or another of these high power scopes?
(again, I'd advise against it though...)

- Jean-Sebastien

- Jean-Sebastien

On Thu, Oct 8, 2015 at 5:24 PM, Piotr Przybylski <piotrp@...> wrote:
  • Hi Sebastien,

    >> In some cases it may not be possible or viable to create new scope for each resource id e.g. short lived resources.

    >Why wouldn't that be possible? What type of short-lived resources did you have in mind?

    For example experimental service version (beta) replaced by release version, usage of which may be reported and metered but not necessarily billed.
    The scope names may need to follow adopter specific conventions so creating scope with predefined name '
    abacus.usage....' may not fit that scheme. Abacus should offer ability to adjust the scope names, otherwise submission may not be possible.


    > Another reason why I'm not sure about short lived resources, is that although you may decide to stop offering a type a resource at some point, once you've metered it, and sent a bill for it >to a customer, I don't think you can really 'forget' about its existence anymore... So in that sense I'm not sure how it can be 'short lived'.
    The short lived resource is only for submission, once it is not offered, its specific scope is not needed. Thad does not mean erasing history of usage.


    Piotr




    Jean-Sebastien Delfino ---10/08/2015 11:10:16 AM---Hi Piotr, > In some cases it may not be possible or viable to create new scope for

    From: Jean-Sebastien Delfino <jsdelfino@...>
    To: "Discussions about Cloud Foundry projects and the system overall." <cf-dev@...>
    Date: 10/08/2015 11:10 AM
    Subject: [cf-dev] Re: Re: Re: [abacus] Usage submission authorization

     





    Hi Piotr,

    > In some cases it may not be possible or viable to create new scope for each resource id e.g. short lived resources.

    Why wouldn't that be possible? What type of short-lived resources did you have in mind?

    The typical use case I've seen is for a Cloud platform to decide to offer a new type of database or analytics or messaging service, or a new type of runtime for example. Before that new resource is offered on the platform, their resource provider needs to get on board, get a user id, auth credentials defined in UAA etc... You probably also need to define how you're going to meter that new resource and the pricing for it.

    Couldn't a scope be created in UAA at that time along all these other on boarding steps?

    Another reason why I'm not sure about short lived resources, is that although you may decide to stop offering a type a resource at some point, once you've metered it, and sent a bill for it to a customer, I don't think you can really 'forget' about its existence anymore... So in that sense I'm not sure how it can be 'short lived'.

    > Some flexibility would also help to accommodate changes related to grouping resources by type as discussed in [1].

    We discussed two options in [1]:
    a) support a resource_type in addition to resource_id for grouping many resource_ids under a single type
    b) a common resource_id for several resources (something like 'node' for all your versions of Node.js build packs for example) 

    Since option (a) is not implemented at this point and Issue #38 is actually assigned to a 'future' milestone, AIUI resource providers need to use option (b) with a common resource_id for multiple resources. Is creating a scope for that common id still too much of a burden then?

    [1] - https://github.com/cloudfoundry-incubator/cf-abacus/issues/38

    Thoughts?

    - Jean-Sebastien

    On Wed, Oct 7, 2015 at 5:51 PM, Piotr Przybylski <piotrp@...> wrote:
      • Hi Sebastien,

        > That OAuth token should include:
        > - a user id uniquely identifying that resource provider;
        > - an OAuth scope named like abacus.usage.<resource_id>.write

        What kind of customization of the above do you plan to expose? In some cases it may not be possible or viable to create new scope for each resource id e.g. short lived resources. The ability to either configure scope to use for validation or provide scope 'mapping' would help to adapt it to existing deployments. Some flexibility would also help to accommodate changes related to grouping resources by type as discussed in [1].

        [1] - https://github.com/cloudfoundry-incubator/cf-abacus/issues/38


        Piotr



        Jean-Sebastien Delfino ---10/07/2015 12:30:05 AM---Hi Piotr, > what kind of authorization is required to submit usage to Abacus ?

        From: Jean-Sebastien Delfino <jsdelfino@...>
        To: "Discussions about Cloud Foundry projects and the system overall." <cf-dev@...>
        Date: 10/07/2015 12:30 AM
        Subject: [cf-dev] Re: [abacus] Usage submission authorization





        Hi Piotr,

        > what kind of authorization is required to submit usage to Abacus ? 
        > Is the oauth token used for submission [1] required to have particular scope, specific to resource or resource provider ?

        A resource provider is expected to present an OAuth token with the usage it submits for a (service or runtime) resource.

        That OAuth token should include:
        - a user id uniquely identifying that resource provider;
        - an OAuth scope named like abacus.usage.<resource_id>.write.

        The precise naming syntax for that scope may still evolve in the next few days as we progress with the implementation of user story 101703426 [1].

        > Is there a different scope required to submit runtimes usage (like cf bridge) versus other services or its possible to use single scope for all the submissions

        I'd like to handle runtimes and services consistently as they're basically just different types of resources, i.e. one scope per 'service' resource, one scope per 'runtime' resource.

        We're still working on the detailed design and implementation, but I'm not sure we'd want to share scopes across (service and runtime) resource providers as that'd allow a resource provider to submit usage for resources owned by another...

        @assk / @sasrin, anything I missed? Thoughts?

        -- Jean-Sebastien


        On Tue, Oct 6, 2015 at 6:29 PM, Piotr Przybylski <piotrp@...> wrote:
              • Hi,
                what kind of authorization is required to submit usage to Abacus ?
                Is the oauth token used for submission [1] required to have particular scope, specific to resource or resource provider ? Is there a different scope required to submit runtimes usage (like cf bridge) versus other services or its possible to use single scope for all the submissions ?



                [1] - https://www.pivotaltracker.com/story/show/101703426

                Piotr

     

     

 

 

 



Re: [abacus] Usage submission authorization

Saravanakumar A. Srinivasan
 

We have enabled scope based authorization for REST endpoints at usage collector and usage reporting service. While we are working on using system OAuth bearer access token at internal Abacus pipeline, Submitting usage to a secured Abacus needs a OAuth bearer access token with 'abacus.usage.write' scope.

Thanks,
Saravanakumar Srinivasan (Assk),

Bay Area Lab, 1001, E Hillsdale Blvd, Ste 400, Foster City, CA - 94404.
E-mail: sasrin@...
Phone: 650 645 8251 (T/L 367-8251)


-----Jean-Sebastien Delfino <jsdelfino@...> wrote: -----
To: "Discussions about Cloud Foundry projects and the system overall." <cf-dev@...>
From: Jean-Sebastien Delfino <jsdelfino@...>
Date: 10/12/2015 07:50PM
Subject: [cf-dev] Re: Re: [cf-dev][abacus] Usage submission authorization

> Also, resource id is an arbitrary identifier, making it part of the scope may create quite complex names e.g. 'abacus.runtimes/node/v12-07.revision-2-buildpack-guid-a3d7ff4d-3cb1-4cc3-a855-fae98e20cf57.write.

Do you have a specific issue in mind with putting the resource uuid in the scope name? We have uuids all over the place in CF, in most of the APIs, the usage docs etc so I'm not sure why it'd be a problem to have one here.

> Any naming convention may not be generic enough, for example for my UAA instance requires the scope names to start with component using it, followed by proper name - 'bss.runtimes.abacus.<resource id>.write'.

Like I said before, if you can't or don't want to use a specific scope per resource, then you can use abacus.usage.write (with the same disclaimers/warnings I gave in my previous post.)

I must be missing something though :) ... aren't you happily using cloud_controller.write for example (or similar other CF scopes) without renaming it to <your client component>.cloud_controller.write? Why would you treat abacus.usage.write different?

Also, I must admit to find a bit surprising a naming convention that will tie the scope name to the client that presents it. Isn't the scope typically defined by the owner of the resource it protects instead of the client? In that case the owner of the resource is not the client component... it is the CF abacus project, hence <abacus>.usage.write. Wouldn't that make more sense?

Finally, I'm also not quite sure how this will work at all if for example Abacus needs to authorize resource access from multiple clients. That would have to be really dynamic then, as each new client would require Abacus to know about a new client specific naming convention (or client component name prefix in the example you gave...)

Now, all that being said, looks like I'm not really following how you're envisioning this to work, so do you think you could maybe submit a pull request with how you concretely propose to make that dynamic scope naming work when it includes client component names, or follows client component specific naming conventions?

Thanks!

- Jean-Sebastien


On Mon, Oct 12, 2015 at 5:22 PM, Piotr Przybylski <piotrp@...> wrote:

Hi Sebastien,
I am not sure why allowing resource provider to explicitly specify scope with which particular resource usage will be submitted is a problem. Just allowing to pick a name would not compromise submission security in any way. It could be done for example by adding scope name to the resource definition.

Any naming convention may not be generic enough, for example for my UAA instance requires the scope names to start with component using it, followed by proper name - 'bss.runtimes.abacus.<resource id>.write'. Also, resource id is an arbitrary identifier, making it part of the scope may create quite complex names e.g.
'abacus.runtimes/node/v12-07.revision-2-buildpack-guid-a3d7ff4d-3cb1-4cc3-a855-fae98e20cf57.write.

Piotr



Jean-Sebastien Delfino ---10/09/2015 09:38:09 PM---Hey Piotr, >>> In some cases it may not be possible or viable to create new scope for

From: Jean-Sebastien Delfino <jsdelfino@...>
To: "Discussions about Cloud Foundry projects and the system overall." <cf-dev@...>
Date: 10/09/2015 09:38 PM
Subject: [cf-dev] Re: Re: Re: Re: Re: [abacus] Usage submission authorization






Hey Piotr,

>>> In some cases it may not be possible or viable to create new scope for each resource id e.g. short lived resources.

>> Why wouldn't that be possible? What type of short-lived resources did you have in mind?

> For example experimental service version (beta) replaced by release version, usage of which may be reported and metered but not necessarily billed. 

OK, that use case makes sense to me. So, your resource is going to be available for a few hours or days. I'm assuming that to get it on board CF and meter it with Abacus you're going to run a cf create-service-broker command or cf update-service-broker, define the resource config specifying how to meter it, and store that config where your Abacus provisioning endpoint implementation can retrieve it.

To secure the submission of usage for it, if I understand correctly how UAA works, you'll then need to do this:
uaac client update <your service provider's client id> --authorities "... existing permissions... abacus.<your resource id>.write"

That's all...

If that's really too much of a burden (really?) compared to the other steps, you're basically looking to do *nothing* to secure that resource. You could just submit usage with the abacus.usage.write scope, but that's the equivalent of the CF cloud_controller.write scope for Abacus, close to all powers... I'd probably advise against it as that's a serious risk but that may be what you're looking for.

> The scope names may need to follow adopter specific conventions so creating scope with predefined name 'abacus.usage....' may not fit that scheme. Abacus should offer ability to adjust the scope names, otherwise submission may not be possible.

These are simple names that we expect in the token used to submit usage. They're just constants like the names of our APIs, parameters, options, fields in our JSON schemas... basically the contract/interface between the Abacus user and its implementation. Not sure if there's a specific issue with that abacus naming convention or if it's just a theoretical question, but I'll be happy to discuss alternate naming conventions:

Do you have another naming convention in mind that you'd like to use?

Is there a specific issue with abacus.usage.write? Is the 'abacus' part in the name a problem?

Would you prefer to submit usage with an existing CF scope like cloud_controller.write or another of these high power scopes?
(again, I'd advise against it though...)

- Jean-Sebastien

- Jean-Sebastien

On Thu, Oct 8, 2015 at 5:24 PM, Piotr Przybylski <piotrp@...> wrote:
    Hi Sebastien,

    >> In some cases it may not be possible or viable to create new scope for each resource id e.g. short lived resources.

    >Why wouldn't that be possible? What type of short-lived resources did you have in mind?

    For example experimental service version (beta) replaced by release version, usage of which may be reported and metered but not necessarily billed.
    The scope names may need to follow adopter specific conventions so creating scope with predefined name '
    abacus.usage....' may not fit that scheme. Abacus should offer ability to adjust the scope names, otherwise submission may not be possible.


    > Another reason why I'm not sure about short lived resources, is that although you may decide to stop offering a type a resource at some point, once you've metered it, and sent a bill for it >to a customer, I don't think you can really 'forget' about its existence anymore... So in that sense I'm not sure how it can be 'short lived'.
    The short lived resource is only for submission, once it is not offered, its specific scope is not needed. Thad does not mean erasing history of usage.



    Piotr




    Jean-Sebastien Delfino ---10/08/2015 11:10:16 AM---Hi Piotr, > In some cases it may not be possible or viable to create new scope for

    From:
    Jean-Sebastien Delfino <jsdelfino@...>
    To:
    "Discussions about Cloud Foundry projects and the system overall." <cf-dev@...>
    Date:
    10/08/2015 11:10 AM
    Subject:
    [cf-dev] Re: Re: Re: [abacus] Usage submission authorization






    Hi Piotr,

    > In some cases it may not be possible or viable to create new scope for each resource id e.g. short lived resources.

    Why wouldn't that be possible? What type of short-lived resources did you have in mind?

    The typical use case I've seen is for a Cloud platform to decide to offer a new type of database or analytics or messaging service, or a new type of runtime for example. Before that new resource is offered on the platform, their resource provider needs to get on board, get a user id, auth credentials defined in UAA etc... You probably also need to define how you're going to meter that new resource and the pricing for it.

    Couldn't a scope be created in UAA at that time along all these other on boarding steps?

    Another reason why I'm not sure about short lived resources, is that although you may decide to stop offering a type a resource at some point, once you've metered it, and sent a bill for it to a customer, I don't think you can really 'forget' about its existence anymore... So in that sense I'm not sure how it can be 'short lived'.

    > Some flexibility would also help to accommodate changes related to grouping resources by type as discussed in [1].

    We discussed two options in [1]:
    a) support a resource_type in addition to resource_id for grouping many resource_ids under a single type
    b) a common resource_id for several resources (something like 'node' for all your versions of Node.js build packs for example) 

    Since option (a) is not implemented at this point and Issue #38 is actually assigned to a 'future' milestone, AIUI resource providers need to use option (b) with a common resource_id for multiple resources. Is creating a scope for that common id still too much of a burden then?

    [1] - https://github.com/cloudfoundry-incubator/cf-abacus/issues/38

    Thoughts?

    - Jean-Sebastien

    On Wed, Oct 7, 2015 at 5:51 PM, Piotr Przybylski <piotrp@...> wrote:
        Hi Sebastien,

        > That OAuth token should include:
        > - a user id uniquely identifying that resource provider;
        > - an OAuth scope named like abacus.usage.<resource_id>.write

        What kind of customization of the above do you plan to expose? In some cases it may not be possible or viable to create new scope for each resource id e.g. short lived resources. The ability to either configure scope to use for validation or provide scope 'mapping' would help to adapt it to existing deployments. Some flexibility would also help to accommodate changes related to grouping resources by type as discussed in [1].

        [1] - https://github.com/cloudfoundry-incubator/cf-abacus/issues/38


        Piotr



        Jean-Sebastien Delfino ---10/07/2015 12:30:05 AM---Hi Piotr, > what kind of authorization is required to submit usage to Abacus ?

        From:
        Jean-Sebastien Delfino <jsdelfino@...>
        To:
        "Discussions about Cloud Foundry projects and the system overall." <cf-dev@...>
        Date:
        10/07/2015 12:30 AM
        Subject:
        [cf-dev] Re: [abacus] Usage submission authorization





        Hi Piotr,

        > what kind of authorization is required to submit usage to Abacus ? 
        > Is the oauth token used for submission [1] required to have particular scope, specific to resource or resource provider ?

        A resource provider is expected to present an OAuth token with the usage it submits for a (service or runtime) resource.

        That OAuth token should include:
        - a user id uniquely identifying that resource provider;
        - an OAuth scope named like abacus.usage.<resource_id>.write.

        The precise naming syntax for that scope may still evolve in the next few days as we progress with the implementation of user story 101703426 [1].

        > Is there a different scope required to submit runtimes usage (like cf bridge) versus other services or its possible to use single scope for all the submissions

        I'd like to handle runtimes and services consistently as they're basically just different types of resources, i.e. one scope per 'service' resource, one scope per 'runtime' resource.

        We're still working on the detailed design and implementation, but I'm not sure we'd want to share scopes across (service and runtime) resource providers as that'd allow a resource provider to submit usage for resources owned by another...

        @assk / @sasrin, anything I missed? Thoughts?

        -- Jean-Sebastien


        On Tue, Oct 6, 2015 at 6:29 PM, Piotr Przybylski <piotrp@...> wrote:
                Hi,
                what kind of authorization is required to submit usage to Abacus ?
                Is the oauth token used for submission [1] required to have particular scope, specific to resource or resource provider ? Is there a different scope required to submit runtimes usage (like cf bridge) versus other services or its possible to use single scope for all the submissions ?



                [1] - https://www.pivotaltracker.com/story/show/101703426

                Piotr






Re: [abacus] Accepting delayed usage within a slack window

Jean-Sebastien Delfino
 

Hi Ben,

I'm adding a 'processed' time field to the usage docs, hoping that'll help
maintain the history of usage within the slack window we've been discussing
here (more precisely that will help you know how much of that history can
roll out of the configured slack window when you process a new usage doc).

That field will also allow us to more clearly distinguish between the usage
event 'start', 'stop' and 'processed' times.

HTH

- Jean-Sebastien

On Mon, Oct 12, 2015 at 9:04 AM, Jean-Sebastien Delfino <jsdelfino(a)gmail.com
wrote:
Hi Ben,

That makes sense to me. What you've described will enable refinements of
accumulated usage for a month as we continue to receive delayed usage
during the first few days of the next month.

To illustrate this with an example: with a 48h time window, on Sept 30 you
can retrieve the Sept 30 usage doc and find 'provisional' usage for Sept in
the 'month time window', not including unknown usage not yet been submitted
to Abacus. Later on Oct 2nd you can retrieve the Oct 2nd usage doc and find
the 'final usage' for Sept in the 'month - 1 time window'. I think this is
better than waiting for Oct 2nd to 'close the Sept window', as our users
typically want to see both their *real time* usage for Sept before Oct 2nd
and their final usage later once it has settled for sure.

I also like that with that approach you don't need to go back to your Sept
30 usage doc to patch it up with delayed usage, as that way you're also
keeping a record of the Sept usage that was really known to us on Sept 30.

Another interesting aspect of this is that the history you're going to
maintain will allow us to write 'marker' usage docs when we transition from
one time window to another. Since a usage doc contains both the usage for
the day and the previous day, you can write the first document you process
each day, as a marker, in a reporting db and that'll give you an easy and
efficient way to retrieve the accumulated usage for the previous day. For
example, to retrieve the usage accumulated at the end of Oct 11, just
retrieve the 'marker' usage doc for Oct 12 and get the usage in its 'day -
1 time window'. That could help us implement the kind of query that Georgi
mentioned on the chat last week when he was looking for an efficient way to
retrieve daily usage for all the days of the month.

Finally, looking at the array of numbers/objects currently used to
maintain our time windows, I'm wondering if keeping the 'yearly' and
'forever' usage time windows is not a bit overkill (and could actually
become a problem).

That data is going to be duplicated in all individual usage docs for
little value IMO as the yearly usage at least is easy to reconstruct at
reporting time with a query over 12 monthly usage docs. Also, maintaining
that 'forever' usage will require us to keep usage docs around for resource
instances that may have been deleted long time ago, and will complicate our
database partitioning scheme as these old resource instances will cause the
databases to grow forever. So, I'd prefer to let old usage data sit in old
monthly database partitions instead of having to carry that old data over
each month forever just to maintain these 'forever' time windows.

In other words, I'm suggesting to change our current array of 7 time
windows [Forever, Y, M, D, h, m, s] to 5 windows [M, D, h, m, s]. Combined
with your slack window proposal, with a 2D slack time we'll be looking at
an array like follows: [[M, M-1], [D, D-1, D-2], [h], [m], [s]]. With a 48h
slack time the array will have 49 hourly entries [h, h-1, h-2, h-3, etc]
instead of one.

Thoughts?


- Jean-Sebastien

On Sun, Oct 11, 2015 at 6:04 AM, Benjamin Cheng <bscheng(a)us.ibm.com>
wrote:

One of the things that need to be supported in abacus is the handling of
delayed usage submissions within a particular slack window after the usage
has happened. For example, given a slack window of 48 hours, a service
provider will be able to submit usage back to September 30th on October 2nd.

An idea that we were discussing about for this was augmenting the
quantity from an array of numbers/objects to an array of arrays of
numbers/objects and using an environmental variable that is currently going
to be called SLACK to hold the configuration of the slack window. SLACK
would follow a format of [0-9]+[YMDhms] with the width of the slack window
and to what precision the slack window should be maintained. 2D and 48h
both are the same time, but 48h will keep track of the history to the hour
level while 2D will only keep it to the day level. If this environment
variable isn't configured, the current idea is to have no slack window as
the default.

The general formula for the length of each array in a time window would
be as follows: 1(This is for usage covered in the current window) + (number
of windows to cover the configured slack window for the particular time
window).
IE: Given a slack of 48h. The year time window would be 1 + 1. Month
would be 1 + 1. Day would be 1 + 2. Hours would be 1 + 48. Minutes/Seconds
would stay at 1.

Thoughts on this idea?


Re: Multi-Line Loggregator events and the new Splunk "HTTP Event Collector" API

Mike Youngstrom
 

Great! Thanks Jim. Sounds completely reasonable. Hopefully we can keep
this thread moving and help derive some future designs out of it. Would
you prefer to keep this discussion on the mailling list or a github issue?

Mike

On Thu, Oct 15, 2015 at 12:26 PM, Jim Campbell <jcampbell(a)pivotal.io> wrote:

New Loggregator PM chiming in.

This is definitely on the Loggregator roadmap. Only issues are selecting a
design and finalizing (as much as is ever possible) where it lives in the
priority order. We would certainly consider a pull request if it met a
consensus architecture model ala Rohit's concerns.



On Thu, Oct 15, 2015 at 11:19 AM, Mike Youngstrom <youngm(a)gmail.com>
wrote:

We have thrown around one approach which solves the problem but would
require changes in the runtime. That solution would expose a socket to the
container where the application could emit logs. The application would now
have control over what delimits a message.
Any thoughts on what protocol this socket would listen too? Raw
Dropsonde? Syslog?

I've been thinking about your questions regarding the '\' approach:

As Erik mentioned in the last thread, multi-line logging is something
which the loggregator team would like to solve. But there are a few
questions to answer before we can come up with a clean solution. We want a
design which solves the problem while not breaking existing apps which do
not require this functionality. Before implementing a solution we would
also want to answer if we want to do it for both runtimes or just Diego,
since the way log lines are sent to Metron differ based on the runtime.
I'd be perfectly happy if the solution was only for Diego. We are
surviving today but I think the feasibility of our current solution is
running out.


I guess the expectation is that loggregator would internally remove the
escape character.
I think this would be optimal.


This has performance implications because now some part of loggregator
will need to inspect the log message and coalesce the message with the
succeeding ones. We will need to do this in a way which respects
multi-tenancy. That means now we are storing additional state related to
log lines per app. We will also need to decide how long loggregator needs
to wait for the next lines in a multi-line log, before deciding to send the
line which it received. To me that's not a simple change.
Can you help me understand what you are referring to with "coalescing
messages" and "storing additional state related to log lines per app"? The
way I see it the current agents buffer until they see a '\n' then they
create an event. Adding an escaped line '\\n' the logic would be very much
the same as it is today, buffer until you find an unescaped new line. Then
unescape the remaining new lines.

Seems somewhat straight forward to me. Thoughts on considering a pull
request that does something like this?

Mike


--
Jim Campbell
Product Manager
Pivotal Labs


Re: Loggregator Community Survey #1 - still use the old metron 51160 endpoint?

Mike Youngstrom
 

We are using it. Though we plan to fix this. If we could get a general
timeline for what release you plan to remove it in then we can plan to
remove it from our code by then.

Mike

On Fri, Sep 18, 2015 at 6:19 PM, Rohit Kumar <rokumar(a)pivotal.io> wrote:

Quick correction: the default legacy metron port is 3456 [1]. The legacy
forwarder only supports log messages.


https://github.com/cloudfoundry/loggregator/blob/develop/bosh/jobs/metron_agent/spec#L28-L30

On Fri, Sep 18, 2015 at 3:20 PM, Erik Jasiak <mjasiak(a)pivotal.io> wrote:

Greetings CF community!

The loggregator team is actively trying to clean up its footprint. To
that - does anyone still use the "old" metron endpoint, on port 51160 by
default? [1]

We are considering turning the old endpoint into an injector[2] to help
shrink the footprint and overhead of core metron, but only if the old
endpoint is still in active usage - so please let us know.

Thanks
Erik
PM - Loggregator


[1]
https://github.com/cloudfoundry/loggregator/blob/develop/src/metron/config/metron.json#L12-L13
[2] https://github.com/cloudfoundry/statsd-injector


Cloud Foundry Java Client V2

Ben Hale <bhale@...>
 

As many of you are aware, the Cloud Foundry Java Client has been a bit neglected lately. There are various reasons for this, but today I’m pleased to announce that we’ve begun a V2 effort and that progress is swift.

We on the Cloud Foundry Java Experience team have been aware for some time that the current implementation of the Java Client is less than ideal. Among the most common complaints were the lack of separation between interface and implementation, the subpar network performance, and the requirement that users understand how to orchestrate common concepts like `push` on their own. (For a more in-depth treatment of issues we identified, please see the stellar work done by Scott Fredrick[1].) V2 aims to address all of these issues with a ground-up redesign of the client.

To address the issue of a lack of separation between interface and implementation, we’ve broken out the API into a project containing no implementation. This project (`cloudfoundry-client`) consists of a collection of interfaces and immutable datatypes. There is only a single dependency, and it isn’t Spring! The intent here was to create an API that could be implemented with multiple strategies, but requiring the minimal amount of code for each of those implementations. The API itself is now reactive (the single dependency is on Reactive Streams, the precursor to reactive support in Java 9) which we believe will more closely align with the trends towards non-blocking network communication. We will be providing a single implementation of this API, based on Spring (`cloudfoundry-client-spring`) but welcome additional implementations. We believe we’ve created a good environment for alternatives and would be happy to hear suggestions on how to improve if that turns out not to be the case.

In V1, the coverage of the APIs[2] was incomplete (about half, if I had to guess). Our commitment is to deliver a complete API and implementation in V2, including all 300+ documented APIs. We’ve observed that this API might not actually be the right level of abstraction for many users though. Knowing that you need to create an application, create a package, stage a droplet, create and start a process, etc. for `push` is quite a burden on many users of the project. So, we’re also providing a `cloudfoundry-operations` project that builds on the `cloudfoundry-client` API but instead of mapping onto the low-level REST API, we’re going to map roughly onto the `cf` CLI API. We suspect that nearly all users will want to `cloudFoundryOperations.push()` instead of the low-level alternative, so both choices are useful. This API and implementation will only depend on `cloudfoundry-client` allowing any implementation of that API to be used. Finally, we’ll be bringing the build-system plugins up to date with the systems that they are built for and ensuring that they cover a breadth of useful functionality at build time.

This leaves the question about what will happen to V1. We have a commitment to fixing up the bugs that have been identified in the code-base, but we’re not going to be doing any work that involves adding APIs. We feel that users who need those APIs are better served moving to V2. I’ll be feeding open issues from the backlog into the V2 stream of work to ensure that we aren’t seeing any resource starvation and you can expect future releases out of the `1.x` branch.

I hope that this comes as welcome news to the community and look forward to the feedback. I highly encourage users to keep an eye Pivotal Tracker[3] to see our progress and submit requests through GitHub[4].


-Ben Hale
Cloud Foundry Java Experience

[1]: https://docs.google.com/document/d/1Ui-67dBPYoADltErL80xXYEr_INPqdNJG9Va4gPBM-I/edit?usp=sharing
[2]: http://apidocs.cloudfoundry.org/221/
[3]: https://www.pivotaltracker.com/projects/816799
[4]: https://github.com/cloudfoundry/cf-java-client


Re: Multi-Line Loggregator events and the new Splunk "HTTP Event Collector" API

Jim CF Campbell
 

New Loggregator PM chiming in.

This is definitely on the Loggregator roadmap. Only issues are selecting a
design and finalizing (as much as is ever possible) where it lives in the
priority order. We would certainly consider a pull request if it met a
consensus architecture model ala Rohit's concerns.

On Thu, Oct 15, 2015 at 11:19 AM, Mike Youngstrom <youngm(a)gmail.com> wrote:

We have thrown around one approach which solves the problem but would
require changes in the runtime. That solution would expose a socket to the
container where the application could emit logs. The application would now
have control over what delimits a message.
Any thoughts on what protocol this socket would listen too? Raw
Dropsonde? Syslog?

I've been thinking about your questions regarding the '\' approach:

As Erik mentioned in the last thread, multi-line logging is something
which the loggregator team would like to solve. But there are a few
questions to answer before we can come up with a clean solution. We want a
design which solves the problem while not breaking existing apps which do
not require this functionality. Before implementing a solution we would
also want to answer if we want to do it for both runtimes or just Diego,
since the way log lines are sent to Metron differ based on the runtime.
I'd be perfectly happy if the solution was only for Diego. We are
surviving today but I think the feasibility of our current solution is
running out.


I guess the expectation is that loggregator would internally remove the
escape character.
I think this would be optimal.


This has performance implications because now some part of loggregator
will need to inspect the log message and coalesce the message with the
succeeding ones. We will need to do this in a way which respects
multi-tenancy. That means now we are storing additional state related to
log lines per app. We will also need to decide how long loggregator needs
to wait for the next lines in a multi-line log, before deciding to send the
line which it received. To me that's not a simple change.
Can you help me understand what you are referring to with "coalescing
messages" and "storing additional state related to log lines per app"? The
way I see it the current agents buffer until they see a '\n' then they
create an event. Adding an escaped line '\\n' the logic would be very much
the same as it is today, buffer until you find an unescaped new line. Then
unescape the remaining new lines.

Seems somewhat straight forward to me. Thoughts on considering a pull
request that does something like this?

Mike

--
Jim Campbell
Product Manager
Pivotal Labs


Re: Recording of this morning's CF CAB call - 14th Oct 2015

Mike Youngstrom
 

Thanks Phil! I couldn't make the call so this saved my bacon.

Mike

On Thu, Oct 15, 2015 at 5:08 AM, Lomov Alexander <
alexander.lomov(a)altoros.com> wrote:

That’s so handy. Thank you very much, Phil.

On Oct 14, 2015, at 8:14 PM, Whelan, Phil <phillip.whelan(a)hpe.com>
wrote:

Hi,

In case you missed it...

https://www.dropbox.com/s/4ueni9fb7w5tvgm/cab_14th_oct_2015.mp3?dl=0

Agenda / notes….

https://docs.google.com/document/d/1SCOlAquyUmNM-AQnekCOXiwhLs6gveTxAcduvDcW_xI/edit?pli=1#

Chat room...


drnic @ starkandwayne1: Hey hey all

drnic @ starkandwayne1: How do I get a calendar feed that has correct
dial in details?

anonymous1 morphed into goehmen

drnic @ starkandwayne1: Got dial in from agenda

Chris(a)IBM: where is all of this?

Marco N. @ Pivotal: @drnic, do you use google calendar? This should(?)
work?
https://www.google.com/calendar/embed?src=cloudfoundry.org_8ms13q67p9jjeeilng6dosnu50%40group.calendar.google.com&ctz=America/Los_Angeles

anonymous1 morphed into Steven Benario

drnic @ starkandwayne1: marco, thx will try

anonymous1 morphed into Cornelia

Simon Moser (co-chair) morphed into Simon Moser

Simon Moser morphed into Simon Moser @ IBM

anonymous1 morphed into lexsys @ altoros

drnic @ starkandwayne1: marco - that is calendar of PMC meetings

drnic @ starkandwayne1: Can someone please add the .ical feed for
meetings into the top of the agenda doc?

Marco N. @ Pivotal: @drnic, sorry I goofed. Try this:
https://www.google.com/calendar/embed?src=cloudfoundry.org_oedb0ilotg5udspdlv32a5vc78%40group.calendar.google.com&ctz=America/Los_Angeles

anonymous1 morphed into Steve Winkler @ GE

Marco N. @ Pivotal: If you accept, I'll put it in the agenda doc as well

drnic @ starkandwayne1:
https://github.com/cloudfoundry-incubator/diego-ssh

drnic @ starkandwayne1: marco, looks good

Marco N. @ Pivotal: cool

drnic @ starkandwayne1: added

pivotal room: Servcie Core update from Marco is now

drnic @ starkandwayne1: marco - your manifest fixes for registrar errand
will go into upstream broker-registrar repo? being able to select which
plans are public by default would be good

drnic @ starkandwayne1: shannon - cool re router access for brokers

drnic @ starkandwayne1: shannon super cool on multi ports

anonymous1 morphed into Sergey

drnic @ starkandwayne1: What is the URL to CAB call minutes/blog from
September? Which blog is it going to?

pivotal room: @Phil can you update Dr.Nic on last month's blog

anonymous2 morphed into Marco(a)Swisscom

drnic @ starkandwayne1: To get GH issues for conversations, I think ppl
need to "Watch" each repo https://github.com/cloudfoundry/go-buildpack

drnic @ starkandwayne1:
https://github.com/cloudfoundry/go-buildpack/issues/22

drnic @ starkandwayne1:
https://github.com/cloudfoundry/nodejs-buildpack/issues/32

drnic @ starkandwayne1: Are we going to continue using both Consul and
ETCD?

drnic @ starkandwayne1: why?

drnic @ starkandwayne1: remote urls for releases is lovely

drnic @ starkandwayne1: to see added

Marco N. @ Pivotal: Nobody's talking about the elephant on the call

julz: there's an elephant on the call? [
http://webconf.soaphub.org/conf/images/wink.gif]

Chris(a)IBM: aaahhhhhRRRRRrrrraaaahhhh

shinichiNakagawa(a)NTT: mute: *6

Phil Whelan: @drnic I wasn't able to write up the notes last month. I
sent a recording of the call to the cf-dev@ mailing list though

drnic @ starkandwayne1: amit, there is an existing bosh release for
route-registrar
https://github.com/cloudfoundry-community/route-registrar-boshrelease;
perhaps promote it if you want it; else we can drop it

Chris(a)IBM: @phil, we'll let it slip just this once

Chris(a)IBM: [http://webconf.soaphub.org/conf/images/wink.gif]

drnic @ starkandwayne1: @phil oh cool re a recording; will go look

Deepak Vij: Question for Simon Moser regarding project Flintstone that
talks about port to Ruby 2.2.3 & JRuby. It came to our attention that
current CC component does not take advantage of multi-CPU/Cores
environment. This may be due to the fact that Ruby MRI GIL provides
thread-safety guarantees though at the cost of overall performance
degradation. Is this the reason current ports to other Ruby versions are
being looked at? Deepak Vij Huawei

Simon Moser @ IBM: @Deepak Vij: yes, thats one reason, although ruby
2.2.3 and jRuby are two separate issues. 2.2.3 is just "up to date
maintenance", whereas jRuby really is aimed at improving performance

Simon Moser @ IBM: concurrency performance

Simon Moser @ IBM: I'll talk about the results and challenges so far
when I get called out in the call

Deepak Vij: Thanks simon

Phil Whelan: @drnic to save you searching [
http://webconf.soaphub.org/conf/images/wink.gif]
https://www.dropbox.com/s/t8xewz5vw708b5q/cab_9th_sept_2015.mp3?dl=0

drnic @ starkandwayne1: thx

Simon Moser @ IBM: @Jim Campbell: can you confirm that , by changing UPD
to TCP in the backend - that change won't affect any existing doppler
firehose clients - correct ?

drnic @ starkandwayne1: perhaps a shared bosh-lite isn't best
performance testing env

drnic @ starkandwayne1: jruby/jvm will want lots of ram to be happy?

drnic @ starkandwayne1: what performance improvements did upgrading to
mri 2.2.3 give?

Simon Moser @ IBM: we didn't measure around 2.2.3

drnic @ starkandwayne1: drmax - There is an Others section in agenda

Simon Moser @ IBM: and the bosh lite isn't shared, its dedicated for
this effort. We have two equal VMs, oneMRI and one jRuby to do the
comparison, but this I agree this isn't an ideal environment (it was
intended to give us a ballpark figure to make the decision, but we expected
a bigger difference than 20%)

drnic @ starkandwayne1: Simon - I mean the host vm is shared amongst all
warden containers

drnic @ starkandwayne1: garden

drnic @ starkandwayne1: and local networking, which you mentioned

drnic @ starkandwayne1: pretty quick to spin up dedicated CF - perhaps
try terraform-aws-cf-install repo

Simon Moser @ IBM: got it - yes, we are aware of that. We thought it
might be good enough for the ballpark, but maybe that was wrong

drnic @ starkandwayne1: MRI is constantly improving its performance

Simon Moser @ IBM: we should try that terraform thing

drnic @ starkandwayne1: And this comes from the person who promoted
JRuby to the world during my time at Engine Yard

Simon Moser @ IBM: lol

Simon Moser @ IBM: and yes, we're aware of the MRI improvements - like I
said, we expected bigger differences

drnic @ starkandwayne1: perhaps ping Jruby team - they might suggest
some tuning

Simon Moser @ IBM: ok

drnic @ starkandwayne1:
https://blog.starkandwayne.com/2015/10/08/introducing-spruce-a-more-intuitive-spiff/

drnic @ starkandwayne1:
https://blog.starkandwayne.com/2015/10/12/try-out-postgresql-9-5beta-on-cloud-foundry/

drnic @ starkandwayne1:
https://blog.starkandwayne.com/2015/09/29/deploying-subway-broker-with-bosh/

drnic @ starkandwayne1: https://github.com/maximilien/cf-swagger

drnic @ starkandwayne1: http://apidocs.cloudfoundry.org/

drnic @ starkandwayne1: ?

anonymous1 morphed into Jan Dubois

drnic @ starkandwayne1: xoxo all y'all

pivotal room: bye


Re: Multi-Line Loggregator events and the new Splunk "HTTP Event Collector" API

Mike Youngstrom
 


We have thrown around one approach which solves the problem but would
require changes in the runtime. That solution would expose a socket to the
container where the application could emit logs. The application would now
have control over what delimits a message.
Any thoughts on what protocol this socket would listen too? Raw
Dropsonde? Syslog?

I've been thinking about your questions regarding the '\' approach:

As Erik mentioned in the last thread, multi-line logging is something which
the loggregator team would like to solve. But there are a few questions to
answer before we can come up with a clean solution. We want a design which
solves the problem while not breaking existing apps which do not require
this functionality. Before implementing a solution we would also want to
answer if we want to do it for both runtimes or just Diego, since the way
log lines are sent to Metron differ based on the runtime.
I'd be perfectly happy if the solution was only for Diego. We are
surviving today but I think the feasibility of our current solution is
running out.


I guess the expectation is that loggregator would internally remove the
escape character.
I think this would be optimal.


This has performance implications because now some part of loggregator
will need to inspect the log message and coalesce the message with the
succeeding ones. We will need to do this in a way which respects
multi-tenancy. That means now we are storing additional state related to
log lines per app. We will also need to decide how long loggregator needs
to wait for the next lines in a multi-line log, before deciding to send the
line which it received. To me that's not a simple change.
Can you help me understand what you are referring to with "coalescing
messages" and "storing additional state related to log lines per app"? The
way I see it the current agents buffer until they see a '\n' then they
create an event. Adding an escaped line '\\n' the logic would be very much
the same as it is today, buffer until you find an unescaped new line. Then
unescape the remaining new lines.

Seems somewhat straight forward to me. Thoughts on considering a pull
request that does something like this?

Mike

7121 - 7140 of 9422