Date   

Re: how to get cf authentication token programmatically

Juan Antonio Breña Moral <bren at juanantonio.info...>
 

Good morning,

I understood your app scenario:

+ Your app has to interact with CF API but your question is about where you can store your configuration.

I think that it is very normal to use a config.json to configurate your App. If you don't like to put parameters in a plain file (e.g. config.json), another alternative is the usage of manifest.yml file.

Read this links:
http://stackoverflow.com/questions/17902803/how-do-you-set-environment-variables-on-cloud-foundry-v2
http://docs.run.pivotal.io/devguide/deploy-apps/environment-variable.html
https://docs.cloudfoundry.org/devguide/deploy-apps/manifest.html#env-block

Juan Antonio


Re: [abacus] Handling Notifications

subhash <sinduri@...>
 

Thanks for getting started on this requirement.

Yes, We were looking into webhooks kind of model, where a registered
endpoint could be invoked when a (or more than one) criteria is met.

A few examples of the criteria, would be a) when total usage/cost for an
org(or a set of orgs) reaches a threshhold or b) when a service usage
exceeds a threshold or c) the usage or cost for an instance reaches a
threshold.

As you can see, it would be helpful if we are be able to enable
notifications at different stages in the pipeline ...accumulated,
aggregated, rated, with the input being the documents at those levels.




--
View this message in context: http://cf-dev.70369.x6.nabble.com/cf-dev-abacus-Handling-Notifications-tp2678p2682.html
Sent from the CF Dev mailing list archive at Nabble.com.


Re: Pluggable Resource Scheduling

Gwenn Etourneau
 

Hi,

Interesting proposition, wondering if it make sense to hook into Diego or
CF.
Diego is connected to CF by the CC-Bridge (big picture) why not create a
CC-Bridge for other scheduling system ?



Thanks

On Thu, Nov 12, 2015 at 5:13 AM, Deepak Vij (A) <deepak.vij(a)huawei.com>
wrote:

Hi folks, I would like to start a discussion thread and get community
thoughts regarding availability of Pluggable Resource Scheduling within
CF/Diego. Just like Kubernetes does, wouldn’t it be nice to have an option
of choosing Diego native scheduling or other uber/global resource
management environments, specifically Mesos.



Look forward to comments and feedback from the community. Thanks.



Regards,

Deepak Vij

(Huawei Software Lab., Santa Clara)


[abacus] Updated CI build to continuously push our services to CF

Jean-Sebastien Delfino
 

Hi all,

Earlier today I added a build and a few scripts to our set of Travis CI
builds [1] to continuously push our Abacus services and apps to CF.

I'm hoping that will partially address some of our recent struggles with
setting up and testing on CF / BOSH-lite in our dev/test environments [2],
and where not deploying and testing on a full blown CF instance on every
commit would let integration issues slip through undetected.

That Travis CI build is currently using my own CF/Bluemix account, and it's
only running one of our integration tests as an experiment. I intend to
keep it that way in observation for now and if we see that this kind of
setup helps us then maybe we can move it somewhere else, include more tests
in that build (with a persistent DB, a few more app instances, perhaps our
app metering bridge too etc) in a few days.

I've also started to look at Concourse.ci [3], in hope of doing something
similar from a Concourse CI pipeline at some point.

HTH

[1] https://travis-ci.org/cloudfoundry-incubator/cf-abacus/builds/90570619
[2]
http://cf-dev.70369.x6.nabble.com/cf-dev-CF-deployment-environments-available-for-CF-incubating-projects-to-use-tt2091.html
[3] http://concourse.ci

- Jean-Sebastien


Re: [abacus] Handling Notifications

Jean-Sebastien Delfino
 

I've begun to work on supporting notifications in abacus.
Great! What you've described here looks like a good start to me. I've
included a few comments and questions below:

We already have a github issue and pivotal tracker story associated with
it as seen here:
https://github.com/cloudfoundry-incubator/cf-abacus/issues/161
https://www.pivotaltracker.com/story/show/101097834
and one more, created from Github #161 by the CF gitbot:
https://www.pivotaltracker.com/n/projects/1385414/stories/107779224
I'm guessing that's the one that'll track the commits under Github #161.

There are two ways that this will probably need to work. One via a
webhook and another via letting a client keep and open connection that
would receive a stream of notifications.

I've asked Subhash (@sinduri, author of Github #161) to post more details
of what he needs here. I'm guessing it's Webhooks but will let him
elaborate.

For the registry, I have considered Eureka <...> I'm not sure if also
using it as a registry for handling notifications would kind of warp the
purpose of its use.

IMO Eureka works fine as a registry of IP addresses for performance
monitoring of our services, but I'm not so sure about it for persistent
registrations of Webhooks and corresponding triggers/criteria, as AIUI
that'd a big deviation from what it's originally designed for. So +1 to
what you said about warping its purpose :)

Documents would probably be keyed to the given URL just for supporting
updates on new criteria for a certain callback URL and prevent multiple
documents for the same URL.

How about keying by criteria as well to know when to trigger a Webhook
call? and maybe allow multiple registrations per URL? (e.g please call me
back at http://foo.com/bar on new usage for org abc, and also when usage
for app xyz goes over 1000, would probably be two separate registration
docs for a single URL)

If a call is an error, it'd be best to retry for a set amount of
time/tries until that particular URL is marked unreachable (and thus isn't
called in subsequent notifications).

+1 for a sort of quarantine on an unreachable Webhook. How about slow
Webhooks causing back pressure problems? Would we quarantine these too?


... And a few more questions that would be good to discuss here as well:

- should we let the rating service app do this or have a separate
notification service app?

- with partitioning of the orgs across multiple deployments of our apps
(for scalability or regional deployments for example) do I need to first
find the right service to register with (e.g. register with the us-south
notification service or the eu-west notification service)? or can I
register with a central notification service that will then figure out
which deployment instance will call me back?

- how do we secure the registration calls and Webhook callbacks?

- do we replay notifications when we can't deliver them?

- can I register to receive all notifications from a certain point in the
logical stream of notifications matching a criteria (e.g. call me back if
this org consumed too much per hour at any point since last week)

- do we have specific examples of notification trigger criteria? (I've
asked Subhash to post some here to initiate that discussion).

More thoughts?

- Jean-Sebastien

On Wed, Nov 11, 2015 at 1:21 PM, Benjamin Cheng <bscheng(a)us.ibm.com> wrote:

I've begun to work on supporting notifications in abacus. We already have
a github issue and pivotal tracker story associated with it as seen here:
https://github.com/cloudfoundry-incubator/cf-abacus/issues/161
https://www.pivotaltracker.com/story/show/101097834
Abacus needs to have some support of sending a notification to some client
based upon some criteria being met.

Right now, I'm in an exploratory phase. There are two ways that this will
probably need to work. One via a webhook and another via letting a client
keep and open connection that would receive a stream of notifications.
Here's a very basic breakdown.

In terms of the webhook, there's two things to consider:
-Maintaining the information of the clients and their criteria
-Calling the provided callback URLs when a certain criteria is met

For the registry, I have considered Eureka(specifically, the module inside
of abacus) since the registration spec has a metadataType property that
hold a sequence of anything. Eureka is used as a registry for middle tier
services(or application instances in our case), so I'm not sure if also
using it as a registry for handling notifications would kind of warp the
purpose of its use. There's also some other things such as the heartbeats
that I'm not sure would be the right fit in terms of a webhook.
Maintaining the list of the URLs in a database with a separate
"Notification Registry/Hander" app could work as well. Documents would
probably be keyed to the given URL just for supporting updates on new
criteria for a certain callback URL and prevent multiple documents for the
same URL. Each document would hold a list of criteria if that's the case.

When some usage gets rated, that usage would be checked against the list
of URLs. All the matching URLs would be called in parallel. If a call is
successful, then there's nothing additional that needs to be done. If a
call is an error, it'd be best to retry for a set amount of time/tries
until that particular URL is marked unreachable(and thus isn't called in
subsequent notifications).

I haven't begun on the other avenue of keeping an open connection but
wanted to start a discussion with just the webhook for now.

Any thoughts, comments, or suggestions on this?


[abacus] Handling Notifications

Benjamin Cheng
 

I've begun to work on supporting notifications in abacus. We already have a github issue and pivotal tracker story associated with it as seen here:
https://github.com/cloudfoundry-incubator/cf-abacus/issues/161
https://www.pivotaltracker.com/story/show/101097834
Abacus needs to have some support of sending a notification to some client based upon some criteria being met.

Right now, I'm in an exploratory phase. There are two ways that this will probably need to work. One via a webhook and another via letting a client keep and open connection that would receive a stream of notifications. Here's a very basic breakdown.

In terms of the webhook, there's two things to consider:
-Maintaining the information of the clients and their criteria
-Calling the provided callback URLs when a certain criteria is met

For the registry, I have considered Eureka(specifically, the module inside of abacus) since the registration spec has a metadataType property that hold a sequence of anything. Eureka is used as a registry for middle tier services(or application instances in our case), so I'm not sure if also using it as a registry for handling notifications would kind of warp the purpose of its use. There's also some other things such as the heartbeats that I'm not sure would be the right fit in terms of a webhook.
Maintaining the list of the URLs in a database with a separate "Notification Registry/Hander" app could work as well. Documents would probably be keyed to the given URL just for supporting updates on new criteria for a certain callback URL and prevent multiple documents for the same URL. Each document would hold a list of criteria if that's the case.

When some usage gets rated, that usage would be checked against the list of URLs. All the matching URLs would be called in parallel. If a call is successful, then there's nothing additional that needs to be done. If a call is an error, it'd be best to retry for a set amount of time/tries until that particular URL is marked unreachable(and thus isn't called in subsequent notifications).

I haven't begun on the other avenue of keeping an open connection but wanted to start a discussion with just the webhook for now.

Any thoughts, comments, or suggestions on this?


Pluggable Resource Scheduling

Deepak Vij
 

Hi folks, I would like to start a discussion thread and get community thoughts regarding availability of Pluggable Resource Scheduling within CF/Diego. Just like Kubernetes does, wouldn’t it be nice to have an option of choosing Diego native scheduling or other uber/global resource management environments, specifically Mesos.

Look forward to comments and feedback from the community. Thanks.

Regards,
Deepak Vij
(Huawei Software Lab., Santa Clara)


Re: applications on CF cannot download files from internet

Rob Dimsdale
 

CF 170 does not have application security groups, so that is not the issue. However, CF 170 is quite old now - we would be in a better position to assist if you are running a more recent version of cf-release.


[cf-abacus] Adding Node 5 to the Abacus CI build

Jean-Sebastien Delfino
 

Hi all,

I'm going to add Node 5 to our CI build later today, as that'll help detect
issues with it early on.

I'll also update our package.json files to require recent versions of Node
and Npm (as they currently say >= 0.10, although most of us use 0.12 or
4.2).

If you're still using Node 0.10 (well... you should upgrade, really) then
please post here and I'll try to see if we can still keep a Node 0.10 build
around.

With Node 5 I'm currently seeing an issue in our doc generator script (some
missing CSS, IIRC Hristo mentioned something like that too last week on the
chat) so I'll try to fix that too to get that Node 5 build going.

HTH

- Jean-Sebastien


Re: Failed compiling packages during cf-215 upgrade in bosh director

Rob Dimsdale
 

When we see issues like this, we often find that retrying fixes the issue (i.e. the failures are not reliably reproducible - often caused by temporary networking issues).

If retrying does not fix the issue, let us know and we can attempt some further investigation.


Re: how to get cf authentication token programmatically

Zuba Al <myfakename90@...>
 

Actually my app needs to get info of other apps running in same cf api domain. To get this info, I'm calling APIs of other apps using HTTP.GETrequest with authentication token in request header. And to get this authentication token, I'm using HTTP:POST request to 'AuthorizationURL/oauth/token' with username and password in the request header.
1.CF environment variables doesn't have all the info needed and so I'm calling app's API directly and getting needed info from the returned JSON.
2. There is a config.json file in my local system in "C:\Users\MyUser\.cf" path and has cf target api and auth token, but it gets populated only when I'm logged into cf in command line. My app once deployed and running in cf, should get the info of other apps even if I'm not logged in. So getting auth token from local system's config.json is not a good idea.

Please suggest me another way to get auth token.


Re: OpenAM integration

Antonio Diaz Arroyo
 

I will try, many thanks Sree!!


unlock local Cloud Foundry user. uaac?

Lukas Lehner <weblehner@...>
 

Hi

see
http://stackoverflow.com/questions/33648724/unlock-local-cloud-foundry-user-uaac

as a workaround I changed password.

gem install cf-uaac
uaac target https://...
uaac token client get admin
uaac password set


Re: how to get cf authentication token programmatically

Juan Antonio Breña Moral <bren at juanantonio.info...>
 

Good morning,

Did you think in the alternatives:

+ a config file
+ CF Environment variables

If you connect with PWS or Bluemix, you are using

https://login.run.pivotal.io
https://login.eu-gb.bluemix.net/UAALoginServerWAR

so your application use a https connection from your apps to UAA and your data send in a secure way.

What is the problem from your point of view?

Juan Antonio


how to get cf authentication token programmatically

Zuba Al <myfakename90@...>
 

My CF app needs to get cf authentication token programmatically in runtime. Currently I'm getting the token using HTTP:POST request to 'AuthorizationURL/oauth/token' with username and password in the request header. But I dont want to put my username password in code. So need another way to get this auth token.
Note: I'm using this token to call other CF apps' API via HTTP request.


Re: CF CAB call for November is RESCHEDULED to Wednesday Nov. 18th, 2015

Michael Maximilien
 

Hi, all,

Since tomorrow is Vereran's Day (a federal holiday here in US) which celebrates the service of all U.S. military veterans, we've agreed to move the call to next week: Wednesday November 18th.

Thanks to Steve Winkler for noticing and mentioning. We apologize for this last minute change and talk to you all next week.

Best,

dr.max
ibm cloud labs
sillicon valley, ca

Sent from my iPhone

On Nov 10, 2015, at 2:07 AM, Michael Maximilien <maxim(a)us.ibm.com> wrote:

Hi, all,

Quick reminder that the CAB call for September is this week Wednesday November 11th @ 8a PDT.

Please add any project updates to Agenda here: https://docs.google.com/document/d/1SCOlAquyUmNM-AQnekCOXiwhLs6gveTxAcduvDcW_xI/edit#heading=h.o44xhgvum2we

If you have something else to share, please also add an entry at the end.

Best,

Chip, James, and Max


Re: Cloud Controller - s3 encryption for droplets

William C Penrod
 


Re: multiple nozzles on firehose

Rohit Kumar
 

Both of the nozzles will get all the events coming from the firehose.

On Tue, Nov 10, 2015 at 4:51 PM, Nix, Lon <Lon.Nix(a)schwab.com> wrote:

That answers my question, thanks!



Out of curiosity, what happens if nozzles with different subscription IDs
subscribe to the same firehose?



--Lon Nix



Re: multiple nozzles on firehose

Nix, Lon <Lon.Nix@...>
 

That answers my question, thanks!

Out of curiosity, what happens if nozzles with different subscription IDs subscribe to the same firehose?

--Lon Nix


Re: Immutability of applications

john mcteague <john.mcteague@...>
 

I think there are several ways this could be approached if there was
appetite to do so.

1. Feature flag that denies mutable operations on a running application
(doesnt help every scenario as some binds only affect restage)
2. Auto restart of apps on change
3. Fully model the distinction between the thing I want to run (desired
state) and the thing I am running (current state) in the API model (I
appreciate this is a massive change)

And I'm sure there are others I haven't considered yet. A purest view makes
me favour number 3.

Within our org we start to build layers on top of CF to enforce the
immutability of applications, so direct API usage gets locked down, but
thats a very black and white approach which prevents valid use cases of the
API by end users.

On Tue, Nov 10, 2015 at 11:31 PM, Marco Nicosia <mnicosia(a)pivotal.io> wrote:

Hi John, thanks for getting back to me.

It seems inconvenient for you to have to monitor divergence of
CF-initiated configuration changes from the Apps' runtime environment.

I'm not PM for these projects, so I get to ask leading questions, but
obviously I can't make any commitments. If you have a question about MySQL,
I'm your guy. :)

It seems like the two areas of exposure are `cf set-env` and `cf
bind-service`. Would it make more sense that, in your environment, any time
either of those are executed the app is _automatically_ restarted (if and
only if the app is already running)? If it turns out to be the case that we
make several, simultaneous changes to environment and bindings, we could
look into batching them together, so there isn't a serial string of rolling
restarts.

That way you wouldn't have to build any additional auditing capabilities;
it theoretically wouldn't be possible for the environment, buildpacks
settings, and service configuration to diverge from what the app is
actively leveraging?

--
Marco Nicosia
Product Manager
Pivotal Cloud Foundry



On Tue, Nov 10, 2015 at 2:13 PM, john mcteague <john.mcteague(a)gmail.com>
wrote:

Marco, So I understand that configuration for an app is mutli-faceted and
not the complete view of what is happening. My goal is to understand what
is running at any moment in time from the information the API provides me.

Buildpack's can be heavily influenced by env var changes. Developers can
affect the version of java or the JVM args via env vars, but it is not
possible via the API to be sure from the variables presented whether they
are those that actually took effect at startup/restage. This causes an
audit concern.

Where an app adjusts its settings based on Redis being bound for example,
that is auditable by the fact we wrote the app and know what is due to
happen upon binding. But because binding can occur on a running app and
that is not guaranteed to be effective immediately, from the API output,
the developer may assume the application is running with that Redis
specific configuration. But that is only true if the app was restarted.

CF cant control every aspect of the apps configuration, but where it is
the method of configuration (bindings, env vars), I feel its important that
appropriate distinctions between actual and desired state are made.

On Tue, Nov 10, 2015 at 5:08 PM, Marco Nicosia <mnicosia(a)pivotal.io>
wrote:

John,

I'd like to understand more about what you are hoping to accomplish.

Are you trying to detect gaps where a new env var has been pushed, but
the app has not been restarted? Is this for monitoring/compliance or
debugging purposes?

It seems like Apps these days get configuration from a pretty wide
variety of sources, env variables being only one. Apps also ship with
defaults, UNIX defaults (max filedes, semaphores, etc), and apps sometimes
adjust their settings based on values in SQL or Redis*, etc.

Each app cares about these settings differently, some are sensitive to
RAM available, but not number of open files, etc.

I only ask because it seems like asking for "effective environment" vs
"actual environment" is only a subset of configurations.

The infrastructure can only express what has been provided to the app,
but for any number of reasons, may not represent the current actual running
state of the app.

I understand that it means more work for the app dev, but I often
implement a quick "/config" endpoint in my apps. This allows me to verify
what configuration the App is actually using at a given moment.

It's a more comprehensive way of validating that all "intended
configuration," is in fact actual configuration.

--
Marco Nicosia
Product Manager
Pivotal Cloud Foundry

* A quick example: the NOC can use an admin endpoint to toggle on/off an
experimental feature, which is detected via Redis, etc.

On Monday, November 9, 2015, Dieu Cao <dcao(a)pivotal.io> wrote:

Hi John,

I had been thinking about this a bit more, and I think it would be
reasonable to consider adding an end point that is similar to the /stats
end point, that queries Diego, to get actual state on environment variables
and as part of that VCAP_SERVICES for the running app instances.
Thoughts on that approach? Would that help address the problem?

-Dieu

On Wed, Nov 4, 2015 at 9:03 AM, john mcteague <john.mcteague(a)gmail.com>
wrote:

I had this conversation with a few different people during the berlin
summit and promised one of them I would repeat it on the mailing list today
to get further feedback.

Today, once we push an application, the droplet is immutable. It
doesnt change until you push the application again or restage. I believe
the entire container could change without a new push if you upgrade the
rootfs and restart all the apps (which the CF operator would do).

However, the environment vars and service bindings can be changed on
an application but they would not take affect until the next restart.The
CF API would report these changes as active when you run *cf env *or *cf
services. *There is no distinction between desired state and current
state when using the API.

To me this is a significant gap as we cannot necessarily get a true
view of the world (i call cf set-env but dont restart the app, how do I
know from the API what value of that env var my app is using).

How are people addressing this in their own environments and is it
something that the core API team should be considering (I ask the latter
publicly even though I asked Dieu during the summit :) ).

John
--
--
Marco Nicosia
Product Manager
Pivotal Software, Inc.
mnicosia(a)pivotal.io
c: 650-796-2948


6741 - 6760 of 9429