Date   

Re: Stemcells, Garden, and rootfs: how are these related?

Steven Benario
 

For a bit more information on how Windows does containerization (hint: it
doesn't involve a rootfs), you can check out a recent blog post from the
Greenhouse team:
https://engineering.pivotal.io/post/windows-containerization-deep-dive/

On Thu, May 19, 2016 at 11:47 AM, Duncan Winn <dwinn(a)pivotal.io> wrote:

sorry JK - it's still early am (ignore what I just said) - the stack is
the specific rootfs - the container image is a combination of a droplet
plus a stack. The container image will then run in your garden process.
For a windows stack you would need a windows vm/cell to run it.

On Thu, May 19, 2016 at 8:43 AM Duncan Winn <dwinn(a)pivotal.io> wrote:

Hey JK, a stack is a combination of the rootfs and droplet. This is
conceptually a "container image" that will then run in your garden
container. For a windows stack you would need a windows vm/cell to run it.

On Thu, May 19, 2016 at 8:26 AM J K <falconwing(a)gmail.com> wrote:

Thank you, Amit. Is it correct to say that a stack (in the `cf stacks`
sense) is a combination of rootfs + stemcell? If you have a "base VM" of,
say, Linux -- and if that's the one bosh is deploying to all the cells, how
does that become a `windows2012R2` stack in CF?
--
Duncan Winn
Cloud Foundry PCF Services
--
Duncan Winn
Cloud Foundry PCF Services


Re: Stemcells, Garden, and rootfs: how are these related?

Duncan Winn
 

sorry JK - it's still early am (ignore what I just said) - the stack is the
specific rootfs - the container image is a combination of a droplet plus a
stack. The container image will then run in your garden process. For a
windows stack you would need a windows vm/cell to run it.

On Thu, May 19, 2016 at 8:43 AM Duncan Winn <dwinn(a)pivotal.io> wrote:

Hey JK, a stack is a combination of the rootfs and droplet. This is
conceptually a "container image" that will then run in your garden
container. For a windows stack you would need a windows vm/cell to run it.

On Thu, May 19, 2016 at 8:26 AM J K <falconwing(a)gmail.com> wrote:

Thank you, Amit. Is it correct to say that a stack (in the `cf stacks`
sense) is a combination of rootfs + stemcell? If you have a "base VM" of,
say, Linux -- and if that's the one bosh is deploying to all the cells, how
does that become a `windows2012R2` stack in CF?
--
Duncan Winn
Cloud Foundry PCF Services
--
Duncan Winn
Cloud Foundry PCF Services


Re: Stemcells, Garden, and rootfs: how are these related?

Duncan Winn
 

Hey JK, a stack is a combination of the rootfs and droplet. This is
conceptually a "container image" that will then run in your garden
container. For a windows stack you would need a windows vm/cell to run it.

On Thu, May 19, 2016 at 8:26 AM J K <falconwing(a)gmail.com> wrote:

Thank you, Amit. Is it correct to say that a stack (in the `cf stacks`
sense) is a combination of rootfs + stemcell? If you have a "base VM" of,
say, Linux -- and if that's the one bosh is deploying to all the cells, how
does that become a `windows2012R2` stack in CF?
--
Duncan Winn
Cloud Foundry PCF Services


Re: Stemcells, Garden, and rootfs: how are these related?

J K
 

Thank you, Amit. Is it correct to say that a stack (in the `cf stacks` sense) is a combination of rootfs + stemcell? If you have a "base VM" of, say, Linux -- and if that's the one bosh is deploying to all the cells, how does that become a `windows2012R2` stack in CF?


Re: Stemcells, Garden, and rootfs: how are these related?

Amit Kumar Gupta
 

Hi J K,

To add to what Nick said:

You can think of Garden as a generic API for something that creates
containers and runs processes in them, and can be backed by various
different backend implementations, e.g. garden-linux or garden-windows. It
leaves the responsibility of what it means to actually create a container
to the backend, but provides an interface for interacting with containers
and container processes as logical resources.

At a high level, a container consists of a root file system image, a set of
resource constraints, and some commands to run inside the container. Cloud
Foundry is able to take your raw application source code and build it
inside a container. The root file system used for the container that
builds your code is the "rootfs". So for the build process, the standard
"cflinuxfs2" rootfs is used to create the container, then your code is
injected and compiled, and the resulting asset is extracted. Next, when
Cloud Foundry runs your application, it again creates a container starting
from the rootfs image, injects your built code, and runs it.

The "create a container and run xyz" responsibility belongs to Garden, so
the relationship between garden and the rootfs is that Cloud Foundry finds
a Garden process somewhere and instructs it to use a particular rootfs to
use to create a container out of it.

A single application developer wants to push their code to Cloud Foundry
and have it run the code in some container. But an instance of the Cloud
Foundry platform can support many (e.g. hundreds of thousands) application
developers all doing the same thing. An operator managing an instance of
Cloud Foundry needs to take Cloud Foundry software itself and have it be
deployed and run by something. The toolchain for this is called BOSH.

BOSH typically operates on VMs in a virtualized infrastructure like AWS of
vSphere. BOSH can run Cloud Foundry, or any number of different
distributed software services, on top of virtualized infrastructure.
Analogous to the rootfs which is a common base layer image for all
applications running on Cloud Foundry, a stemcell is a common base VM image
for all services deployed by BOSH (e.g. each of the components comprising
Cloud Foundry itself).

The documentation Nick has linked you to can give you more detailed
information about these concepts, as well as the reasoning behind them.

Best,
Amit

On Wed, May 18, 2016 at 4:53 PM, Nicholas Calugar <ncalugar(a)pivotal.io>
wrote:

Hi JK,

I don't work on any of these parts of Cloud Foundry but I can give a very
basic overview:

[1] rootfs: the filesystem used for applications deployed on Cloud
Foundry. The current default is cflinuxfs2, based on Ubuntu 14.04

[2] garden: "A rich golang client and server for container creation and
management with pluggable backends for linux, windows and The Open
Container Initiative Spec". Garden is running on the Diego cells where
Cloud Foundry applications are run.

[3] stemcells: The base image for VMs deployed by bosh.


Highly recommend staring with the Cloud Foundry documentation:
http://docs.cloudfoundry.org/. You can also "kick the tires" by trying
one of the hosted versions of Cloud Foundry:
https://www.cloudfoundry.org/learn/certified-providers/


Nick


[1] https://github.com/cloudfoundry/stacks
[2] https://github.com/cloudfoundry-incubator/garden
[3] http://bosh.cloudfoundry.org/docs/stemcell.html

On Wed, May 18, 2016 at 1:00 PM, J K <falconwing(a)gmail.com> wrote:

hi,

I'm learning about Cloud Foundry's ideas of how to organize applications
and how it works under the covers. At the moment I'm having a little
trouble understanding the relationship between a few ideas in CF: "rootfs",
"Garden", and "stem cells".

Can someone explain:

(1) their understanding of what these ideas are; and
(2) how they are related?

Thank you!


Re: Buildpacks Checksum Site for Release Validation

Gwenn Etourneau
 

How the pws / cloud-ops team verify the buildpack checksum before deploying
it ?

On Thu, May 19, 2016 at 3:57 AM, Danny Rosen <drosen(a)pivotal.io> wrote:

In the future we may consider implementing a json feed. If we did go
forward with this work, would you be interested in creating example CI / CD
implementations as references?

On Tue, May 17, 2016 at 10:30 PM, Gwenn Etourneau <getourneau(a)pivotal.io>
wrote:

Hi,
Any json feed /api ?
Can be nice and more easy to integrate with any CI/CD tool.

Thanks

On Wed, May 18, 2016 at 11:15 AM, taichi nakashima <nsd22843(a)gmail.com>
wrote:

Great,

I hope cloudfoundry/cli will provide the same thing.

cf.
https://lists.cloudfoundry.org/archives/list/cf-dev(a)lists.cloudfoundry.org/thread/K3BEBY4A2WSUKS7YS5IF2UDQHHSU35A7/

Taichi Nakashima

2016年5月18日(水) 6:20 David Jahn <djahn(a)pivotal.io>:

Dear Cloud Foundry Users,

To help operators and users of Cloud Foundry establish a "chain of
custody" for buildpacks, we have launched the following checksum site:

https://buildpackverify.cloudfoundry.org

This site provides a checksum for all cached buildpack release zip
files (except for the java-buildpack). Whenever the buildpacks team
generates a new buildpack release, we will immediately compute the SHA256
checksum of that file and upload it to this website.

The site is hosted on a different repository from the main buildpack
github repositories. It allows operators to validate that the zip file we
produced is the same artifact that has been downloaded and installed.

Additionally, if an operator wishes to further investigate the
components of a buildpack, the "manifest.yml" in each buildpack root
directory (for
example,
https://github.com/cloudfoundry/go-buildpack/blob/master/manifest.yml)
provides a catalog of every third party component in the buildpack, a URL
of that component's location, and an MD5 checksum of that component.

We hope that this will assist people in auditing the source of their
buildpack code!

Cheers,
Buildpacks Team


Re: Proposal: Reducing State in Service Brokers - Service Broker API Enhancement

Long Nguyen
 

Oh interesting! I was thinking about creating a broker that can do same thing, Are you looking to open source this broker?

On May 17, 2016 at 12:02:48 PM, Alex Ley (aley(a)pivotal.io) wrote:

hat builds lots of service brokers. We are currently working on broker that backs onto BOSH and want to move towards making it stateless. We have written a proposal to enhance the CF Service Broker API to allow us to achieve this. We believe this will he


Re: Stemcells, Garden, and rootfs: how are these related?

Nicholas Calugar
 

Hi JK,

I don't work on any of these parts of Cloud Foundry but I can give a very
basic overview:

[1] rootfs: the filesystem used for applications deployed on Cloud Foundry.
The current default is cflinuxfs2, based on Ubuntu 14.04

[2] garden: "A rich golang client and server for container creation and
management with pluggable backends for linux, windows and The Open
Container Initiative Spec". Garden is running on the Diego cells where
Cloud Foundry applications are run.

[3] stemcells: The base image for VMs deployed by bosh.


Highly recommend staring with the Cloud Foundry documentation:
http://docs.cloudfoundry.org/. You can also "kick the tires" by trying one
of the hosted versions of Cloud Foundry:
https://www.cloudfoundry.org/learn/certified-providers/


Nick


[1] https://github.com/cloudfoundry/stacks
[2] https://github.com/cloudfoundry-incubator/garden
[3] http://bosh.cloudfoundry.org/docs/stemcell.html

On Wed, May 18, 2016 at 1:00 PM, J K <falconwing(a)gmail.com> wrote:

hi,

I'm learning about Cloud Foundry's ideas of how to organize applications
and how it works under the covers. At the moment I'm having a little
trouble understanding the relationship between a few ideas in CF: "rootfs",
"Garden", and "stem cells".

Can someone explain:

(1) their understanding of what these ideas are; and
(2) how they are related?

Thank you!


Re: User defined variable "key" validation doesn't happen at cf set-env phase

Nicholas Calugar
 

No, we are trying to focus on development of the V3 API, we will only fix
major issues with the V2 API.

Thanks,

Nick

On Sun, May 15, 2016 at 8:44 PM, Padmashree B <padmashree.b(a)sap.com> wrote:

Hi Nick,

Thanks for the update!
Are there any plans of bringing it to V2 APIs as well?

Kind Regards,
Padma


Re: aligning cf push health-check default value

Nicholas Calugar
 

Hi Dies,

The CC doesn't see these as steps of a push, they are all distinct API
requests. We cannot make decisions based on previous requests or lack
thereof, it has to be done in the CLI.

Based on a quick test, unknown fields are ignored in POST bodies in V2, so
you can send "health_check_type":"none" on older CC versions.


Thanks,

Nick

On Sun, May 15, 2016 at 5:23 PM, Koper, Dies <diesk(a)fast.au.fujitsu.com>
wrote:

Hi Nick, Eric,



I have confirmed that with the current CC release we could change the
default behaviour on the CLI side without making additional API calls:

In the body of POST /v2/apps?async=true we simply add
"health_check_type":"none" if the user specified “--no-route” and that
seems to be accepted regardless whether the app defaults to DEA or Diego.

Can you confirm CC does not return an error on older CC versions (before
the health_check_type attribute was introduced) if we add this attribute to
the API request?



Also, can you confirm you definitely don’t want CC to change its default
behaviour to align with what the user (and CLI) sees compared with Diego?



The current API sequence the CLI uses is:

1. POST /v2/apps?async=true – create app

2. POST /v2/routes?async=true&inline-relations-depth=1 – create
route(s)

3. PUT /v2/apps/guid/routes/guid – map route(s)

4. PUT /v2/resource_match & PUT /v2/apps/guid/bits?async=true –
upload app

5. GET /v2/jobs/guid – check its status



Although at step 1 CC would not know whether the app will get any routes
(i.e. whether the user specified “--no-route”), at step 4 CC would, so it
could toggle the health-check-type accordingly. Not sure how CC does this
for apps deployed to DEA – is it the DEA backend that makes this decision
at the timing of 4?



Regards,

Dies Koper
Cloud Foundry CLI PM





*From:* Nicholas Calugar [mailto:ncalugar(a)pivotal.io]
*Sent:* Thursday, May 12, 2016 4:34 AM
*To:* Discussions about Cloud Foundry projects and the system overall.
*Subject:* [cf-dev] Re: Re: Re: aligning cf push health-check default
value



Hi Dies,



I spoke with Eric and he explained that this could be the desired UX for a
majority of the apps pushed with --no-route. There are more advanced
deployment strategies that might set --no-route and bind to routes later,
but I think we can expect these users to be explicit with their health
check as well. I think my discomfort with this arose when you mentioned to
me that we might want to do this in the Cloud Controller. As long as this
continues to be explicit from the API perspective, I'm fine with changing
the UX of the CLI per your above proposal.





Thanks,



Nick



On Wed, May 4, 2016 at 1:13 PM, Shannon Coen <scoen(a)pivotal.io> wrote:

Hi Dies,



IMO the healthcheck of the app should be determined independently of
whether a developer wants their app routable.



My understanding of the implications for you proposal are that a developer
could not have a port-based healthcheck without mapping a route. This seems
unnecessarily restrictive. Soon developers will be able to specify http
healthchecks. Would these be prevented also?



Best,


Shannon Coen

Product Manager, Cloud Foundry

Pivotal, Inc.



On Wed, May 4, 2016 at 12:26 PM, Nicholas Calugar <ncalugar(a)pivotal.io>
wrote:

Hi Dies,



I considered this a bit more after we chatted yesterday and I don't think
we should try to create parity between DEAs and Diego in this case. My
personal opinion is that behavior should be explicit and these two flags
provide a more correct experience with the Diego backend.





Thanks,



Nick



On Mon, Apr 25, 2016 at 6:09 PM, Koper, Dies <diesk(a)fast.au.fujitsu.com>
wrote:

Hi CLI users,



With apps deployed to DEAs, a health check is performed at application
start-up targeting the app’s port, unless you specified `--no-route`, in
which case the process is monitored.

With Diego, the health check is performed continuously and the type of
check was exposed through an option to the `cf push` command.

This option defaults to `port`, which isn't always appropriate for apps
pushed without a route, such as worker apps.



We propose fixing the `--health-check-type` option’s default value to
align with the behaviour seen for DEAs, i.e. to use “none” if option
`--no-route` is used:



--health-check-type, -u Application health check type (Default:
'none' if '--no-route' is set, otherwise 'port')`

Would anyone object to such change?



Cheers,

Dies Koper
Cloud Foundry CLI PM











Re: [abacus] Separate time-based from discrete usage metrics

Hristo Iliev
 

Hi,

When we talk for a DB storing stateful metrics, do we really mean a single
DB storing all the Abacus pipeline data, or an input & output DBs for each
of the Abacus micro-services?

+1 for storing the data in the historical/log-like databases. This gives us
the possibility to extend the implementation of the failed events
management to the stateful measures.

We already started a spike on dataflow module. It would be straightforward
to detect stateful metrics in POST requests. This can be done by extending
the account plugin API and the metering config & schema.

We have two challenges:
* GET requests do not have access to the stateful flag, so we need a way to
detect stateful data using the document id. The idea we have is to use a
new ID schema (or just a prefix?), as you proposed on the last IPM.
* We think Replay function might miss some of the data, exactly due to the
problem we try to solve.

Regards,
Hristo Iliev

2016-05-18 19:34 GMT+03:00 Jean-Sebastien Delfino <jsdelfino(a)gmail.com>:

Hi,

To fix the issue we decided to:
1. Distinguish between time-based (linux-container) and discrete usage
metrics (the rest basically)
2. Store the time-based metrics in a separate DB(s)
Your proposal looks good to me. Piotr, Kevin and Raj and I had several
design discussions on this topic in the last few days and we've come up
with a few more ideas on top of what you're describing here:

- The distinction between time-based and discrete resource usage metering
could also be understood as usage metering of a stateful vs stateless
resource or metric. In the stateful case, we simply store the state of the
resource instance in a separate DB like you're proposing (e.g. we store the
fact that an app or a container is currently running, or has stopped), and
update that state in place when it changes. Then to compute and report
usage later on we just need to the current resource instance state from
that DB.

- We could continue to store the metrics in the current historical
databases as well (on top of that new DB) to preserve the resource instance
history as many users typically want to know their past usage.

- Some of us were not sure if the time-based / discrete distinction should
be at the resource type level or at the metric level... IMO your proposal
to do that at the metric level is cleaner so I'm happy with it :)

- The dataflow module will probably need a few minor code changes to
detect the case where some of the output docs need to go to a separate DB
(IIRC you or someone else also mentioned that on one of our scrums or on
slack...)

- Like you said, we may still need to maintain 2 DBs to purge old entries.
If that's easier, we could also adjust the usage accumulator service and
the dataflow module a bit to delete entries for inactive resource instances
right away (e.g. when an app or container stops.)

Thoughts?

P.S. I'll add these comments to issue #88 as well to make it be easier to
follow up there.

- Jean-Sebastien

On Thu, May 12, 2016 at 5:54 AM, Hristo Iliev <hsiliev(a)gmail.com> wrote:

Hi,

We're trying to fix Abacus issue 88: Missing aggregated usage for the
running application [1].

Background
=========

See the jsdelfino comment in the GitHub issue [2]. TL;DR: Resource
providers have to send a 'ping' doc per month for time-based metrics.

Proposed solution
==============

We decided to implement a solution in Abacus that frees the usage
providers from sending the 'ping' submission.

To fix the issue we decided to:
1. Distinguish between time-based (linux-container) and discrete usage
metrics (the rest basically)
2. Store the time-based metrics in a separate DB(s)

We already drafted a proposal for adding measurement type in the usage
plans with PR #320 [3].

We're about to spike on storing the time-based metrics in their own
Database, but we wanted to get the community opinion on the topic.

Motivation
========

The discrete usage submitted to Abacus is:
* stored in partitioned databases, due to their size/number
* like an event log, storing the history of the usage/resources

In contrast the current time-based metrics are:
* limited number (usually around 2 million on a loaded CF system)
* storing just the app resources usage state (GB/h consumed so far, GB/h
consuming currently)

Therefore it looks like a good idea to separate the two usage metrics
types and store the time-based metrics in a separate database. This will
allow us not only to solve the issue, but also to store and query the data
more effectively.

We may still need to maintain 2 databases and swap new/old (irrelevant)
metrics to reduce the DB size on the month boundaries.


Regards,
Hristo & Adriana

[1] https://github.com/cloudfoundry-incubator/cf-abacus/issues/88
[2]
https://github.com/cloudfoundry-incubator/cf-abacus/issues/88#issuecomment-148498164
[3] https://github.com/cloudfoundry-incubator/cf-abacus/pull/320


Re: Team

Amulya Sharma <amulya.sharma@...>
 

THUMBS UP Layne Peng

On Tue, May 17, 2016 at 11:51 PM Layne Peng <layne.peng(a)emc.com> wrote:

but we all are saying same thing .. believe me after running multiple
bosh
services managing marketplace in a large org is kind of a messy jobs ..
I cannot agree more!


Stemcells, Garden, and rootfs: how are these related?

J K
 

hi,

I'm learning about Cloud Foundry's ideas of how to organize applications and how it works under the covers. At the moment I'm having a little trouble understanding the relationship between a few ideas in CF: "rootfs", "Garden", and "stem cells".

Can someone explain:

(1) their understanding of what these ideas are; and
(2) how they are related?

Thank you!


Re: Buildpacks Checksum Site for Release Validation

Danny Rosen
 

In the future we may consider implementing a json feed. If we did go
forward with this work, would you be interested in creating example CI / CD
implementations as references?

On Tue, May 17, 2016 at 10:30 PM, Gwenn Etourneau <getourneau(a)pivotal.io>
wrote:

Hi,
Any json feed /api ?
Can be nice and more easy to integrate with any CI/CD tool.

Thanks

On Wed, May 18, 2016 at 11:15 AM, taichi nakashima <nsd22843(a)gmail.com>
wrote:

Great,

I hope cloudfoundry/cli will provide the same thing.

cf.
https://lists.cloudfoundry.org/archives/list/cf-dev(a)lists.cloudfoundry.org/thread/K3BEBY4A2WSUKS7YS5IF2UDQHHSU35A7/

Taichi Nakashima

2016年5月18日(水) 6:20 David Jahn <djahn(a)pivotal.io>:

Dear Cloud Foundry Users,

To help operators and users of Cloud Foundry establish a "chain of
custody" for buildpacks, we have launched the following checksum site:

https://buildpackverify.cloudfoundry.org

This site provides a checksum for all cached buildpack release zip files
(except for the java-buildpack). Whenever the buildpacks team generates a
new buildpack release, we will immediately compute the SHA256 checksum of
that file and upload it to this website.

The site is hosted on a different repository from the main buildpack
github repositories. It allows operators to validate that the zip file we
produced is the same artifact that has been downloaded and installed.

Additionally, if an operator wishes to further investigate the
components of a buildpack, the "manifest.yml" in each buildpack root
directory (for
example,
https://github.com/cloudfoundry/go-buildpack/blob/master/manifest.yml)
provides a catalog of every third party component in the buildpack, a URL
of that component's location, and an MD5 checksum of that component.

We hope that this will assist people in auditing the source of their
buildpack code!

Cheers,
Buildpacks Team


Re: Miniconda has been added to the Python Buildpack

Amit Kumar Gupta
 

Awesome!

On Wed, May 18, 2016 at 11:10 AM, Mike Dalessio <mdalessio(a)pivotal.io>
wrote:

Fantastic! Great work, Buildpacks team. I'm especially excited that this
will allow data scientists to more easily run their applications on Cloud
Foundry.


On Wed, May 18, 2016 at 1:56 PM, Danny Rosen <drosen(a)pivotal.io> wrote:

Over the last few weeks the Buildpacks team has experimented with
different methods of getting the Python buildpack working with various data
science dependencies (ex: sci-py, num-py, sklearn). We believe we have
found a solution that does not create significant challenges and fulfills
the underlying goal of enabling these dependencies in a native CF
application.

We have included the addition of Miniconda into the newest version [1
<https://github.com/cloudfoundry/python-buildpack/releases/tag/v1.5.6>]
of the python-buildpack and urge you to give it a try. To do so, include an
environment.yml [2] in your app and cf push *or* try out this sample app
[3 <https://github.com/ihuston/pydata_package_test>].

We're very excited about this addition and note that this is our first
step towards providing the functionality. As always, we're interested in
obtaining feedback and pull requests for improvement.

[1] -
https://github.com/cloudfoundry/python-buildpack/releases/tag/v1.5.6
[2] -
http://conda.pydata.org/docs/using/envs.html#use-environment-from-file
[3] - https://github.com/ihuston/pydata_package_test/


Re: Miniconda has been added to the Python Buildpack

Mike Dalessio
 

Fantastic! Great work, Buildpacks team. I'm especially excited that this
will allow data scientists to more easily run their applications on Cloud
Foundry.

On Wed, May 18, 2016 at 1:56 PM, Danny Rosen <drosen(a)pivotal.io> wrote:

Over the last few weeks the Buildpacks team has experimented with
different methods of getting the Python buildpack working with various data
science dependencies (ex: sci-py, num-py, sklearn). We believe we have
found a solution that does not create significant challenges and fulfills
the underlying goal of enabling these dependencies in a native CF
application.

We have included the addition of Miniconda into the newest version [1
<https://github.com/cloudfoundry/python-buildpack/releases/tag/v1.5.6>]
of the python-buildpack and urge you to give it a try. To do so, include an
environment.yml [2] in your app and cf push *or* try out this sample app [
3 <https://github.com/ihuston/pydata_package_test>].

We're very excited about this addition and note that this is our first
step towards providing the functionality. As always, we're interested in
obtaining feedback and pull requests for improvement.

[1] - https://github.com/cloudfoundry/python-buildpack/releases/tag/v1.5.6
[2] -
http://conda.pydata.org/docs/using/envs.html#use-environment-from-file
[3] - https://github.com/ihuston/pydata_package_test/


Miniconda has been added to the Python Buildpack

Danny Rosen
 

Over the last few weeks the Buildpacks team has experimented with different
methods of getting the Python buildpack working with various data science
dependencies (ex: sci-py, num-py, sklearn). We believe we have found a
solution that does not create significant challenges and fulfills the
underlying goal of enabling these dependencies in a native CF application.

We have included the addition of Miniconda into the newest version [1
<https://github.com/cloudfoundry/python-buildpack/releases/tag/v1.5.6>] of
the python-buildpack and urge you to give it a try. To do so, include an
environment.yml [2] in your app and cf push *or* try out this sample app [3
<https://github.com/ihuston/pydata_package_test>].

We're very excited about this addition and note that this is our first step
towards providing the functionality. As always, we're interested in
obtaining feedback and pull requests for improvement.

[1] - https://github.com/cloudfoundry/python-buildpack/releases/tag/v1.5.6
[2] - http://conda.pydata.org/docs/using/envs.html#use-environment-from-file
[3] - https://github.com/ihuston/pydata_package_test/


Re: Regarding UAA service

Siva Balan <mailsiva@...>
 

Hi Dax,
Can I request to log your issue at https://forum.predix.io ? There are a
lot more Predix focussed developers on that forum than this mailing list
and you are more likely to get a faster answer there.

Thanks
Siva

On Wed, May 18, 2016 at 8:25 AM, Sree Tummidi <stummidi(a)pivotal.io> wrote:

Hi Dax,
This is happening because your SAML has not been set up properly.
The email, first name and last name need to be mapped to attributes from
the incoming SAML assertion.
Please reach out to the Predix team so that they can set the correct
attribute mappings.

Thanks,
Sree

Sent from my iPhone

On May 17, 2016, at 7:22 PM, Dax Joshi <dax.joshi(a)tcs.com> wrote:

Hi,

Any update on this ?

Please let me know. I need to solve this issue as soon as possible.


Thanks & Regards,

Dax Joshi
Systems Engineer
Tata Consultancy Services
GARIMA PARK,IT/ITES SEZ,
PLOT # 41,
Gandhinagar - 382007,Gujarat
India
Cell:- 9586581656
Mailto: dax.joshi(a)tcs.com
Website: http://www.tcs.com
____________________________________________
Experience certainty. IT Services
Business Solutions
Consulting
____________________________________________




From: Dax Joshi/AHD/TCS
To: Sree Tummidi <stummidi(a)pivotal.io>
Cc: Jonathan Lo <jlo(a)us.ibm.com>, cf-dev(a)lists.cloudfoundry.org
Date: 05/14/2016 10:55 AM
Subject: Re: Regarding UAA service
------------------------------


Hi Sree and Jonathan,

Thank you very much for your consideration and reply.

I am working on predix. I have bind my UAA service with SAML.

I use the GE's common login page to login so that using SSO anyone from
the same business can use my application.

*https://<uaa-url>.predix-uaa.run.asv-pr.ice.predix.io/oauth/authorize?client_id=<client-id>&response_type=code*
<https://489afafd-c6b4-4d81-ae52-e51116af4597.predix-uaa.run.asv-pr.ice.predix.io/oauth/authorize?client_id=ppduaa&response_type=code>


Which redirect me to GE's common login page. After successful login SAML
is sending user information to

*https://<uaa-url>.predix-uaa.run.asv-pr.ice.predix.io/saml/SSO/alias/<uaa-url>.cloudfoundry-saml-login*
<https://489afafd-c6b4-4d81-ae52-e51116af4597.predix-uaa.run.asv-pr.ice.predix.io/saml/SSO/alias/489afafd-c6b4-4d81-ae52-e51116af4597.cloudfoundry-saml-login>

While this redirection I have seen in browser network, in encoded format
as form data SAML is passing user's correct and full information to UAA.
including first name, last name, email, groups, roles etc..

After that UAA service redirects the browser to my landing page which I
have set as *redirect_uri *with one cookie named *TS0164a009* and one *code
in request param.*

In our application we use that code with */oauth/token* service and get
the *auth_token. *After this we use the *auth_token *with */check_token*
or /*userinfo* service of uaa to get the user information.

In which we get following json



{
"user_id": "d9cf7779-744a-407d-a846-36e0570d70d9",
"user_name": "sso",
"email": "sso(a)unknown.org",
"client_id": "client_id",
"exp": 1462921362,
"scope": [
"scim.me",
"openid"
],
"jti": "684643f2-a15a-4fca-b9ca-2f9ba2c22f82",
"aud": [
"scim",
"openid",
"ppduaa"
],
"sub": "d9cf7779-744a-407d-a846-36e0570d70d9",
"iss": "
*https://<uaa-url>.predix-uaa.run.asv-pr.ice.predix.io/oauth/token"*
<https://489afafd-c6b4-4d81-ae52-e51116af4597.predix-uaa.run.asv-pr.ice.predix.io/oauth/token%27>
,
"iat": 1462878162,
"cid": "client-id",
"grant_type": "authorization_code",
"azp": "client_id",
"auth_time": 1462878076,
"zid": "489afafd-c6b4-4d81-ae52-e51116af4597",
"rev_sig": "d8ddc2e6",
"origin": "gefssstg"
}


Here I have replaced *uaa-url, sso *and *client_id* with its actual
values.

You can notice that I am not even receiving correct mail id. it gives @
unknown.org

In this json nothing except sso seems to be useful to me.

Please guide me how to get the exact user details from UAA that SAML is
passing to it.

Let me know if anything else you need.


Regards,

Dax Joshi
Systems Engineer
Tata Consultancy Services
GARIMA PARK,IT/ITES SEZ,
PLOT # 41,
Gandhinagar - 382007,Gujarat
India
Cell:- 9586581656
Mailto: dax.joshi(a)tcs.com
Website: http://www.tcs.com
____________________________________________
Experience certainty. IT Services
Business Solutions
Consulting
____________________________________________





From: Sree Tummidi <stummidi(a)pivotal.io>
To: Jonathan Lo <jlo(a)us.ibm.com>
Cc: Dax Joshi <dax.joshi(a)tcs.com>
Date: 05/14/2016 04:01 AM
Subject: Re: Regarding UAA service
------------------------------



Hi Dax,

Nice to meet you virtually. In the future you can use the cf-dev@
*lists.cloudfoundry.org* <http://lists.cloudfoundry.org/> for any UAA &
CloudFoundry related questions.
Can you elaborate on what you are trying to achieve with UAA & SAML
Integration and the use-case you have in mind


Thanks,
Sree Tummidi
Sr. Product Manager
Identity - Pivotal Cloud Foundry


On Fri, May 13, 2016 at 10:50 AM, Jonathan Lo <*jlo(a)us.ibm.com*
<jlo(a)us.ibm.com>> wrote:
Hi Dax,

Could you provide a bit more detail so that I can better direct your
query? As far as I know, you would be able to decode your access token in
order to obtain a user id, with which you could then get more user
information.

I've CCed Sree, our UAA PM, on the email.

Regards,

Jonathan

Sent from my iPhone

------------------------------
On May 13, 2016, 4:30:16 AM, *dax.joshi(a)tcs.com* <dax.joshi(a)tcs.com>
wrote:

From: *dax.joshi(a)tcs.com* <dax.joshi(a)tcs.com>
To: *jlo(a)us.ibm.com* <jlo(a)us.ibm.com>
Cc:
Date: May 13, 2016 4:30:16 AM
Subject: Regarding UAA service

Hi Jonathan,

This is Dax Joshi From TCS.

I found you email from *https://github.com/GESoftware-CF/uaa*
<https://github.com/GESoftware-CF/uaa>.

I have a query regarding getting loggedin
user details in case of UAA service and SAML Integration.

Please let me know if we can talk over
phone at your convenient time.

Please include other persons in this
loop if they can help me.


Thanks,

Dax Joshi
Systems Engineer
Tata Consultancy Services
GARIMA PARK,IT/ITES SEZ,
PLOT # 41,
Gandhinagar - 382007,Gujarat
India
Cell:- *9586581656* <9586581656>
Mailto: *dax.joshi(a)tcs.com* <dax.joshi(a)tcs.com>
Website: *http://www.tcs.com* <http://www.tcs.com/>
____________________________________________
Experience certainty. IT Services

Business Solutions

Consulting
____________________________________________

=====-----=====-----=====
Notice: The information contained in this e-mail
message and/or attachments to it may contain
confidential or privileged information. If you are
not the intended recipient, any dissemination, use,
review, distribution, printing or copying of the
information contained in this e-mail message
and/or attachments to it are strictly prohibited. If
you have received this communication in error,
please notify us by reply e-mail or telephone and
immediately and permanently delete the message
and any attachments. Thank you






Re: [abacus] Separate time-based from discrete usage metrics

Jean-Sebastien Delfino
 

Hi,

To fix the issue we decided to:
1. Distinguish between time-based (linux-container) and discrete usage
metrics (the rest basically)
2. Store the time-based metrics in a separate DB(s)
Your proposal looks good to me. Piotr, Kevin and Raj and I had several
design discussions on this topic in the last few days and we've come up
with a few more ideas on top of what you're describing here:

- The distinction between time-based and discrete resource usage metering
could also be understood as usage metering of a stateful vs stateless
resource or metric. In the stateful case, we simply store the state of the
resource instance in a separate DB like you're proposing (e.g. we store the
fact that an app or a container is currently running, or has stopped), and
update that state in place when it changes. Then to compute and report
usage later on we just need to the current resource instance state from
that DB.

- We could continue to store the metrics in the current historical
databases as well (on top of that new DB) to preserve the resource instance
history as many users typically want to know their past usage.

- Some of us were not sure if the time-based / discrete distinction should
be at the resource type level or at the metric level... IMO your proposal
to do that at the metric level is cleaner so I'm happy with it :)

- The dataflow module will probably need a few minor code changes to detect
the case where some of the output docs need to go to a separate DB (IIRC
you or someone else also mentioned that on one of our scrums or on slack...)

- Like you said, we may still need to maintain 2 DBs to purge old entries.
If that's easier, we could also adjust the usage accumulator service and
the dataflow module a bit to delete entries for inactive resource instances
right away (e.g. when an app or container stops.)

Thoughts?

P.S. I'll add these comments to issue #88 as well to make it be easier to
follow up there.

- Jean-Sebastien

On Thu, May 12, 2016 at 5:54 AM, Hristo Iliev <hsiliev(a)gmail.com> wrote:

Hi,

We're trying to fix Abacus issue 88: Missing aggregated usage for the
running application [1].

Background
=========

See the jsdelfino comment in the GitHub issue [2]. TL;DR: Resource
providers have to send a 'ping' doc per month for time-based metrics.

Proposed solution
==============

We decided to implement a solution in Abacus that frees the usage
providers from sending the 'ping' submission.

To fix the issue we decided to:
1. Distinguish between time-based (linux-container) and discrete usage
metrics (the rest basically)
2. Store the time-based metrics in a separate DB(s)

We already drafted a proposal for adding measurement type in the usage
plans with PR #320 [3].

We're about to spike on storing the time-based metrics in their own
Database, but we wanted to get the community opinion on the topic.

Motivation
========

The discrete usage submitted to Abacus is:
* stored in partitioned databases, due to their size/number
* like an event log, storing the history of the usage/resources

In contrast the current time-based metrics are:
* limited number (usually around 2 million on a loaded CF system)
* storing just the app resources usage state (GB/h consumed so far, GB/h
consuming currently)

Therefore it looks like a good idea to separate the two usage metrics
types and store the time-based metrics in a separate database. This will
allow us not only to solve the issue, but also to store and query the data
more effectively.

We may still need to maintain 2 databases and swap new/old (irrelevant)
metrics to reduce the DB size on the month boundaries.


Regards,
Hristo & Adriana

[1] https://github.com/cloudfoundry-incubator/cf-abacus/issues/88
[2]
https://github.com/cloudfoundry-incubator/cf-abacus/issues/88#issuecomment-148498164
[3] https://github.com/cloudfoundry-incubator/cf-abacus/pull/320


Re: Regarding UAA service

Sree Tummidi
 

Hi Dax,
This is happening because your SAML has not been set up properly.
The email, first name and last name need to be mapped to attributes from the incoming SAML assertion.
Please reach out to the Predix team so that they can set the correct attribute mappings.

Thanks,
Sree

Sent from my iPhone

On May 17, 2016, at 7:22 PM, Dax Joshi <dax.joshi(a)tcs.com> wrote:

Hi,

Any update on this ?

Please let me know. I need to solve this issue as soon as possible.


Thanks & Regards,

Dax Joshi
Systems Engineer
Tata Consultancy Services
GARIMA PARK,IT/ITES SEZ,
PLOT # 41,
Gandhinagar - 382007,Gujarat
India
Cell:- 9586581656
Mailto: dax.joshi(a)tcs.com
Website: http://www.tcs.com
____________________________________________
Experience certainty. IT Services
Business Solutions
Consulting
____________________________________________




From: Dax Joshi/AHD/TCS
To: Sree Tummidi <stummidi(a)pivotal.io>
Cc: Jonathan Lo <jlo(a)us.ibm.com>, cf-dev(a)lists.cloudfoundry.org
Date: 05/14/2016 10:55 AM
Subject: Re: Regarding UAA service


Hi Sree and Jonathan,

Thank you very much for your consideration and reply.

I am working on predix. I have bind my UAA service with SAML.

I use the GE's common login page to login so that using SSO anyone from the same business can use my application.
https://<uaa-url>.predix-uaa.run.asv-pr.ice.predix.io/oauth/authorize?client_id=<client-id>&response_type=code


Which redirect me to GE's common login page. After successful login SAML is sending user information to
https://<uaa-url>.predix-uaa.run.asv-pr.ice.predix.io/saml/SSO/alias/<uaa-url>.cloudfoundry-saml-login

While this redirection I have seen in browser network, in encoded format as form data SAML is passing user's correct and full information to UAA. including first name, last name, email, groups, roles etc..

After that UAA service redirects the browser to my landing page which I have set as redirect_uri with one cookie named TS0164a009 and one code in request param.

In our application we use that code with /oauth/token service and get the auth_token. After this we use the auth_token with /check_token or /userinfo service of uaa to get the user information.

In which we get following json



{
"user_id": "d9cf7779-744a-407d-a846-36e0570d70d9",
"user_name": "sso",
"email": "sso(a)unknown.org",
"client_id": "client_id",
"exp": 1462921362,
"scope": [
"scim.me",
"openid"
],
"jti": "684643f2-a15a-4fca-b9ca-2f9ba2c22f82",
"aud": [
"scim",
"openid",
"ppduaa"
],
"sub": "d9cf7779-744a-407d-a846-36e0570d70d9",
"iss": "https://<uaa-url>.predix-uaa.run.asv-pr.ice.predix.io/oauth/token",
"iat": 1462878162,
"cid": "client-id",
"grant_type": "authorization_code",
"azp": "client_id",
"auth_time": 1462878076,
"zid": "489afafd-c6b4-4d81-ae52-e51116af4597",
"rev_sig": "d8ddc2e6",
"origin": "gefssstg"
}



Here I have replaced uaa-url, sso and client_id with its actual values.

You can notice that I am not even receiving correct mail id. it gives @unknown.org

In this json nothing except sso seems to be useful to me.

Please guide me how to get the exact user details from UAA that SAML is passing to it.

Let me know if anything else you need.


Regards,

Dax Joshi
Systems Engineer
Tata Consultancy Services
GARIMA PARK,IT/ITES SEZ,
PLOT # 41,
Gandhinagar - 382007,Gujarat
India
Cell:- 9586581656
Mailto: dax.joshi(a)tcs.com
Website: http://www.tcs.com
____________________________________________
Experience certainty. IT Services
Business Solutions
Consulting
____________________________________________





From: Sree Tummidi <stummidi(a)pivotal.io>
To: Jonathan Lo <jlo(a)us.ibm.com>
Cc: Dax Joshi <dax.joshi(a)tcs.com>
Date: 05/14/2016 04:01 AM
Subject: Re: Regarding UAA service



Hi Dax,

Nice to meet you virtually. In the future you can use the cf-dev(a)lists.cloudfoundry.org for any UAA & CloudFoundry related questions.
Can you elaborate on what you are trying to achieve with UAA & SAML Integration and the use-case you have in mind


Thanks,
Sree Tummidi
Sr. Product Manager
Identity - Pivotal Cloud Foundry


On Fri, May 13, 2016 at 10:50 AM, Jonathan Lo <jlo(a)us.ibm.com> wrote:
Hi Dax,

Could you provide a bit more detail so that I can better direct your query? As far as I know, you would be able to decode your access token in order to obtain a user id, with which you could then get more user information.

I've CCed Sree, our UAA PM, on the email.

Regards,

Jonathan

Sent from my iPhone

On May 13, 2016, 4:30:16 AM, dax.joshi(a)tcs.com wrote:

From: dax.joshi(a)tcs.com
To: jlo(a)us.ibm.com
Cc:
Date: May 13, 2016 4:30:16 AM
Subject: Regarding UAA service

Hi Jonathan,

This is Dax Joshi From TCS.

I found you email from https://github.com/GESoftware-CF/uaa.

I have a query regarding getting loggedin
user details in case of UAA service and SAML Integration.

Please let me know if we can talk over
phone at your convenient time.

Please include other persons in this
loop if they can help me.


Thanks,

Dax Joshi
Systems Engineer
Tata Consultancy Services
GARIMA PARK,IT/ITES SEZ,
PLOT # 41,
Gandhinagar - 382007,Gujarat
India
Cell:- 9586581656
Mailto: dax.joshi(a)tcs.com
Website: http://www.tcs.com
____________________________________________
Experience certainty. IT Services

Business Solutions

Consulting
____________________________________________
=====-----=====-----=====
Notice: The information contained in this e-mail
message and/or attachments to it may contain
confidential or privileged information. If you are
not the intended recipient, any dissemination, use,
review, distribution, printing or copying of the
information contained in this e-mail message
and/or attachments to it are strictly prohibited. If
you have received this communication in error,
please notify us by reply e-mail or telephone and
immediately and permanently delete the message
and any attachments. Thank you




4481 - 4500 of 9421