|
Re: Any news on Tasks in CF Diego?
Koper, Dies <diesk@...>
Hi Andreas,
If you’re interested in the CLI UX, refer to the stories in the following epic: https://www.pivotaltracker.com/epic/show/3028607 In this iteration, there is no CLI command to retrieve the logs of certain task – mainly because in the log stream we have no identifier unique to a single task. Regards, Dies Koper Cloud Foundry Product Manager - CLI From: Nicholas Calugar [mailto:ncalugar(a)pivotal.io] Sent: Saturday, October 22, 2016 12:12 PM To: Andreas Mueller; Discussions about Cloud Foundry projects and the system overall. Subject: [cf-dev] Re: Re: Re: Any news on Tasks in CF Diego? Hi Andreas, My apologies, I didn’t intend to imply that app logs were the only channel for the task starter to consume the task results. The V3 Task resource will contain the result in the form of state of SUCCEEDED or FAILED. What I meant by, "state is left up to the developer”, is the task code should offload anything that needs persisted from the task run. For example, the task could stream data to a service, write to a database, write a file to a blobstore, etc. The platform is just responsible for running the task and logs shouldn’t be relied upon for state or results. The log source for tasks are “APP/TASK/<task_name>”. You’ll see something like “APP/TASK/0E858E94/0” in the log stream. Please note, the 0 is the instance id which will always be 0 as there is never more than 1 task. The sequence id was added for the CLI UX and is never sent to the backend. This allows the CLI to uniquely identify a task within an app without exposing the GUID or the non-unique Task name. I think this will all become a lot more clear when we complete the CLI UX, we are hoping to finish that work in the coming weeks. Thanks, Nick -- Nicholas Calugar Product Manager - Cloud Foundry API Pivotal Software, Inc. On October 21, 2016 at 12:55:55 AM, Andreas Mueller (andr.mueller(a)sap.com<mailto:andr.mueller(a)sap.com>) wrote: Hi Nicholas, thanks a lot for the quick and helpful response, which is in line with the latetest "release candidate" version of the V3 REST API docu. So, the app log is the only channel for the task starter to consume the task results and multiple tasks with the same name can be started in parallel on a single app droplet. Based on these facts I'd assume that the log lines belonging to the task can be easily identified in the app's log, by means of the task name and newly introduced sequenceId. Could you please elaborate a bit more on the detailed format (source tagging) of log lines produced by an task belonging to an app? I've found a hint in [1] (at that time there was no sequenceId, yet), but there does not seem to be some official documentation yet, about how exactly log lines produced by an app's task can be identified and grouped together. Thanks and kind regards, Andreas [1] https://www.pivotaltracker.com/n/projects/966314/stories/111472198
|
|
|
|
Re: Any news on Tasks in CF Diego?
Nicholas Calugar
Hi Andreas,
toggle quoted messageShow quoted text
My apologies, I didn’t intend to imply that app logs were the only channel for the task starter to consume the task results. The V3 Task resource will contain the result in the form of state of SUCCEEDED or FAILED. What I meant by, "state is left up to the developer”, is the task code should offload anything that needs persisted from the task run. For example, the task could stream data to a service, write to a database, write a file to a blobstore, etc. The platform is just responsible for running the task and logs shouldn’t be relied upon for state or results. The log source for tasks are “APP/TASK/<task_name>”. You’ll see something like “APP/TASK/0E858E94/0” in the log stream. Please note, the 0 is the instance id which will always be 0 as there is never more than 1 task. The sequence id was added for the CLI UX and is never sent to the backend. This allows the CLI to uniquely identify a task within an app without exposing the GUID or the non-unique Task name. I think this will all become a lot more clear when we complete the CLI UX, we are hoping to finish that work in the coming weeks. Thanks, Nick -- Nicholas Calugar Product Manager - Cloud Foundry API Pivotal Software, Inc.
On October 21, 2016 at 12:55:55 AM, Andreas Mueller (andr.mueller(a)sap.com)
wrote: Hi Nicholas, thanks a lot for the quick and helpful response, which is in line with the latetest "release candidate" version of the V3 REST API docu. So, the app log is the only channel for the task starter to consume the task results and multiple tasks with the same name can be started in parallel on a single app droplet. Based on these facts I'd assume that the log lines belonging to the task can be easily identified in the app's log, by means of the task name and newly introduced sequenceId. Could you please elaborate a bit more on the detailed format (source tagging) of log lines produced by an task belonging to an app? I've found a hint in [1] (at that time there was no sequenceId, yet), but there does not seem to be some official documentation yet, about how exactly log lines produced by an app's task can be identified and grouped together. Thanks and kind regards, Andreas [1] https://www.pivotaltracker.com/n/projects/966314/stories/111472198
|
|
|
|
Re: [security] CVE 2016-6655: Utility script command injection
Molly Crowther
Hello Hector,
The answer is that it varies. I can tell you that we have followed the steps we need to do to get it into the database, it just takes time depending on how fast MITRE makes the updates. It is not an automatic process. Let me know if you have any other questions. Thanks, Molly Crowther CFF Security Team On Fri, Oct 21, 2016 at 2:46 AM, Hector Rivas Gandara < hector.rivas.gandara(a)digital.cabinet-office.gov.uk> wrote: Hello,
|
|
|
|
Re: [security] CVE 2016-6655: Utility script command injection
Hector Rivas Gandara
Hello,
Thank you for reporting this. I observed that the CVE in mitre.org did not get updated: http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2016-6655 How long does it take to get it updated? On 17 October 2016 at 20:34, Molly Crowther <mcrowther(a)cloudfoundry.org> wrote: Hello all - many people were asking for more information, so we have -- Regards Hector Rivas | GDS / Multi-Cloud PaaS
|
|
|
|
Re: Any news on Tasks in CF Diego?
Andreas Mueller
Hi Nicholas,
thanks a lot for the quick and helpful response, which is in line with the latetest "release candidate" version of the V3 REST API docu. So, the app log is the only channel for the task starter to consume the task results and multiple tasks with the same name can be started in parallel on a single app droplet. Based on these facts I'd assume that the log lines belonging to the task can be easily identified in the app's log, by means of the task name and newly introduced sequenceId. Could you please elaborate a bit more on the detailed format (source tagging) of log lines produced by an task belonging to an app? I've found a hint in [1] (at that time there was no sequenceId, yet), but there does not seem to be some official documentation yet, about how exactly log lines produced by an app's task can be identified and grouped together. Thanks and kind regards, Andreas [1] https://www.pivotaltracker.com/n/projects/966314/stories/111472198
|
|
|
|
Re: Any news on Tasks in CF Diego?
Nicholas Calugar
Hi Andreas,
toggle quoted messageShow quoted text
This is a lot, but I’ll try my best to help. I encourage you to stop by our [1] Slack channel anytime for a quick chat. 1. The V3 Task resource should eventually end up in either SUCCEEDED or FAILED state. If the state is FAILED, the result.failure_reason will have a human readable message, e.g. "Exited with status 1”. Task logs will appear in the log stream of the app. Otherwise, state is left up to the developer. 2. I think this is the same question and answer as #1. If not, maybe let’s chat in Slack and update this thread. 3. I think you are asking about the “V3 CLI Plugin”. This was only intended to help developers validate the API, and is definitely not full-featured. We are working with the CLI team to properly expose a coherent Tasks UX. 4. The Task Completion Callback URL is an internal Diego Tasks feature and is not exposed to the enduser. As for when the Tasks feature will be ready to use, we are working very closely with the CLI team to expose an MVP of Tasks. Aside from a few minor changes, the API in CF-245 appears to be supporting this effort. Thanks, Nick [1] https://cloudfoundry.slack.com/messages/capi/ -- Nicholas Calugar Product Manager - Cloud Foundry API Pivotal Software, Inc.
On October 20, 2016 at 4:45:46 AM, Andreas Mueller (andr.mueller(a)sap.com)
wrote: Hi all, we at SAP are very interested in the new task feature, because we currently run some dummy/fake apps, in order to simulate task-like behavior, which includes some cumbersome app log parsing. The promises of the tasks are exactly what we are looking for, since the starter of the task would just like to see the result of the task action in some convenient and easy to consume way. Unfortunately we currently know of a number of older and newer resources (URLs) with information on the task topic, which do not give much information on the details of the result handling and seem to even be slightly contradicting in some low level details. As a starting point, it would be good to know the most recent official documentation on the tasks (for the task implementor and consumer). To shortly summarize, our most important questions from task implementation and consumption perspective are: 1) How does the task implementation provide its result - either through process exit code and stderr/out or via tracing (+ optional/internal result file)? Is there some definite contract or at least best-practice for the task implementation side? 2) How does the infrastructure collect the task result and especially how is it exposed to the consumer? In the V3 API docu there’s just a result JSON object, with only one “failure_reason” property. We’d also expect the process exit code and maybe stdout/err to be part of the result JSON. Or in case the (optional) ResultFile and callback URL are the way to go, there should be some env var exposed to the task implementation, so that one knows to which file the result data shall be written. 3) It seems like the result from the V3 REST API is not exposed via the V3 tasks client command (at least in its current state)? 4) Is the “Task Completion Callback URL” mechanism that is mentioned in several sources only intended for internal use by client implementations, or also for external consumers? It would also be interesting, when the tasks will be ready to use. In the talk on V3 given on European CF Summit (in Sept. in Frankfurt) the following statement on tasks was given ([1] – slides 46/47): “When? Very soon (~ next few weeks)” I’ll add some more detailed observations about the potential result handing as a second post. Thanks a lot and kind regards, Andreas Mueller – SAP SE [1] http://schd.ws/hosted_files/cfsummiteu2016/2a/Unleasing%20the%20Power%20of%20the%20V3%20API_Frankfurt.pdf
|
|
|
|
Re: CF environmental variables - org is missing
Jason Sherron
Hi Vinod,
toggle quoted messageShow quoted text
I was referring to filtering packets by application at the network layer, nothing to do with the format of application logs or other human-readable text. Sorry for the confusion. Jason
On Wed, Oct 19, 2016 at 6:49 PM Vinod Singh <vinoddandy(a)gmail.com> wrote:
Hi Jaison,
|
|
|
|
Re: Contribute mongodb 3.X bosh release
Shannon Coen
Hello Lyn,
toggle quoted messageShow quoted text
Thank you for offering to donate your MongoDB 3.x bosh release! I would be the person to talk to about contributing your service to the Cloud Foundry Foundation. I have a question, does your release use the v2 broker API? The services-contrib-release were originally based on the v1 broker API and README suggests they still are. Earlier this year Cloud Foundry dropped support for the v1 broker API. TL;DR The cf-service-contrib-release repo started as the original five cloud foundry services developed by VMWare. They used the v1 broker API. In 2012 when the v2 broker API was released the original services were donated to the cloudfoundry-community github org. Since then they have been maintained by Ruben Koster, among others, and the number of services grew. Earlier this year Cloud Foundry dropped support for the v1 broker API. Assuming the README for this repo is still accurate and they still depend on the v1 broker API, these services are no longer compatible with latest versions of Cloud Foundry. Hopefully, your service has been updated to use the v2 broker API. I'll contact you out of band about next steps. Best, Shannon Shannon Coen Product Manager, Cloud Foundry Pivotal, Inc.
On Wed, Oct 19, 2016 at 7:05 PM, Lynn Lin <Lynn.Lin(a)emc.com> wrote:
All,
|
|
|
|
Re: Any news on Tasks in CF Diego?
Andreas Mueller
Here are the details, mentioned before:
[1] - This document describes the very low level technical/internal details of the CF task lifecycle, i.e. it does not necessarily match with the V3 API for consumption (e.g. the internal task states are PENDING -> CLAIMED -> RUNNING -> COMPLETED -> RESOLVING, while the states in V3 are PENDING, RUNNING, SUCCEEDED, CANCELING, and FAILED). - The document is quite interesting, but it not intended as end-user documentation, as it deals with the intrinsic mechanisms, which guarantee the at “most once” execution guarantee of tasks (via BBs, etcd's atomic and consistent CompareAndSwap, claiming and resolving). - An interesting detail is the fact, that the internal v1 task description still has some (redundant) “failed” Boolean and a “result” field next to the “failure_reason”. “The result is the output of the last FetchResult action.” – whatever this means. -Also interesting: “If the Task is PENDING: And the Task was created more than 30 minutes ago (this is configurable via –timeToClaimTask in executor): then the task is marked as failed.” -> later documents have more/different information about this aspect [2] - This document seems to be some very internal discussion state. - I found the following non-goals interesting: a) Specifying a path to a (small) file to store and return result of the task b) Specifying a custom callback web url to be notified when the task is completed - And: a) Only X number of tasks are kept on an app before rolling off older, completed tasks. b) Stdout/stderr from the task will be available on the apps firehose logs. [3] - Very nice and understandable but still internal (BBS) overview document on tasks. - “When the Task completes, the Cell sets the Failed, FailureReason, and Result fields on the Task as appropriate, and sets the Task's state to COMPLETED.” – “Result field” obviously refers to the internal JSON description of a task. How does that correlate to the (external) Result JSON object in the V3 API? - “At this point it is up to the Diego client to detect and resolve the completed Task. It can do this either by having set a completion callback URL on the Task when defined, or by polling for the Task and resolving and deleting it itself.” – So the callback URL is only intended for client impls? Seems so, since otherwise there would need to be an array of URLs, if the actual task creator/starter could also provide such an callback URL, in addition. [4] seems to confirm this assumption. - “If ResultFile was specified and the Task has completed successfully, Result will include the first 10KB of the ResultFile.” – How does this result file relate to the log file of the app/task and how can the task impl write to such an result file? Also, such a result file cannot be specified, when creating a task via V3 API. - “Diego will automatically delete completed Tasks that remain unresolved after 2 minutes.” – How does that fit to the “Only X number of tasks are kept on an app before rolling off older, completed tasks.” mentioned in some other/previous document? [5] and [6] - Seem to be the most recent and official external documentation - Unclear, how the client/task starter is supposed to retrieve the task result, other than the “failure_message” [1] https://github.com/cloudfoundry/diego-release/wiki/Diego-Task-Lifecycle (2014/04) [2] https://docs.google.com/document/d/1CCHDUa2UWRjXkxEdksX4M9BGQ8hBqiMys46wxeF5XE4/edit# (2016/04) [3] https://github.com/cloudfoundry/bbs/blob/master/doc/tasks.md (2016/06) [4] https://github.com/cloudfoundry/bbs/blob/master/doc/defining-tasks.md [5] http://v3-apidocs.cloudfoundry.org/version/release-candidate/#the-task-object [6] https://github.com/cloudfoundry/v3-cli-plugin/tree/master/commands
|
|
|
|
Any news on Tasks in CF Diego?
Andreas Mueller
Hi all,
we at SAP are very interested in the new task feature, because we currently run some dummy/fake apps, in order to simulate task-like behavior, which includes some cumbersome app log parsing. The promises of the tasks are exactly what we are looking for, since the starter of the task would just like to see the result of the task action in some convenient and easy to consume way. Unfortunately we currently know of a number of older and newer resources (URLs) with information on the task topic, which do not give much information on the details of the result handling and seem to even be slightly contradicting in some low level details. As a starting point, it would be good to know the most recent official documentation on the tasks (for the task implementor and consumer). To shortly summarize, our most important questions from task implementation and consumption perspective are: 1) How does the task implementation provide its result - either through process exit code and stderr/out or via tracing (+ optional/internal result file)? Is there some definite contract or at least best-practice for the task implementation side? 2) How does the infrastructure collect the task result and especially how is it exposed to the consumer? In the V3 API docu there’s just a result JSON object, with only one “failure_reason” property. We’d also expect the process exit code and maybe stdout/err to be part of the result JSON. Or in case the (optional) ResultFile and callback URL are the way to go, there should be some env var exposed to the task implementation, so that one knows to which file the result data shall be written. 3) It seems like the result from the V3 REST API is not exposed via the V3 tasks client command (at least in its current state)? 4) Is the “Task Completion Callback URL” mechanism that is mentioned in several sources only intended for internal use by client implementations, or also for external consumers? It would also be interesting, when the tasks will be ready to use. In the talk on V3 given on European CF Summit (in Sept. in Frankfurt) the following statement on tasks was given ([1] – slides 46/47): “When? Very soon (~ next few weeks)” I’ll add some more detailed observations about the potential result handing as a second post. Thanks a lot and kind regards, Andreas Mueller – SAP SE [1] http://schd.ws/hosted_files/cfsummiteu2016/2a/Unleasing%20the%20Power%20of%20the%20V3%20API_Frankfurt.pdf
|
|
|
|
Failed to push app to Diego windows cell
Meng, Xiangyi <Xiangyi.Meng@...>
Hi, all
I wanted to build up a windows cell for my Diego env(CF 237 + Diego 0.1472). I pushed https://github.com/cloudfoundry-incubator/NET-sample-app for a test, but failed at staging phase. I checked auctioneer log and seems the task has been added to windows cell. {"timestamp":"1476928530.877959251","source":"auctioneer","message":"auctioneer.auction.task-added-to-cell","log_level":1,"data":{"cell-guid":"WIN-8SD7KKQPJEE","session":"75341","task-guid":"ff42893c-6482-464b-8bb9-e5cddcfb4dcd-8d6d55561ae846b1b9f80cce77bf6be9"}} But somehow windows cell could not get task. {"timestamp":"1476967109.798601389","source":"rep","message":"rep.executing-container-operation.task-processor.failed-fetching-task","log_level":2,"data":{"container-guid":"763aaf66-cfd3-4f21-8dc4-d49650c7bce9-4f7684bd02b54757bc45d0b627bc1409","container-state":"reserved","error":"Invalid Response with status code: 404","session":"369.1"}} I also get quite a lot of 404 error when sync. {"timestamp":"1476967231.735509396","source":"rep","message":"rep.running-bulker.sync.failed-to-generate-operations","log_level":2,"data":{"error":"failed to retrieve tasks: Invalid Response with status code: 404","session":"11.305"}} Could someone shed a light on how to debug this issue? Which part I should look further? Any help will be appreciated. Regards, Maggie
|
|
|
|
cloudfoundry file descriptor limit is too small
peter huang
Hi, All,
My cloudfoundry version is 225, when i run some java spring-boot program in the cf, sometime it will throw the "too many files open" exception, i know it is about the file limit. So i login to the runner, and use `ulimit -n` to see, they are 1024, but i found in the warden .\cf-release-225\jobs\dea_next\templates\dea_ctl.erb it call `ulimit -n 4096`, looks like it doesn't work. and in .\cf-release-225\packages\dea_next\dea_next\vendor\cache\warden-181b550918c8\warden\spec\container\linux_spec.rb Line 1418: response = client.run(:handle => handle, :script => "ulimit -n", :rlimits => rlimits) no sure it will overwrite the exist one(i don't know ruby) my question is, does they some way to expand the size of limit ? or we need to upgrade the cloudfoundry? Thanks, Peter
|
|
|
|
Runtime PMC - Persi Project Lead
Dieu Cao <dcao@...>
Hello all,
Ted Young, the Project Lead for the Persi team within the Runtime PMC, last day at Pivotal was this past week. We wish him all the best in his future endeavors and thank him for his time with us. Dell would like to nominate Julian Hjortshoj for the Project Lead for the Persi team. Julian joined the Diego Persistence team as a software developer in April. During his 25-odd year career making software he has served in various technical leadership roles (manager, project lead, architect). Prior to joining CF, Julian worked for the ECM division of Dell/EMC as the application/ui architect for a suite of Cloud Foundry deployed content management applications. If there are any questions/objections to this appointment, please send that to me by end of day October 27, 2016. -Dieu Cao Runtime PMC Lead
|
|
|
|
Contribute mongodb 3.X bosh release
Lynn Lin
All,
Previously we from DELL-EMC use MongoDB service 2.X from this project under cloudfoundry community https://github.com/cloudfoundry-community/cf-services-contrib-release . Months ago we want to use mongoDB 3.X which improves performance a lot however we don’t find the available mongoDB 3.X under cloudfoundy github so our team developed MongoDB 3.X bosh release and provide additional cluster support compared to https://github.com/cloudfoundry-community/cf-services-contrib-release . This version is deployed to production site and running for quite a while , it looks stable . Who should we contact to start conversation to see how we can contribute back for this bosh mongoDB 3.X release if it is worth Thanks Lynn Lin
|
|
|
|
Re: CF environmental variables - org is missing
Vinod Singh <vinoddandy@...>
Hi Jaison,
toggle quoted messageShow quoted text
If CF team can include app_name+space+org in Loggregator logs, that would solve many of our use cases. The current application id is created at runtime and external systems finding difficult to segregate message based on application id. Do we have any blog item to include app_name in loggregator messages ? Regards, Vinod
On Tue, Oct 18, 2016 at 8:03 AM, Peter Dotchev <dotchev(a)gmail.com> wrote:
Hi Jason,
|
|
|
|
cf-release Submodule Path Recursion Error (and fix!)
Chunyi Lyu <clyu@...>
We've seen an issue with ./scripts/update in cf-release. If you have a
sufficiently old version of `cf-release`, and then you try to run the update script, you'll get an error that looks like this: ``` Failed to recurse into submodule path 'src/consul-release' ``` Here are the steps to reproduce: - `git clone https://github.com/cloudfoundry/cf-release` - `git checkout 58f89c63b1cd673df34ca98302e4573ed60c1bf3` - `git submodule update --init --recursive` - `git checkout develop` - `./scripts/update` You should see the error at this point. If you hit this, we've got three fixes for you. First and potentially simplest, just run one more git submodule sync --recursive && ./scripts/update after the failure. If that doesn't work, you can either delete and re-clone cf-release from scratch, or go edit your git modules: Edit `git/modules/src/consul-release/modules/src/confab/vendor/ github.com/hashicorp/serf/config`, so that it's remote url points to ` https://github.com/cloudfoundry-incubator/serf` instead of ` https://github.com/hashicorp/serf`. Hopefully this won't have inconvenienced too many of you! Regards- -Jesse Alford && Chunyi Lyu CF Release Integration Team
|
|
|
|
Re: How to run cronjobs with Application
Anuj Jain <anuj17280@...>
Thanks Nick I will try it
toggle quoted messageShow quoted text
On 19 Oct 2016 8:21 a.m., "Nicholas Calugar" <ncalugar(a)pivotal.io> wrote:
Hi Anuj,
|
|
|
|
Re: How to run cronjobs with Application
Nicholas Calugar
Hi Anuj,
toggle quoted messageShow quoted text
For scheduling, you might want to look into the Advanced Python Scheduler [1]. You would push the scheduler as a separate app to Cloud Foundry. You could also experiment with V3 Tasks that we are close to releasing. You can think of Tasks as commands to run against your app code that only take up resources while executing. As of CF-245, you can push an application and then run a task against it. There’s currently only support via curl [2], but we are working on supporting this via CLI. [1] http://apscheduler.readthedocs.io/en/3.0/ [2] http://v3-apidocs.cloudfoundry.org/version/release-candidate/#push-v2-app-and-run-a-v3-task Thanks, Nick -- Nicholas Calugar
On October 18, 2016 at 8:03:38 AM, Anuj Jain (anuj17280(a)gmail.com) wrote:
Hi, We have an application (lets say django app) - which we deployed on Cloud Foundry - now there are few commands which we want to run periodically (cron jobs) - what is the best way to handle this with Cloud Foundry. Thanks - Anuj
|
|
|
|
Contribute mongodb 3.X bosh release
Lin, Lynn <Lynn.X.Lin@...>
All,
Previously we use MongoDB service 2.X from this project under cloudfoundry community https://github.com/cloudfoundry-community/cf-services-contrib-release . Months ago we want to use mongoDB 3.X which improves performance a lot however we don't find the available mongoDB 3.X under cloudfoundy github so peter,huang (cc'ed) developed MongoDB 3.X bosh release and provide additional cluster support compared to https://github.com/cloudfoundry-community/cf-services-contrib-release . This version is deployed to production site and running for quite a while , it looks stable . Who should we contact to start conversation to see how we can contribute back for this bosh mongoDB 3.X release if it is worth Thanks Lynn
|
|
|
|
Re: UAA Standalone Deployment
Sree Tummidi
Hi Abhishek,
toggle quoted messageShow quoted text
As part of Pivotal deployments UAA is being used to secure both the platform and Apps running on the platform. Our hosted Cloud Foundry deployment (aka Pivotal Web Services) is running on AWS has two UAA instances handling a load of 20-30 requests per second per instance. You need to scale the UAA instances depending on your concurrency requirements. Thanks, Sree Tummidi Staff Product Manager Identity - Pivotal Cloud Foundry
On Tue, Oct 18, 2016 at 3:21 PM, Siva Balan <mailsiva(a)gmail.com> wrote:
Hi Abhishek,
|
|
|