|
Re: [vcap-dev] Java OOM debugging
Daniel Jones
Hi Lari et al,
toggle quoted messageShow quoted text
Thanks for your help Lari. David and I are pairing on this issue, and we're yet to resolve it. We're in the process of creating a repeatable test case (our most crashy app makes calls to external services that need mocking), but in the meantime, here's what we've seen. Between Java Buildpack commit e89e546 and 17162df, we see apps crashing with Warden out of memory errors. killjava.sh never fires, and this has led us to believe that the kernel is shooting a cgroup process in the head after the cgroup oversteps its memory limit. We cannot find any evidence of the OOM killer firing in any logs, but we may not be looking in the right place. The JBP is setting heap to be 70%, metaspace to be 15% (with max set to the same as initial), 5% for "stack", 5% for "normalised stack" and 10% for "native". We do not understand why this adds up to 105%, but haven't looked into the JBP algorithm yet. Any pointers on what "normalised stack" is would be much appreciated, as this doesn't appear in the list of heuristics supplied via app env. Other team members tried applying the same settings that you suggested - thanks for this. Apps still crash with these settings, albeit less frequently. After reading the blog you linked to ( http://java.dzone.com/articles/java-8-permgen-metaspace) we wondered whether the increased *reserved *metaspace claimed after metaspace GC might be causing a problem; however we reused the test code to create a metaspace leak in a CF app and saw metaspace GCs occur correctly, and memory usage never grow over MaxMetaspaceSize. This figures, as the committed metaspace is still less than MaxMetaspaceSize, and the reserved appears to be whatever RAM is free across the whole DEA. We noted that an Oracle blog ( https://blogs.oracle.com/poonam/entry/about_g1_garbage_collector_permanent) mentions that the metaspace size parameters are approximate. We're currently wondering if native allocations by Tomcat (APR, NIO) are taking up more container memory, and so when the metaspace fills, it's creeping slightly over the limit and triggering the kernel's OOM killer. Any suggestions would be much appreciated. We've tried to resist tweaking heuristics blindly, but are running out of options as we're struggling to figure out how the Java process is using *committed* memory. pmap seems to show virtual memory, and so it's hard to see if things like the metaspace or NIO ByteBuffers are nabbing too much and trigger the kernel's OOM killer. Thanks for all your help, Daniel Jones & David Head-Rapson
On Wed, Apr 29, 2015 at 8:07 PM, Lari Hotari <Lari(a)hotari.net> wrote:
Hi, --
Regards, Daniel Jones EngineerBetter.com
|
|
|
|
Re: Is there an auto-completion script?
Takeshi Morikawa
Hi Daniel
toggle quoted messageShow quoted text
I found this cf(cli) completion https://github.com/cf-buildpacks/cf_completion bosh cli completion https://github.com/anfernee/bosh-completion Is my answer what you're hoping for? 2015-05-08 14:28 GMT+09:00 Daniel Kaplan <dkaplan(a)pivotal.io>:
Hi DevList,
|
|
|
|
Is there an auto-completion script?
Daniel Kaplan
Hi DevList,
I think it would be extra convenient if there was Cloud Foundry auto-completion script that worked similar to the way git's git-completion <https://github.com/git/git/blob/master/contrib/completion/git-completion.bash> works. Does one already exist? If not, I might write it in my free time. Let me know your thoughts. Thanks, Dan
|
|
|
|
Meeting Minutes for Services PMC 2015-05-07
Shannon Coen
https://docs.google.com/document/d/10aOoLF_FPxuHYQfI813VCTF9z2VRC_dIuYDIGND3xbU/edit?usp=sharing
Highlights: 1. Two projects were approved for incubation: Brooklyn service broker from Cloudsoft, and a MSSQL Server service broker from HP. 2. Updates provides for the Service Enablement and Notifications projects. Best, Shannon Coen Product Manager, Cloud Foundry Pivotal, Inc.
|
|
|
|
Re: Logging Infrastructure for CF components
Erik Jasiak <ejasiak@...>
Hi Ronak,
We're trying to make sure we understand the question here - * Component logs do not go through loggregator today, only to the rsyslog endpoint designated. * All app logs would be available to an operator via the firehose. * " it is more like pulling model from end component side so if components before it are slow in sending logs then there will be buffers" - component logs again are not part of the firehose - even then, there would be buffers, but all messages are timestamped (if you're worried about ordering?) * "is there a need to implement something in between doppler and traffic controller to push all the application logs " - not sure I understand, but though you need at least dopper in every AZ that a metron is in today, doppler can speak to traffic controllers cross-AZ. Did that help? Erik On Wed, May 6, 2015 at 7:48 PM, ronak banka <ronakbanka.cse(a)gmail.com> wrote: Hi everyone,
|
|
|
|
Re: Utilities PMC - 2015-05-05 Notes
Mike Dalessio
Hi all,
toggle quoted messageShow quoted text
In response to several suggestions, I've moved the Utilities PMC notes into markdown files in a github repo. I've created this public github repo: https://github.com/cloudfoundry/pmc-notes and the Utilities PMC notes will be within it, at: https://github.com/cloudfoundry/pmc-notes/tree/master/Utilities I've added a document to the GDrive directing visitors to the Github repo. Cheers, -m
On Wed, May 6, 2015 at 3:24 PM, Mike Dalessio <mdalessio(a)pivotal.io> wrote:
Hey everyone,
|
|
|
|
Re: Buildpacks PMC - 2015-05-04 Notes
Mike Dalessio
Hi all,
toggle quoted messageShow quoted text
In response to several suggestions, I've moved the Buildpacks notes into markdown files in a github repo. I've created this public github repo: https://github.com/cloudfoundry/pmc-notes and the Buildpacks PMC notes will be within it, at: https://github.com/cloudfoundry/pmc-notes/tree/master/Buildpacks I've added a document to the GDrive directing visitors to the Github repo. Cheers, -m
On Mon, May 4, 2015 at 1:50 PM, Mike Dalessio <mdalessio(a)pivotal.io> wrote:
Hi all,
|
|
|
|
Re: Failed to start Native apps in CF using null-build pack
JT Archie <jarchie@...>
Balaramaraju,
It looks like in your *CF logs* that you are pushing the app with an incorrect start command, most likely with the meta information from a Procfile. Please try deploying the application like this: *cf push helloWorld2 -b https://github.com/ryandotsmith/null-buildpack <https://github.com/ryandotsmith/null-buildpack> --no-route -c "helloWorld2.sh" -s lucid64* If you experience further problems with the app not starting. Please try the new binary-buildpack <https://github.com/cloudfoundry/binary-buildpack> that cf-release will start supporting. Kind regards, JT On Thu, May 7, 2015 at 5:34 AM, Balaramaraju JLSP <balaramaraju(a)gmail.com> wrote: Hi All,
|
|
|
|
Re: Discrepancy between `cf apps` memory usage and cgroup's memory.usage_in_bytes in CF 2.13.0
Daniel Jones
Aha - just found it, and it does indeed tally up. Thanks again!
On Thu, May 7, 2015 at 3:06 PM, Daniel Jones < daniel.jones(a)engineerbetter.com> wrote: Thanks for the reply Matthew! -- Regards, Daniel Jones EngineerBetter.com
|
|
|
|
Re: Discrepancy between `cf apps` memory usage and cgroup's memory.usage_in_bytes in CF 2.13.0
Daniel Jones
Thanks for the reply Matthew!
We saw that bit of code when we were following it through. When we go as far as the call from the DEA's Warden client to the Warden Server, we struggled to find where those stats (total_rss, total_cache, total_inactive_file) came from. DO you happen to know what the source of truth is for those data? Thanks for your help. On Thu, May 7, 2015 at 1:58 PM, Matthew Sykes <matthew.sykes(a)gmail.com> wrote: The reported statistic is calculated here: -- Regards, Daniel Jones EngineerBetter.com
|
|
|
|
Re: Buildpacks PMC - 2015-05-04 Notes
Mike Dalessio
Hi Ryan,
toggle quoted messageShow quoted text
Thanks for asking this question. The "risk" called out in the inception encompassed a number of things, but what they really all boil down to is that the java-buildpacks team has its own roadmap and conventions; and the two teams don't often communicate about sharing resources, infrastructure, or planning. I think a reasonable first step is for you and I (and maybe JT and Ben, the engineering anchors for each team) to have a regular chat on our calendars. I'd prefer not to get bitten by Conway's Law if we can easily mitigate this risk. I'll ship you a calendar invite; as well as make sure the java-buildpack team, as well as voting members of the PMC, are represented in the next inception or roadmap discussion. -m
On Mon, May 4, 2015 at 2:43 PM, Ryan Morgan <ryanmorgan(a)gmail.com> wrote:
|
|
|
|
Re: Discrepancy between `cf apps` memory usage and cgroup's memory.usage_in_bytes in CF 2.13.0
Matthew Sykes <matthew.sykes@...>
The reported statistic is calculated here:
https://github.com/cloudfoundry/dea_ng/blob/310797e1097dcd5531bff4077ccd8f02f6091219/lib/dea/stat_collector.rb#L92-L94 On Thu, May 7, 2015 at 8:28 AM, Daniel Jones < daniel.jones(a)engineerbetter.com> wrote: Hi all, -- Matthew Sykes matthew.sykes(a)gmail.com
|
|
|
|
Re: use null-buildpack to support standalone non-web
Daniel Mikusa
On Thu, May 7, 2015 at 2:09 AM, Balaramaraju JLSP <balaramaraju(a)gmail.com>
wrote: I think it would depend, you'd probably be OK with your simple hello world app. Something more complicated using third party libraries, may have trouble. I'd personally try to avoid that and just compile on a machine that is as close to the runtime environment as possible. To match the cflinuxfs2 stack, you could use an Ubuntu 14.04 VM (Vagrant makes this very easy) or this docker repo. https://registry.hub.docker.com/u/cloudfoundry/cflinuxfs2/ This looks like you're having trouble connecting to github to download the build pack. Do you have a network or security group that might prevent this? Another option if you have admin access would be to download the src [1] and use `cf create-buildpack` to upload the build pack manually. You can then just use "-b null_buildpack" or whatever name you give it with `cf create-buildpack`. [1] - https://github.com/ryandotsmith/null-buildpack/archive/master.zip Dan App logs from Server :
|
|
|
|
Discrepancy between `cf apps` memory usage and cgroup's memory.usage_in_bytes in CF 2.13.0
Daniel Jones
Hi all,
Whilst investigating the Java Buildpack out-of-memory issues David Head-Rapson mailed about the other day, we discovered a discrepancy between the memory usage stat provided by `cf app` and the value stored in the corresponding cgroup's `memory.usage_in_bytes` file. The latter seems to be bumping right along the maximum allowed. - We did a `cf app`, and got a memory stat of 847.6MiB of 896MiB. - We got the appId from CF_TRACE, `bosh ssh`'d onto the right DEA - We then did `cat tmp/warden/cgroup/memory/instance-id/memory.usage_in_bytes` and got 939,515,904, which equates to 895.99ish MiB. Does anyone know why the latter is so high, and why it would differ from what the DEA reports back to the Cloud Controller? There's clearly a gap in our understanding somewhere, so any help would be much appreciated. Many thanks, Daniel Jones EngineerBetter.com
|
|
|
|
Failed to start Native apps in CF using null-build pack
Balaramaraju JLSP <balaramaraju@...>
Hi All,
We are unable to push a sample c++ application using null-buildpack; seems it has worked for others (as documented here : https://groups.google.com/a/cloudfoundry.org/forum/#!searchin/vcap-dev/null-buildpack/vcap-dev/oTYbHg_JJXU/_e30a2m3qr4J) , but we are not able to get it to work yet. Steps followed : 1. build a sample c++ application using g++ compiler on the linux vm; 2. Transfer that file to windows system; 3. Push that application using null-buildpack to both pivotal CF Source :- #include <stdio.h> int main(int argc, char* argv[]) { while(1==1) { printf("Hello World\n"); } return 0; } Build Command :- gcc -Wall helloWorld.c -o bin/helloWorld.sh OS :- Cent OS 6.5 x64 CF command [from win 7] :- D:\Cloud\Native>*cf push **helloWorld2 **-b https://github.com/ryandotsmith/null-buildpack <https://github.com/ryandotsmith/null-buildpack> --no-route -c "* *helloWorld2 **.sh"* CF logs :- *D:\Cloud\Native>cf push helloWorld2 -b https://github.com/ryandotsmith/null-buildpack <https://github.com/ryandotsmith/null-buildpack> --no-route -c "web: helloWorld2.sh" -s lucid64* *Using stack lucid64...* *OK* *Updating app helloWorld2 in org rootOrg / space development as .* *OK* *App helloWorld2 is a worker, skipping route creation* *Uploading helloWorld2...* *Uploading app files from: D:\Cloud\Native* *Uploading 6.6K, 2 files* *OK* *Stopping app helloWorld2 in org rootOrg / space development as .* *OK* *Starting app helloWorld2 in org rootOrg / space development as ...* *OK* *-----> Downloaded app package (4.0K)* *-----> Downloaded app buildpack cache (4.0K)* * Cloning into '/tmp/buildpacks/null-buildpack'...* * -----> Nothing to do.* *-----> Uploading droplet (4.0K)* *0 of 1 instances running, 1 down* *0 of 1 instances running, 1 down* *0 of 1 instances running, 1 down* Any help offered is greatly appreciated!!! Is this native application need to build on Ubuntu alone, since CF uses it ?
|
|
|
|
use null-buildpack to support standalone non-web
Balaramaraju JLSP <balaramaraju@...>
So far we are unable to push a sample c++ application using null-buildpack;
seems it has worked for others (as documented here : https://groups.google.com/a/cloudfoundry.org/forum/#!searchin/vcap-dev/null-buildpack/vcap-dev/oTYbHg_JJXU/_e30a2m3qr4J) , but we are not able to get it to work yet. Steps followed : 1. build a sample c++ application using g++ compiler on the linux vm; 2. Transfer that file to windows system; 3. Push that application using null-buildpack to both pivotal CF Source :- #include <stdio.h> int main(int argc, char* argv[]) { while(1==1) { printf("Hello World\n"); } return 0; } Build Command :- gcc -Wall helloWorld.c -o bin/helloWorld.sh OS :- Cent OS 6.5 x64 CF command [from win 7] :- D:\Cloud\Native>*cf push helloWorld -b https://github.com/ryandotsmith/null-buildpack <https://github.com/ryandotsmith/null-buildpack> --no-route -c "helloWorld.sh"* is this native application need to build on Ubuntu alone, since CF uses it ? CF logs :- *Creating app helloworld1 in org ClinicalPF / space ClinicalPF_learning as santhosh.kulandaiyan(a)philips.com <santhosh.kulandaiyan(a)philips.com>...* *OK* *App helloworld1 is a worker, skipping route creation* *Uploading helloworld1...* *Uploading app files from: d:\pcf\cpp\sample* *Uploading 6.9K, 1 files* *Done uploading* *OK* *Starting app helloworld1 in org ClinicalPF / space ClinicalPF_learning as santhosh.kulandaiyan(a)philips.com <santhosh.kulandaiyan(a)philips.com>...* *-----> Downloaded app package (4.0K)* * Cloning into '/tmp/buildpacks/null-buildpack'...* *fatal: unable to access 'https://github.com/ryandotsmith/null-buildpack/ <https://github.com/ryandotsmith/null-buildpack/>': Failed connect to github.com:443 <http://github.com:443>; Operation now in progress* -- App logs from Server : *2015-04-22 11:03:26 +0000 [STG]* ERR Cloning into '/tmp/buildpacks/null-buildpack'... *2015-04-22 11:03:26 +0000 [STG]* OUT -----> Downloaded app package (4.0K) *2015-04-22 11:03:26 +0000 [API]* OUT Updated app with guid f9b1e566-23c4-469b-b2b1-035c200380be ({"state"=>"STARTED"}) *2015-04-22 11:03:25 +0000 [DEA]* OUT Got staging request for app with id f9b1e566-23c4-469b-b2b1-035c200380be *2015-04-22 11:03:19 +0000 [API]* OUT Created app with guid f9b1e566-23c4-469b-b2b1-035c200380be J L S P Balaramaraju
|
|
|
|
Logging Infrastructure for CF components
Ronak Banka
Hi everyone,
I have some queries regarding persistent storage of application logs and cf component logs . As per my understanding -->For application logs: we can send the application logs to doppler with help of metron agent and further stream using traffic controller (User Side). -->For CF component syslog: We can send cf component syslog via metron to custom syslog endpoint (followed by parsing and other mining stuff) On the operator side how can we store "Application logs for all the applications" to a persistent storage?? If i look at firehose (or using noaa to get all the logs), it is more like pulling model from end component side so if components before it are slow in sending logs then there will be buffers . Application logs are distributed on different doppler nodes based on AZ of metron and doppler itself , so is there a need to implement something in between doppler and traffic controller to push all the application logs ?? Thanks Ronak Banka
|
|
|
|
TOSCA Compliance within Cloud Foundry
Deepak Vij
Hi folks, I would like to start an email discussion on this particular topic. As we all know that "BOSH" is the prevalent orchestration mechanism employed within Cloud Foundry. However, TOSCA (Topology and Orchestration Specification for Cloud Applications) standard based orchestration is getting big traction in the software industry recently. I brought this issue up in one of the recent CAB meetings as far as support for TOSCA standard within Cloud Foundry environment. At that time, folks from IBM & SAP did show interest in pursuing this further. I would like to follow up on that in the hope of pursuing this further. Following is a brief description to this regards.
Because of the complex service topologies which are especially present in the Cloud environment, the paradigm of model-driven management of services and their underlying infrastructure is getting a big traction in the software industry. TOSCA is such a standard supported by prominent companies in the industry. TOSCA essentially enables the specification of portable topology models. TOSCA has the combination of declarative descriptions of the application topology with all its components - including the load balancer, network, the compute resources, software and everything else, along with an imperative set of workflows to describe the logic of any process we need to automate. From Telecom industry perspectives, TOSCA is very good when it comes to defining virtual application topologies, dependencies and relationships, actions to be performed as part of a lifecycle. This significantly simplifies the complexities involved with exposing networking elements and end-to-end lifecycle management for NFV (Network Function Virtualization), by abstracting the networking piece of deployment into an application blueprint. The fact that industry such as Telecom is known to be very standards driven, TOSCA is becoming quite prominent as part of the ongoing "Network Function Virtualization" initiatives. Juju is yet another model-drive approach. However it is tightly bound to Ubuntu Linux. Based on my understanding, folks at Altoros have recently been able to deploy Cloud Foundry environment purely using Juju instead of BOSH. Maybe, we can leverage lot of this Juju work in order to be able to port "BOSH" environment towards standards based TOSCA environment. I think pursuing this effort would be a win-win for the whole CF community. I would look forward to feedback from the community on this topic. Thanks. Regards, Deepak Vij
|
|
|
|
Re: [cf-bosh] Links to Nabble archives of the CF lists
Christopher B Ferris <chrisfer@...>
+! nice job!
toggle quoted messageShow quoted text
Cheers, Christopher Ferris IBM Distinguished Engineer, CTO Open Cloud IBM Software Group, Open Technologies email: chrisfer(a)us.ibm.com twitter: @christo4ferris blog: http://thoughtsoncloud.com/index.php/author/cferris/ phone: +1 508 667 0402 From: Chip Childers <cchilders(a)cloudfoundry.org> To: "Huber, Aaron M" <aaron.m.huber(a)intel.com> Cc: "cf-dev(a)lists.cloudfoundry.org" <cf-dev(a)lists.cloudfoundry.org>, "cf-lattice(a)lists.cloudfoundry.org" <cf-lattice(a)lists.cloudfoundry.org>, "cf-bosh(a)lists.cloudfoundry.org" <cf-bosh(a)lists.cloudfoundry.org> Date: 05/06/2015 01:51 PM Subject: Re: [cf-dev] [cf-bosh] Links to Nabble archives of the CF lists Sent by: cf-dev-bounces(a)lists.cloudfoundry.org Thanks Aaron! Chip Childers | Technology Chief of Staff | Cloud Foundry Foundation
On Wed, May 6, 2015 at 4:28 PM, Huber, Aaron M <aaron.m.huber(a)intel.com>
wrote: I’ve created Nabble archives of the CF lists here: http://cf-bosh.70367.x6.nabble.com/ http://cf-dev.70369.x6.nabble.com/ http://cf-lattice.70370.x6.nabble.com/ The archives are searchable and allow web viewing of the mailing list without subscribing via email. There is also an RSS feed for each list. Aaron _______________________________________________ cf-bosh mailing list cf-bosh(a)lists.cloudfoundry.org https://lists.cloudfoundry.org/mailman/listinfo/cf-bosh _______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
|
|
|
|
Re: [cf-bosh] Links to Nabble archives of the CF lists
Chip Childers <cchilders@...>
Thanks Aaron!
Chip Childers | Technology Chief of Staff | Cloud Foundry Foundation On Wed, May 6, 2015 at 4:28 PM, Huber, Aaron M <aaron.m.huber(a)intel.com> wrote: I’ve created Nabble archives of the CF lists here:
|
|
|
