Runtime PMC: 2015-05-19 Notes
Eric Malm <emalm@...>
Hi, all,
The Runtime PMC met on Tuesday, 2015-05-19. Permanent notes are available at: https://github.com/cloudfoundry/pmc-notes/blob/master/Runtime/2015-05-19-runtime.md and are included below. Best, Eric --- *# Runtime PMC Meeting 2015-05-19* *## Agenda* 1. Current Backlog and Priorities 1. PMC Lifecycle Activities 1. Open Discussion *## Attendees* * Chip Childers, Cloud Foundry Foundation * Matt Sykes, IBM * Atul Kshirsagar, GE * Erik Jasiak, Pivotal * Sree Tummidi, Pivotal * Eric Malm, Pivotal * Shannon Coen, Pivotal * Will Pragnell, Pivotal * Marco Nicosia, Pivotal *## Current Backlog and Priorities* *### Runtime* * Shannon filling in for Dieu this week * support for context-based routing; delivered * investigating query performance * addressing outstanding pull requests * bump to UAA * issues with loggregator in acceptance environment, blocker to cutting stabilization release for collector *### Diego* * ssh access largely done, currently working routing ssh traffic to proxy * performance breadth: completed 50 cell test, investigating bulk processing in jobs that do so * refining CI to improve recording compatible versions of Diego and CF * processing of PRs from Garden and Lattice are prioritized * Stories queued up to investigate securing identified gaps in Diego *### UAA* * 2.2.6, 2.3.0 releases, notes available * upgraded Spring versions * update to JRE expected in v210 of cf-release * more LDAP work, chaining in identity zone: both LDAP and internal authentication can work simultaneously * support for New Relic instrumentation, will appear after v209 * upcoming: * risk assessment of persistent token storage: understand performance implications * starting work on password policy: multi-tenant for default zone and additional zones * OAuth client groups: authorization to manage clients * SAML support * question from Matt Sykes: * would like to discuss IBM PR for UAA DB migration strategy with the team *### Garden* * investigating management of disk quotas * replacing C/Bash code with Go to enable instrumentation, security, and maintainability * planning to remove default VCAP user in Garden *### Lattice* * nearly done with last stories before releasing 0.2.5 * Cisco contributed openstack support * baking deployment automation into published images on some providers * improved documentation for how to install lattice on VMs * next work planned is support for CF-like app lifecycle management (pushing code in addition to docker) *### TCP Router* * building out icebox to reflect inception * question from Matt Sykes: * how to incorporate new project into PMC? IBM parties surprised with announcement at Summit * Chip: inconsistent policy so far; maybe this belongs alongside gorouter in Runtime PMC * working on process for review, discussion of incubating project * Shannon: first step will be to produce proposal, discuss with community *### LAMB* * big rewind project on datadog firehose nozzle: limitation in doppler about size of messages, dropping messages * working to resolve those problems: improving number of concurrent reads, marshaling efficiency * seeing increases in message loss in Runtime environments: may be other source of contention, working with them to resolve * Datadog nozzle work: * looking at developing a Graphite nozzle from community work * will investigate community interest in Graphite support * naming alignment from loggregator to doppler * instrumentation of statsd for larger message sizes, work to phase out collector and NATS in CF * goal is to stream metrics directly to firehose * question from Matt Sykes: story about protobuf protocol proposal * best way to support vm tagging in log messages: distinguish between types of data in log messages * goal would be to improve the implementation: more generic API for message data; understand implications of this change *### Greenhouse* * Accepted code from HP * will get support from Microsoft with regard to interest in entire Microsoft stack *## PMC Lifecycle Activities* None to report. *## Open Discussion* None to report.
|
|
Re: cf-release v209 published
Simon Johansson <simon@...>
i wanted to share the great news that the new skinny buildpacks reducedthe size of cf-release from 5.2gb -> 3.5gb! This is great news, good job buildpack team! On Thu, May 21, 2015 at 4:40 PM, James Bayer <jbayer(a)pivotal.io> wrote: skinny buildpacks refer to each buildpack no longer shipping old
|
|
Question about services on Cloud Foundry
Kinjal Doshi
Hi,
From the architecture point of view I understand that there are no service explicitly associated with CF. However, the following doc is very confusing: http://docs.cloudfoundry.org/devguide/services/managed.html Would be great if some one can explain the meaning of manages services her. Thanks, Kinjal
|
|
Re: List Reply-To behavior
James Bayer
yes, this has affected me
toggle quoted messageShow quoted text
On Fri, May 22, 2015 at 4:33 AM, Daniel Mikusa <dmikusa(a)pivotal.io> wrote:
--
Thank you, James Bayer
|
|
Re: List Reply-To behavior
Daniel Mikusa
On Fri, May 22, 2015 at 6:22 AM, Matthew Sykes <matthew.sykes(a)gmail.com>
wrote: The vcap-dev list used to use a Reply-To header pointing back to the list+1 and +1 Dan
|
|
List Reply-To behavior
Matthew Sykes <matthew.sykes@...>
The vcap-dev list used to use a Reply-To header pointing back to the list
such that replying to a post would automatically go back to the list. The current mailman configuration for cf-dev does not set a Reply-To header and the default behavior is to reply to the author. While I understand the pros and cons of setting the Reply-To header, this new behavior has bitten me several times and I've found myself re-posting a response to the list instead of just the author. I'm interested in knowing if anyone else has been bitten by this behavior and would like a Reply-To header added back... Thanks. -- Matthew Sykes matthew.sykes(a)gmail.com
|
|
Re: container cannot communicate with the host
Matthew Sykes <matthew.sykes@...>
Warden explicitly disables access to the container host. If you move up to
toggle quoted messageShow quoted text
a more recent level of cf-release, that behavior is configurable with the `allow_host_access` flag. When that flag is true, this line is skipped: https://github.com/cloudfoundry/warden/blob/4f1e5c049a12199fdd1f29cde15c9a786bd5fac8/warden/root/linux/net.sh#L128 At the level you're at, that rule is always specified so you'd have to manually change it. https://github.com/cloudfoundry/warden/blob/17f34e2d7ff1994856a61961210a82e83f24ecac/warden/root/linux/net.sh#L124
On Fri, May 22, 2015 at 3:21 AM, Youzhi Zhu <zhuyouzhi03(a)gmail.com> wrote:
Hi all --
Matthew Sykes matthew.sykes(a)gmail.com
|
|
Re: container cannot communicate with the host
Lev Berman <lev.berman@...>
As far as I know, it is so by design - in order to setup a connection to
toggle quoted messageShow quoted text
the same host you need to explicitly tell Warden to allow external traffic - https://github.com/cloudfoundry/warden/blob/master/warden/README.md#net-handle-out-addressmaskport . In more details: 1) ssh into your VM with DEA 2) find your Warden handle in /var/vcap/data/dea_ng/db/instances.json - "warden_handle" field for the hash describing your specific application ("application_id" value is the same as cf app --guid) 3) cd into /var/vcap/packages/warden/warden 4) bundle install 5) ./bin/warden --socket /var/vcap/data/warden/warden.sock 6) > net_out --handle <your handle from instances.json> --port <your port to open> This is for CF v208, an earlier version of Warden client may have slightly different API - see command help.
On Fri, May 22, 2015 at 10:21 AM, Youzhi Zhu <zhuyouzhi03(a)gmail.com> wrote:
Hi all --
Lev Berman Altoros - Cloud Foundry deployment, training and integration Github *: https://github.com/ldmberman <https://github.com/ldmberman>*
|
|
container cannot communicate with the host
Youzhi Zhu
Hi all
I have an app A and a service B, service B is running on the dea server(ip 10.0.0.254), app A need to connect with service B through tcp, it works normally in my LAN, but when I push A to cf, it cannot connect to B, then I execute bin/wsh to get into the container and ping the host ip, it's unreachable, as below: *root(a)18mkbd9n808:~# ping 10.0.0.254PING 10.0.0.254 (10.0.0.254) 56(84) bytes of data.From 10.0.0.254 icmp_seq=1 Destination Port UnreachableFrom 10.0.0.254 icmp_seq=2 Destination Port Unreachable^C--- 10.0.0.254 ping statistics ---2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 1002ms* But if I ping another host in the LAN. it can be reached!!! *root(a)18mkbd9n808:~# ping 10.0.0.253PING 10.0.0.253 (10.0.0.253) 56(84) bytes of data.64 bytes from 10.0.0.253 <http://10.0.0.253>: icmp_seq=1 ttl=63 time=1.60 ms64 bytes from 10.0.0.253 <http://10.0.0.253>: icmp_seq=2 ttl=63 time=0.421 ms^C--- 10.0.0.253 ping statistics ---2 packets transmitted, 2 received, 0% packet loss, time 1001msrtt min/avg/max/mdev = 0.421/1.013/1.606/0.593 ms* It's wired!!! my cf-release is cf-175 and I have only one dea server.Does anyone met this situation before? thanks!
|
|
Re: Setting up API endpoint failed in Local CF
Balaramaraju JLSP <balaramaraju@...>
Hi All,
I found the issue , got the details from deploy.yml thanks Balaramaraju On Fri, May 22, 2015 at 10:44 AM, Balaramaraju JLSP <balaramaraju(a)gmail.com> wrote: Hi All, -- J L S P Balaramaraju
|
|
Setting up API endpoint failed in Local CF
Balaramaraju JLSP <balaramaraju@...>
Hi All,
using the *https://github.com/yudai/cf_nise_installer <https://github.com/yudai/cf_nise_installer>* i install local CF and able to the start services ".\scripts\start.sh" logs:- All processes have been started! -u admin -p c1oudc0w --skip-ssl-validation'ip.io Download CF CLI from https://github.com/cloudfoundry/cli but while setting up the endpiont is it failing vagrant(a)vagrant-ubuntu-trusty-64:/vagrant$ cf api --skip-ssl-validation https://api.vagrant-ubuntu-tip.io-64.io Setting api endpoint to https://api.vagrant-ubuntu-trusty-64.io... FAILED Error performing request: Get https://api.vagrant-ubuntu-trusty-64.ip.io/v2/info: dial tcp 50.21.180.100:443: i/o timeout any help is appreciated . -- Balaramaraju
|
|
Re: cf-release v209 published
James Bayer
skinny buildpacks refer to each buildpack no longer shipping old
unsupported or insecure versions of runtimes. you can still customize the buildpacks to include older runtimes by building the buildpack yourself with the admin buildpack feature. On Thu, May 21, 2015 at 7:34 AM, Long Nguyen <long.nguyen11288(a)gmail.com> wrote: Wooot! Thanks James. Is skinny just having latest 2 version of language? -- Thank you, James Bayer
|
|
Re: cf-release v209 published
Long Nguyen
Wooot! Thanks James. Is skinny just having latest 2 version of language?
toggle quoted messageShow quoted text
On May 21, 2015 at 1:42:11 AM, James Bayer (jbayer(a)pivotal.io) wrote:
more info is coming soon, we don't have all of the release notes published yet because dieu and shannon are out of the office. i wanted to share the great news that the new skinny buildpacks reduced the size of cf-release from 5.2gb -> 3.5gb! the "what's in the deploy" file is awaiting approval b/c of mailman limits. thanks buildpacks team! -- Thank you, James Bayer _______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
|
|
Re: cf-release v209 published
Wayne E. Seguin
That's an incredible savings and excellent to see, thanks Buildpacks team!
toggle quoted messageShow quoted text
:)
On Thu, May 21, 2015 at 1:42 AM, James Bayer <jbayer(a)pivotal.io> wrote:
more info is coming soon, we don't have all of the release notes published
|
|
cf-release v209 published
James Bayer
more info is coming soon, we don't have all of the release notes published
yet because dieu and shannon are out of the office. i wanted to share the great news that the new skinny buildpacks reduced the size of cf-release from 5.2gb -> 3.5gb! the "what's in the deploy" file is awaiting approval b/c of mailman limits. thanks buildpacks team! -- Thank you, James Bayer
|
|
cf-release v209 published
James Bayer
more info is coming soon, we don't have all of the release notes published
yet because dieu and shannon are out of the office. i wanted to share the great news that the new skinny buildpacks reduced the size of cf-release from 5.2gb -> 3.5gb! thanks buildpacks team! -- Thank you, James Bayer
|
|
Buildpacks PMC - 2015-05-20 Notes
Mike Dalessio
Howdy all,
We had a meeting of the Buildpacks PMC today, which was only lightly attended and we adjourned quickly. Permanent notes are at: https://github.com/cloudfoundry/pmc-notes/blob/master/Buildpacks/2015-05-20-buildpacks.md but I've helpfully also included a snapshot of those notes below. Happy Wednesday! -mike --- *# Buildpacks PMC Meeting 2015-05-20* *## Agenda* 1. Update on Java Buildpack (Ryan Morgan) 2. Update on core Buildpacks (Mike Dalessio) 3. Open Discussion *## Attendees* * Chip Childers, Cloud Foundry Foundation * Mike Dalessio, Pivotal (PMC lead) * Matthew Sykes, IBM *## Update on Java Buildpack (Ryan Morgan)* * Added support for Wily Introscope. * Memory calculator re-written in Go and supports java memory configuration at startup time rather than staging time. Some details on this feature were posted to cf-dev by Chris Frost last week. * Team currently discussing how to 'pin' buildpack dependency versions to allow for repeatable offline buildpack creation. * Team also discussing moving from Jenkins to Concourse for CI. Mike will follow up with the team to discuss how `buildpack-packager` might be used to pin cached dependencies. *## Update on core Buildpacks (Mike Dalessio)* The [binary buildpack][binary] was added into `cf-release` last week ([PR here][binary-pr]), and moved into the `cloudfoundry` github org. This is the same idea as what's commonly called a "null" buildpack, where developers can simply execute a binary at runtime. The [staticfile buildpack][static] was added into `cf-release` last week ([PR here][static-pr]), and moved into the `cloudfoundry` github org. Originally created by Dr. Nic, this buildpacks allows a static website to be published behind nginx, and for nginx to be configured in a few interesting ways. At long last, [skinny buildpacks][skinny] made it into `cf-release` ([PRs here][skinny-pr]). There was also some interesting discussion on the mailing lists, both [old][skinny-thread1] and [new][skinny-thread2]. Notable near-term goals: * ability to generate and test CF rootfs-specific binaries; and tooling for CF operators to do the same * work more closely with the java-buildpacks team *## Open Discussion* Mike: Just want to note that IBM open-sourced their linux-based ASP.NET/Mono buildpack this week. Looks awesome! https://github.com/cloudfoundry-community/asp.net5-buildpack --- [binary]: https://github.com/cloudfoundry/binary-buildpack [binary-pr]: https://github.com/cloudfoundry/cf-release/pull/677 [static]: https://github.com/cloudfoundry/staticfile-buildpack [static-pr]: https://github.com/cloudfoundry/cf-release/pull/668 [skinny]: https://github.com/cloudfoundry-incubator/buildpack-packager/issues/4 [skinny-pr]: https://github.com/cloudfoundry/cf-release/pulls?utf8=%E2%9C%93&q=is%3Apr+buildpack+skinny+ [skinny-thread1]: https://groups.google.com/a/cloudfoundry.org/forum/#!searchin/vcap-dev/addressing$20buildpack/vcap-dev/1HmGK4wU3Rc/lk186OOtdbMJ [skinny-thread2]: http://lists.cloudfoundry.org/pipermail/cf-dev/2015-May/000005.html
|
|
Re: [vcap-dev] Java OOM debugging
Daniel Mikusa
On Thu, May 14, 2015 at 10:23 AM, Daniel Jones <
daniel.jones(a)engineerbetter.com> wrote: Hi Lari,Since it seems activity related, have you tried monitoring the number of threads in the JVM? While you can cap the number of threads Tomcat uses for processing requests, I don't believe that you can cap the number of threads it's possible to create in the JVM. The reason I mention this is because each thread causes the amount of memory required to go up by the thread stack size (Xss * <threads> = total thread memory). Perhaps as activity increases, so does the thread count and that's pushing you over the limit. Are you setting a custom -Xss value or using the default? From memory, the default is pretty large. If you're not using a custom one, you might try a smaller one, like 256k and see if that has any impact on the problem. Alternatively, you could adjust the memory weightings in the build pack so that heap consumes a smaller amount of the total memory and there's more memory available for native / stack and other memory. I'm not sure I follow your question, but in terms of NIO I suspect the JVM is going to handle memory allocation not Tomcat. Given that, I it should happen just like any other Java code that uses the NIO. APR is unlikely to be an issue, see my next comment. Is there a way of telling from pmap whether pages are being used for NIOUnless you compile the APR native library and include it with your version of the build pack, Tomcat's not going to use it. It'll use NIO by default, with Tomcat version 8. You can confirm by looking at the logs when you start your app. One of the first things it logs will be the protocol handler. This is from a test app, where it's using NIO. Ex: ``` [CONTAINER] org.apache.coyote.http11.Http11NioProtocol INFO Initializing ProtocolHandler ["http-nio-63227"] ``` Dan
|
|
Re: [vcap-dev] Java OOM debugging
Daniel Mikusa
On Thu, May 14, 2015 at 2:59 PM, Lari Hotari <Lari(a)hotari.net> wrote:
On 15-05-14 10:23 AM, Daniel Jones wrote:Sorry for the duplicate. Resending as my first reply didn't go back to theThanks again for your input. Have you seen this problem with versionsI don't have proper data gathered from older than 8.0.20, so I cannot list. Since the Connector in server.xml is not specifically setting an implementation it should use the NIO connector, which is the default in Tomcat 8. A quick test on PWS confirmed this for me. https://github.com/cloudfoundry/java-buildpack/blob/master/resources/tomcat/conf/server.xml#L22 It would be interesting to see if the BIO or APR connectors have similar issues. BIO would be easy to test, just add `protocol="org.apache.coyote.http11.Http11Protocol"` to the Connector tag on line #22. APR would be trickier as you'd need to compile the native library and pull that into the environment. Dan
|
|
Announcing cf-riak-cs-release v10
Marco Nicosia
On behalf of the CF Core Services team, I am pleased to announce v10 of
cf-riak-cs-release <https://github.com/cloudfoundry/cf-riak-cs-release>. cf-riak-cs-release <https://github.com/cloudfoundry/cf-riak-cs-release> is a BOSH release that delivers an S3-compatible, highly available object store-as-a-service for Cloud Foundry. Through Cloud Foundry, users can provision S3-style buckets and deliver unique credentials to bound applications. v10 is a minor update, mostly enabling additional security and some additional documentation: - Default to SSL endpoints for communication between Cloud Controller and Broker. 93485814 <https://www.pivotaltracker.com/story/show/93485814> - Improve documentation around contributions. 90029504 <https://www.pivotaltracker.com/story/show/90029504> - Expose ssl_enabled flag to enforce SSL. 80055322 <https://www.pivotaltracker.com/story/show/93485814> - Addition of a small utility script to run the broker-registrar errand on bosh-lite. - Deployment manifest stub changes We've also promoted the domain properties to top-level properties in manifests: - domain is now .properties.domain - meta.cf.apps_domain is now .properties.app_domains.[0] All this and more can be found on Github <https://github.com/cloudfoundry/cf-riak-cs-release> and bosh.io <http://bosh.io/releases/github.com/cloudfoundry/cf-riak-cs-release?version=10> <http:>. -- Marco Nicosia Product Manager Pivotal Software, Inc.
|
|