cf-dev@lists.cloudfoundry.org

Re: Buildpacks PMC - 2015-05-04 Notes

Mike Dalessio

#48

Hi Ryan,

Thanks for asking this question.

The "risk" called out in the inception encompassed a number of things, but
what they really all boil down to is that the java-buildpacks team has its
own roadmap and conventions; and the two teams don't often communicate
about sharing resources, infrastructure, or planning.

I think a reasonable first step is for you and I (and maybe JT and Ben, the
engineering anchors for each team) to have a regular chat on our calendars.
I'd prefer not to get bitten by Conway's Law if we can easily mitigate this
risk. I'll ship you a calendar invite; as well as make sure the
java-buildpack team, as well as voting members of the PMC, are represented
in the next inception or roadmap discussion.

-m

Show quoted text

On Mon, May 4, 2015 at 2:43 PM, Ryan Morgan <ryanmorgan(a)gmail.com> wrote:


Thanks for the update Mike.  Can we get a bit more detail on
java-buildpack divergence from the other buildpacks?

-Ryan

On Mon, May 4, 2015 at 10:50 AM, Mike Dalessio <mdalessio(a)pivotal.io>
wrote:

Hi all,

We held the first Buildpacks PMC meeting today; I'd like to share the
agenda and notes.

For reference, all agendas notes for the Buildpacks PMC will be kept in a
public Google Drive folder at this URL:

http://bit.ly/cf-buildpacks-pmc


I realize GDrive isn't the most convenient medium for some in the CF
community; I'd love to hear how we can better support transparency for
everyone.

Please feel free to respond with comments and questions!

Cheers,
-m

----

Attendees:

   -

   Chip Childers, Cloud Foundry Foundation
   -

   Mike Dalessio, Pivotal (PMC lead)
   -

   Christopher Ferriss, IBM
   -

   Michael Fraenkel, IBM
   -

   Mark Kropf, Pivotal



Recent Inception Report and Stated Goals

The Buildpacks core development team held a project inception on
2015-04-20, to gain a shared understanding of upcoming goals and tracks of
work.


Goals


   - Expand supported ecosystem to include more languages & frameworks
      - Cloud Foundry ownership of Buildpacks
      - Leverage new primitives in Diego (“app lifecycle”)
      - Enable 3rd party extensions to the Developer experience
      - Enable application developer extensions to the Developer
      experience
      - Set patterns for creating new buildpacks and for extending the
      Developer experience
      - Generate clearer diagnostics during staging
      - Enable Operator ease of updating common dependencies
      - Keep the `bin/detect` experience: buildpacks should Just Work™
      - Exert more ownership over the rootfs
      - Binary buildpack support


Risks


   - java-buildpack is diverging quickly from the core buildpacks
      - Lack of deep experience in some ecosystems
      - Wide variety in implementations across buildpacks
      - rootfs: with great power comes great responsibility (e.g.,
      security response)
      - tight coupling between buildpacks and rootfs
      - versioning between buildpacks and rootfs


Current Backlog and Priorities

See https://www.pivotaltracker.com/n/projects/1042066

Notable near-term goals:


   -

   staticfile-buildpack support in `cf-release`
   -

   binary buildpack (a.k.a. “null buildpack”) support in `cf-release`
   -

   ability to generate and test CF rootfs-specific binaries; and tooling
   for CF operators to do the same



Proposal: Buildpack Incubation Process

Discussion today for PMC input; a draft document will be circulated for
comment to cf-dev@ mailing list after the meeting, in a separate thread.




_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

attachment.html

Re: Discrepancy between `cf apps` memory usage and cgroup's memory.usage_in_bytes in CF 2.13.0

Matthew Sykes <matthew.sykes@...>

#47

The reported statistic is calculated here:

https://github.com/cloudfoundry/dea_ng/blob/310797e1097dcd5531bff4077ccd8f02f6091219/lib/dea/stat_collector.rb#L92-L94

On Thu, May 7, 2015 at 8:28 AM, Daniel Jones <
daniel.jones(a)engineerbetter.com> wrote:

Hi all,

Whilst investigating the Java Buildpack out-of-memory issues David
Head-Rapson mailed about the other day, we discovered a discrepancy between
the memory usage stat provided by `cf app` and the value stored in the
corresponding cgroup's `memory.usage_in_bytes` file. The latter seems to be
bumping right along the maximum allowed.


   - We did a `cf app`, and got a memory stat of 847.6MiB of 896MiB.
   - We got the appId from CF_TRACE, `bosh ssh`'d onto the right DEA
   - We then did `cat
   tmp/warden/cgroup/memory/instance-id/memory.usage_in_bytes` and got
   939,515,904, which equates to 895.99ish MiB.

Does anyone know why the latter is so high, and why it would differ from
what the DEA reports back to the Cloud Controller? There's clearly a gap in
our understanding somewhere, so any help would be much appreciated.

Many thanks,

Daniel Jones
EngineerBetter.com

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


-- 
Matthew Sykes
matthew.sykes(a)gmail.com

attachment.html

Re: use null-buildpack to support standalone non-web

Daniel Mikusa

#46

On Thu, May 7, 2015 at 2:09 AM, Balaramaraju JLSP <balaramaraju(a)gmail.com>
wrote:

So far we are unable to push a sample c++ application using
null-buildpack; seems it has worked for others (as documented here :
https://groups.google.com/a/cloudfoundry.org/forum/#!searchin/vcap-dev/null-buildpack/vcap-dev/oTYbHg_JJXU/_e30a2m3qr4J)
, but we are not able to get it to work yet.

Steps followed :

1. build a sample c++ application using g++ compiler on the linux vm;

2. Transfer that file to windows system;

3. Push that application using null-buildpack to both pivotal CF

Source :-

#include <stdio.h>

int main(int argc, char* argv[]) {

  while(1==1) {

    printf("Hello World\n");

  }

  return 0;

}

Build Command :- gcc -Wall helloWorld.c -o bin/helloWorld.sh

OS :- Cent OS 6.5 x64

CF command [from win 7] :- D:\Cloud\Native>*cf push helloWorld -b
https://github.com/ryandotsmith/null-buildpack
<https://github.com/ryandotsmith/null-buildpack> --no-route -c
"helloWorld.sh"*

is this native application need to build on Ubuntu alone, since CF uses it
?
I think it would depend, you'd probably be OK with your simple hello world
app.  Something more complicated using third party libraries, may have
trouble.  I'd personally try to avoid that and just compile on a machine
that is as close to the runtime environment as possible.   To match the
cflinuxfs2 stack, you could use an Ubuntu 14.04 VM (Vagrant makes this very
easy) or this docker repo.

   https://registry.hub.docker.com/u/cloudfoundry/cflinuxfs2/

CF logs :-

*Creating app helloworld1 in org ClinicalPF / space ClinicalPF_learning as
santhosh.kulandaiyan(a)philips.com <santhosh.kulandaiyan(a)philips.com>...*

*OK*

*App helloworld1 is a worker, skipping route creation*

*Uploading helloworld1...*

*Uploading app files from: d:\pcf\cpp\sample*

*Uploading 6.9K, 1 files*

*Done uploading*

*OK*

*Starting app helloworld1 in org ClinicalPF / space ClinicalPF_learning as
santhosh.kulandaiyan(a)philips.com <santhosh.kulandaiyan(a)philips.com>...*

*-----> Downloaded app package (4.0K)*

*    Cloning into '/tmp/buildpacks/null-buildpack'...*

*fatal: unable to access 'https://github.com/ryandotsmith/null-buildpack/
<https://github.com/ryandotsmith/null-buildpack/>': Failed connect to
github.com:443 <http://github.com:443>; Operation now in progress*

--

This looks like you're having trouble connecting to github to download the
build pack.  Do you have a network or security group that might prevent
this?

Another option if you have admin access would be to download the src [1]
and use `cf create-buildpack` to upload the build pack manually.  You can
then just use "-b null_buildpack" or whatever name you give it with `cf
create-buildpack`.

[1] - https://github.com/ryandotsmith/null-buildpack/archive/master.zip

Dan

App logs from Server :

*2015-04-22 11:03:26 +0000 [STG]* ERR Cloning into
'/tmp/buildpacks/null-buildpack'...

*2015-04-22 11:03:26 +0000 [STG]* OUT -----> Downloaded app package (4.0K)

*2015-04-22 11:03:26 +0000 [API]* OUT Updated app with guid
f9b1e566-23c4-469b-b2b1-035c200380be ({"state"=>"STARTED"})

*2015-04-22 11:03:25 +0000 [DEA]* OUT Got staging request for app with id
f9b1e566-23c4-469b-b2b1-035c200380be

*2015-04-22 11:03:19 +0000 [API]* OUT Created app with guid
f9b1e566-23c4-469b-b2b1-035c200380be

J L S P Balaramaraju

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

attachment.html

Discrepancy between `cf apps` memory usage and cgroup's memory.usage_in_bytes in CF 2.13.0

Daniel Jones

#45

Hi all,

Whilst investigating the Java Buildpack out-of-memory issues David
Head-Rapson mailed about the other day, we discovered a discrepancy between
the memory usage stat provided by `cf app` and the value stored in the
corresponding cgroup's `memory.usage_in_bytes` file. The latter seems to be
bumping right along the maximum allowed.


   - We did a `cf app`, and got a memory stat of 847.6MiB of 896MiB.
   - We got the appId from CF_TRACE, `bosh ssh`'d onto the right DEA
   - We then did `cat
   tmp/warden/cgroup/memory/instance-id/memory.usage_in_bytes` and got
   939,515,904, which equates to 895.99ish MiB.

Does anyone know why the latter is so high, and why it would differ from
what the DEA reports back to the Cloud Controller? There's clearly a gap in
our understanding somewhere, so any help would be much appreciated.

Many thanks,

Daniel Jones
EngineerBetter.com

attachment.html

Failed to start Native apps in CF using null-build pack

Balaramaraju JLSP <balaramaraju@...>

#44

Hi All,

We are unable to push a sample c++ application using null-buildpack; seems
it has worked for others (as documented here :
https://groups.google.com/a/cloudfoundry.org/forum/#!searchin/vcap-dev/null-buildpack/vcap-dev/oTYbHg_JJXU/_e30a2m3qr4J)
, but we are not able to get it to work yet.

Steps followed :

1. build a sample c++ application using g++ compiler on the linux vm;

2. Transfer that file to windows system;

3. Push that application using null-buildpack to both pivotal CF

Source :-

#include <stdio.h>

int main(int argc, char* argv[]) {

  while(1==1) {

    printf("Hello World\n");

  }

  return 0;

}

Build Command :- gcc -Wall helloWorld.c -o bin/helloWorld.sh

OS :- Cent OS 6.5 x64

CF command [from win 7] :- D:\Cloud\Native>*cf push **helloWorld2 **-b
https://github.com/ryandotsmith/null-buildpack
<https://github.com/ryandotsmith/null-buildpack> --no-route -c "*
*helloWorld2 **.sh"*

CF logs :-

*D:\Cloud\Native>cf push helloWorld2 -b
https://github.com/ryandotsmith/null-buildpack
<https://github.com/ryandotsmith/null-buildpack> --no-route -c "web:
helloWorld2.sh" -s lucid64*

*Using stack lucid64...*

*OK*

*Updating app helloWorld2 in org rootOrg / space development as .*

*OK*

*App helloWorld2 is a worker, skipping route creation*

*Uploading helloWorld2...*

*Uploading app files from: D:\Cloud\Native*

*Uploading 6.6K, 2 files*

*OK*

*Stopping app helloWorld2 in org rootOrg / space development as .*

*OK*

*Starting app helloWorld2 in org rootOrg / space development as ...*

*OK*

*-----> Downloaded app package (4.0K)*

*-----> Downloaded app buildpack cache (4.0K)*

*    Cloning into '/tmp/buildpacks/null-buildpack'...*

*    -----> Nothing to do.*

*-----> Uploading droplet (4.0K)*

*0 of 1 instances running, 1 down*

*0 of 1 instances running, 1 down*

*0 of 1 instances running, 1 down*

Any help offered is greatly appreciated!!!

Is this native application need to build on Ubuntu alone, since CF uses it ?

attachment.html

use null-buildpack to support standalone non-web

Balaramaraju JLSP <balaramaraju@...>

#43

So far we are unable to push a sample c++ application using null-buildpack;
seems it has worked for others (as documented here :
https://groups.google.com/a/cloudfoundry.org/forum/#!searchin/vcap-dev/null-buildpack/vcap-dev/oTYbHg_JJXU/_e30a2m3qr4J)
, but we are not able to get it to work yet.

Steps followed :

1. build a sample c++ application using g++ compiler on the linux vm;

2. Transfer that file to windows system;

3. Push that application using null-buildpack to both pivotal CF

Source :-

#include <stdio.h>

int main(int argc, char* argv[]) {

  while(1==1) {

    printf("Hello World\n");

  }

  return 0;

}

Build Command :- gcc -Wall helloWorld.c -o bin/helloWorld.sh

OS :- Cent OS 6.5 x64

CF command [from win 7] :- D:\Cloud\Native>*cf push helloWorld -b
https://github.com/ryandotsmith/null-buildpack
<https://github.com/ryandotsmith/null-buildpack> --no-route -c
"helloWorld.sh"*

is this native application need to build on Ubuntu alone, since CF uses it ?

CF logs :-

*Creating app helloworld1 in org ClinicalPF / space ClinicalPF_learning as
santhosh.kulandaiyan(a)philips.com <santhosh.kulandaiyan(a)philips.com>...*

*OK*

*App helloworld1 is a worker, skipping route creation*

*Uploading helloworld1...*

*Uploading app files from: d:\pcf\cpp\sample*

*Uploading 6.9K, 1 files*

*Done uploading*

*OK*

*Starting app helloworld1 in org ClinicalPF / space ClinicalPF_learning as
santhosh.kulandaiyan(a)philips.com <santhosh.kulandaiyan(a)philips.com>...*

*-----> Downloaded app package (4.0K)*

*    Cloning into '/tmp/buildpacks/null-buildpack'...*

*fatal: unable to access 'https://github.com/ryandotsmith/null-buildpack/
<https://github.com/ryandotsmith/null-buildpack/>': Failed connect to
github.com:443 <http://github.com:443>; Operation now in progress*

--

App logs from Server :

*2015-04-22 11:03:26 +0000 [STG]* ERR Cloning into
'/tmp/buildpacks/null-buildpack'...

*2015-04-22 11:03:26 +0000 [STG]* OUT -----> Downloaded app package (4.0K)

*2015-04-22 11:03:26 +0000 [API]* OUT Updated app with guid
f9b1e566-23c4-469b-b2b1-035c200380be ({"state"=>"STARTED"})

*2015-04-22 11:03:25 +0000 [DEA]* OUT Got staging request for app with id
f9b1e566-23c4-469b-b2b1-035c200380be

*2015-04-22 11:03:19 +0000 [API]* OUT Created app with guid
f9b1e566-23c4-469b-b2b1-035c200380be

J L S P Balaramaraju

attachment.html

Logging Infrastructure for CF components

Ronak Banka

#42

Hi everyone,

I have some queries regarding persistent storage of application logs and cf
component logs .

As per my understanding

-->For application logs:
we can send the application logs to doppler with help of metron agent and
further stream using traffic controller (User Side).

-->For CF component syslog:
We can send cf component syslog via metron to custom syslog endpoint
(followed by parsing and other mining stuff)

On the operator side how can we store "Application logs for all the
applications" to a persistent storage??

If i look at firehose (or using noaa to get all the logs), it is more like
pulling model from end component side so if components before it are slow
in sending logs then there will be buffers .

Application logs are distributed on different doppler nodes based on AZ of
metron and doppler itself , so is there a need to implement something in
between doppler and traffic controller to push all the application logs ??

Thanks
Ronak Banka

attachment.html

TOSCA Compliance within Cloud Foundry

Deepak Vij

#41

Hi folks, I would like to start an email discussion on this particular topic. As we all know that "BOSH" is the prevalent orchestration mechanism employed within Cloud Foundry. However, TOSCA (Topology and Orchestration Specification for Cloud Applications) standard based orchestration is getting big traction in the software industry recently. I brought this issue up in one of the recent CAB meetings as far as support for TOSCA standard within Cloud Foundry environment. At that time, folks from IBM & SAP did show interest in pursuing this further. I would like to follow up on that in the hope of pursuing this further. Following is a brief description to this regards.

Because of the complex service topologies which are especially present in the Cloud environment, the paradigm of model-driven management of services and their underlying infrastructure is getting a big traction in the software industry. TOSCA is such a standard supported by prominent companies in the industry. TOSCA essentially enables the specification of portable topology models. TOSCA has the combination of declarative descriptions of the application topology with all its components - including the load balancer, network, the compute resources, software and everything else, along with an imperative set of workflows to describe the logic of any process we need to automate.

From Telecom industry perspectives, TOSCA is very good when it comes to defining virtual application topologies, dependencies and relationships, actions to be performed as part of a lifecycle. This significantly simplifies the complexities involved with exposing networking elements and end-to-end lifecycle management for NFV (Network Function Virtualization), by abstracting the networking piece of deployment into an application blueprint. The fact that industry such as Telecom is known to be very standards driven, TOSCA is becoming quite prominent as part of the ongoing "Network Function Virtualization" initiatives.

Juju is yet another model-drive approach. However it is tightly bound to Ubuntu Linux. Based on my understanding, folks at Altoros have recently been able to deploy Cloud Foundry environment purely using Juju instead of BOSH. Maybe, we can leverage lot of this Juju work in order to be able to port "BOSH" environment towards standards based TOSCA environment.

I think pursuing this effort would be a win-win for the whole CF community. I would look forward to feedback from the community on this topic. Thanks.

Regards,
Deepak Vij

attachment.html

Re: [cf-bosh] Links to Nabble archives of the CF lists

Christopher B Ferris <chrisfer@...>

#40

+! nice job!

Cheers,

Christopher Ferris
IBM Distinguished Engineer, CTO Open Cloud
IBM Software Group, Open Technologies
email: chrisfer(a)us.ibm.com
twitter: @christo4ferris
blog: http://thoughtsoncloud.com/index.php/author/cferris/
phone: +1 508 667 0402



From:	Chip Childers <cchilders(a)cloudfoundry.org>
To:	"Huber, Aaron M" <aaron.m.huber(a)intel.com>
Cc:	"cf-dev(a)lists.cloudfoundry.org"
            <cf-dev(a)lists.cloudfoundry.org>,
            "cf-lattice(a)lists.cloudfoundry.org"
            <cf-lattice(a)lists.cloudfoundry.org>,
            "cf-bosh(a)lists.cloudfoundry.org"
            <cf-bosh(a)lists.cloudfoundry.org>
Date:	05/06/2015 01:51 PM
Subject:	Re: [cf-dev] [cf-bosh] Links to Nabble archives of the CF lists
Sent by:	cf-dev-bounces(a)lists.cloudfoundry.org



Thanks Aaron!

Chip Childers | Technology Chief of Staff | Cloud Foundry Foundation

Show quoted text

On Wed, May 6, 2015 at 4:28 PM, Huber, Aaron M <aaron.m.huber(a)intel.com>
wrote:
  I’ve created Nabble archives of the CF lists here:

  http://cf-bosh.70367.x6.nabble.com/

  http://cf-dev.70369.x6.nabble.com/

  http://cf-lattice.70370.x6.nabble.com/

  The archives are searchable and allow web viewing of the mailing list
  without subscribing via email.  There is also an RSS feed for each list.

  Aaron

  _______________________________________________
  cf-bosh mailing list
  cf-bosh(a)lists.cloudfoundry.org
  https://lists.cloudfoundry.org/mailman/listinfo/cf-bosh

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

attachment.html

Re: [cf-bosh] Links to Nabble archives of the CF lists

Chip Childers <cchilders@...>

#39

Thanks Aaron!

Chip Childers | Technology Chief of Staff | Cloud Foundry Foundation

On Wed, May 6, 2015 at 4:28 PM, Huber, Aaron M <aaron.m.huber(a)intel.com>
wrote:

 I’ve created Nabble archives of the CF lists here:

http://cf-bosh.70367.x6.nabble.com/

http://cf-dev.70369.x6.nabble.com/

http://cf-lattice.70370.x6.nabble.com/

The archives are searchable and allow web viewing of the mailing list
without subscribing via email.  There is also an RSS feed for each list.

Aaron

_______________________________________________
cf-bosh mailing list
cf-bosh(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-bosh

attachment.html

Re: [cf-bosh] Links to Nabble archives of the CF lists

Dieu Cao <dcao@...>

#38

+1
Nice work Aaron!

On Wed, May 6, 2015 at 1:28 PM, Huber, Aaron M <aaron.m.huber(a)intel.com>
wrote:

 I’ve created Nabble archives of the CF lists here:

http://cf-bosh.70367.x6.nabble.com/

http://cf-dev.70369.x6.nabble.com/

http://cf-lattice.70370.x6.nabble.com/

The archives are searchable and allow web viewing of the mailing list
without subscribing via email.  There is also an RSS feed for each list.

Aaron

_______________________________________________
cf-bosh mailing list
cf-bosh(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-bosh

attachment.html

Links to Nabble archives of the CF lists

Aaron Huber

#37

I've created Nabble archives of the CF lists here:

http://cf-bosh.70367.x6.nabble.com/
http://cf-dev.70369.x6.nabble.com/
http://cf-lattice.70370.x6.nabble.com/

The archives are searchable and allow web viewing of the mailing list without subscribing via email.  There is also an RSS feed for each list.

Aaron

attachment.html

Re: Which components can be HA?

John Wong

#36

Hi Dieu

Thank you for the answers. They are very helpful.

Regarding #4, you are right, I believe when I do CF deployment I get these
short-lived VMs that compile different CF jobs.

Regarding #5, I think it is doppler in our latest deployment (v193, I know
still behind the most current version). I think in very old CF version
there used to be

loggreator_traffic
loggreator
syslog_loggreator

(as seen in the documentation mentions syslog loggreator).

So we probably don't need to worry about syslog then.


It seems like these are the one we can run >=2
NAT
DEA
UAA
HM9000
CC
Workers
Doppler
Log traffic controller
Gorouter
NFS (use s3 in our case)
Postgres (use RDS in our case)


These are the one not to run with > 1
collector
bosh
clock


Not sure:
stats server (metro agent?)


Thanks.

John

Show quoted text

On Wed, May 6, 2015 at 2:27 AM, Dieu Cao <dcao(a)pivotal.io> wrote:

1) I'll ask our doc team to clarify the title of the section.
It's not recommended to run more than 1 collector.  This component
collects metrics from system components.  We use it in combination with
Datadog to monitor the many components of cloud foundry.  This component is
not strictly required for an HA system.

2) HM9000 can have multiple active instances. No need for a standby mode.

3) The Cloud Controller clock periodically schedules Cloud Controller
clean up tasks for app usage events, audit events, failed jobs, and more.
Only single instance of this job is necessary.

4) Likely the job called api_workers is actually the cloud controller
workers. These are not compilation vms.
Cloud Controller worker processes background tasks submitted via clients
of the api.

5) I'm not sure what you mean by this. Do you mean loggregator? or doppler?

-Dieu
CF Runtime PM

On Tue, May 5, 2015 at 1:19 PM, John Wong <gokoproject(a)gmail.com> wrote:

Hi

Is http://docs.cloudfoundry.org/concepts/high-availability.html up to
date?

1) Why is collector listed 1 but in scalable process table?

2) How do you run a second Health Manager in standby mode if only 1 can
run at any time?

3) Do we still need clock job? Is it also 1 instance?

4) I notice I have a job called api_workers, and I believe that's
compilation machine. I run two of these 24x7, is that necessary? The doc
said it is active if we need to compile things (say deploying a new
release). Is that all? I don't think they handle application code
compilation.

5) What about syslog? Can it have 2? I understand we have to choose what
to be HA or not...I am not sure "the BOSH resurrector will recover the VM
if it becomes non-responsive" convinces me because all of these jobs are
deployed with BOSH but if BOSH is down I am facing some outage. I know Dr.
Nic has some article regarding HA bosh.


Correct me if I am wrong.

Thanks.

John

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

attachment.html

Utilities PMC - 2015-05-05 Notes

Mike Dalessio

#35

Hey everyone,

We held the first Utilities PMC meeting yesterday; I'd like to share the
agenda and notes.

For reference, all agendas notes for the Utilities PMC will be kept in a
public Google Drive folder at this URL:

http://bit.ly/cf-utilities-pmc


I realize GDrive isn't the most convenient medium for some in the CF
community; I'd love to hear how we can better support transparency for
everyone.

Please feel free to respond with comments and questions!

Cheers,
-m

---

*Attendees:*

   - Chip Childers, Cloud Foundry Foundation
   - Mike Dalessio, Pivotal (PMC lead)
   - Christopher Ferriss, IBM
   - Michael Fraenkel, IBM
   - James Bayer, Pivotal
   - Greg Oehmen, Pivotal
   - Ryan Morgan, PIvotal


Utilities PMC Agenda and Notes - 2015-05-05


   1.

   Update on CI tools (Mike Dalessio)
   2.

   Update on CLI (Greg Oehman)
   3.

   Update on Eclipse plugin and Java tools (Ryan Morgan)
   4.

   Open Discussion



Update on CI tools (Mike Dalessio)

GoCD <http://www.go.cd/> still in use for some projects, but there’s
movement towards Concourse <http://concourse.ci/> and teams are
enthusiastic about it. Currently Diego, Garden, BOSH-lite, Loggregator, and
CLI have converted to Concourse; and BOSH, Services API, and Buildpacks are
in progress.

Timeline is open for individual teams to move to Concourse; some teams may
decide not to. Having a heterogenous CI environment is OK, as both GoCD and
Concourse can integrate via S3 buckets, which is where generated artifacts
are generally kept.


Update on CLI (Greg Oehmen)


Released 6.11.0 - 4/17

Released 6.11.1 - 4/20

Released 6.11.2 - 4/28

Big uptick in issues/PRs

Plugin API feature

Look Ahead:

1. help refactor work,

- refactor help

- invert syntax (object - action)

- tab/bash completion

2.  support the move to cc API 3.0 and services api changes

3.  the user security work (pwd expiration, inactivity-based session
timeout, RBAC maturation, etc.)

4.   installer emphasis

- Auto-update within CLI

- signed mac installer

- signed windows installer

- etc

5.  APM integration - something like blessed-contrib:
https://github.com/yaronn/blessed-contrib


Update on Eclipse plugin and Java tools (Ryan Morgan)

CF Eclipse Tooling: (1 dev at Pivotal, 4 splitting time at IBM)

   -

   1.8.0 (Released Feb 13th)
   -

      New Service wizard allowing for multiple service creation
      -

      Remote debug support via ngrok.com
      -

   1.8.1 (Released March 25th)
   -

      Map/Unmap project feature to map an existing eclipse workspace to an
      app
      -

      Update password fixes
      -

      Free service plans now marked in the UI and preferred over paid plans
      -

   1.8.2 (Release imminent)
   -

      JRebel support
      -

      Working on some last minute UI changes
      -

   Working on a proposal to move the Eclipse tooling to the Eclipse
   Foundation
   -

      Should have a proposal for review mid-late May.  Targeting Eclipse
      4.5 SR1 update in the fall.  Lots of work to be done to make
that deadline.


CF Java Client: (1 dev at Pivotal, splitting time)

   -

   1.1.2 Released April 13th
   -

   No active development, PRs and Issues reviewed on-demand
   -

      Support of CC v3
      -

      Removal of Spring dependencies (v2.0 item)



Open Discussion

Please add any other suggested agenda topics for discussion here:

*Imminent additions to the Utilities PMC from HP (Chip).*

Voting took place via email on 2015-05-05 with unanimous consent to add the
following to the Utilities PMC as incubating projects:


   - CF .NET SDK https://github.com/hpcloud/cf-dotnet-sdk
   - CF Visual Studio Extension
   https://github.com/hpcloud/cf-vs-extension-wpf
   (will be renamed to https://github.com/hpcloud/cf-vs-extension)
   - CF MSBuild Tasks https://github.com/hpcloud/cf-msbuild-tasks

attachment.html

Removing Jackson 1 support from UAA

Filip Hanik

#34

The UAA team is currently upgrading Spring Framework dependencies in order
to stay up to date and be able to receive security and feature updates
faster.

With the story
https://www.pivotaltracker.com/story/show/93706906

the spring framework has dropped support for Jackson 1
(org.codehaus.jackson).
We have decided to follow the same path, rather than increasing our testing
matrix and try to be backwards compatible,.

With this upgrade, we instead will be supporting Jackson 2,
https://github.com/cloudfoundry/uaa/issues/108
directly from our classes.

While we attempt to be backwards compatible in most of our upgrade
scenarios, this one was instance we decided against it since Jackson 1 has
not had any software releases since 2013, and we most likely can't expect
any fixes against this library.

Feel free to ask any questions or raise any concerns. As a result, next UAA
release will be bumped to 2.3.x

Filip

attachment.html

Can't Create Service Instance in Cloud Foundry

Matthew Landry <mhlandry@...>

#33

I'm trying to get the introduction to spring cloud
<https://spring.io/blog/2014/06/03/introducing-spring-cloud> app working
with an instance of Cloud Foundry that I'm running on my machine. I tried
to push the app and I get this message:

        Could not find service postgres-service to bind to
hello-spring-cloud

That makes sense to me so I started tracking down the postgres service.
When I run `cf marketplace`, I get:

        service      plans     description
        mongodb      default   MongoDB NoSQL database
        postgresql   default   PostgreSQL database
        rabbitmq     default   RabbitMQ message queue
        redis        default   Redis key-value store

When I try to create a service instance of the postgresql service and I
get:

        $ cf create-service postgresql default postgresql-service
        Creating service instance postgresql-service in org xyz / space
development as admin...
        FAILED
        Server error, status code: 500, error code: 10001, message: Service
broker error: Not authorized

Here are the permissions for the space:

        Getting users in org xyz / space development as admin

        SPACE MANAGER
        admin

        SPACE DEVELOPER
        Admin

The cf service-access command doesn¹t yield anything interesting:

        $ cf service-access
        Getting service access as admin

Then it shows nothing on the terminal. So I tried to enable service access

        $ cf enable-service-access postgresql
        Enabling access to all plans of service postgresql for all orgs as
admin...
        All plans of the service are already accessible for all orgs
        OK

For the life of me, I can¹t figure out what¹s going on.  I posted a
question to Stack Overflow <
http://stackoverflow.com/questions/30034143/cant-create-service-instance-in-cloud-foundry>
and was referred here after some debugging
attempts were fruitless.  Anybody got any ideas?

attachment.html

Re: Which components can be HA?

Dieu Cao <dcao@...>

#32

1) I'll ask our doc team to clarify the title of the section.
It's not recommended to run more than 1 collector.  This component collects
metrics from system components.  We use it in combination with Datadog to
monitor the many components of cloud foundry.  This component is not
strictly required for an HA system.

2) HM9000 can have multiple active instances. No need for a standby mode.

3) The Cloud Controller clock periodically schedules Cloud Controller clean
up tasks for app usage events, audit events, failed jobs, and more. Only
single instance of this job is necessary.

4) Likely the job called api_workers is actually the cloud controller
workers. These are not compilation vms.
Cloud Controller worker processes background tasks submitted via clients of
the api.

5) I'm not sure what you mean by this. Do you mean loggregator? or doppler?

-Dieu
CF Runtime PM

Show quoted text

On Tue, May 5, 2015 at 1:19 PM, John Wong <gokoproject(a)gmail.com> wrote:

Hi

Is http://docs.cloudfoundry.org/concepts/high-availability.html up to
date?

1) Why is collector listed 1 but in scalable process table?

2) How do you run a second Health Manager in standby mode if only 1 can
run at any time?

3) Do we still need clock job? Is it also 1 instance?

4) I notice I have a job called api_workers, and I believe that's
compilation machine. I run two of these 24x7, is that necessary? The doc
said it is active if we need to compile things (say deploying a new
release). Is that all? I don't think they handle application code
compilation.

5) What about syslog? Can it have 2? I understand we have to choose what
to be HA or not...I am not sure "the BOSH resurrector will recover the VM
if it becomes non-responsive" convinces me because all of these jobs are
deployed with BOSH but if BOSH is down I am facing some outage. I know Dr.
Nic has some article regarding HA bosh.


Correct me if I am wrong.

Thanks.

John

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

attachment.html

Runtime PMC - 2015-05-04 Notes

Dieu Cao <dcao@...>

#31

Hi,

We held the first Runtime PMC bi-weekly meeting today.
Following on Mike D's lead, I'd like to share the notes from the meeting.
http://bit.ly/cf-runtime-pmc

-Dieu
CF Runtime PM

---

CF Runtime PMC

May 5, 2015

Attendees:

Chip Childers

Alex Tarpinian

Chris Sterling

Chris Ferris

Atul Kshirsagar

Michael Fraenkel

Steve Winkler

Mark Kropf

Zach Robinson

Erik Jasiak

Sree Tummidi

Dieu Cao

Eric Malm

Onsi Fakhouri

Proposal to move cfoundry to the cloudfoundy-attic

It’s not being maintained.

No objections raised during the PMC

Backlog reviews of:

Runtime

   -

   nearly complete with Route API and context path routing
   -

   Proposal to be sent out later this week for basic rate limiting for
   cloud controller to protect against users DOS’ing cloud controller
   -

   Work planned soon to move to Concourse for CI
   -

   Inception on Friday for rate limiting for CC and route services

UAA

   -

   wrapping up LDAP multi-tenancy support
   -

   starting on handling SAML attributes in tokens
   -

   password policies is up after SAML support

LAMB

   -

   Discovered dopplers were more cpu/thread-bound than thought
   -

      Needed to unwind some goals until doppler efficiency improved.
      -

      Fix is in progress and going well.
      -

   Up next #1 - Will bring back nozzles when pipeline can handle load
   -

   Up next #2 - Working with Runtime on etcd upgrade.
   -

   Up next #3 -

Diego

   -

   Completed 50-cell performance experiments and generated stories, will
   start 100-cell one soon
   -

   SSH access to containers progressing, currently setting up
   app/space/system access policies in CC and external routability
   -

   stories to come soon for versioning releases and schemas and improving
   inter-component security

Greenhouse

   -

   Tracking the Diego release well and hopes to have their first release
   coordinated with the first Diego release.
   - Greenhouse team working with Century Link and HP to improve container
   isolation.

attachment.html

stdout.log and stderr.log not show in CF197 with loggregator enabled

Tina Zhang

#30

Hi,

We upgrade from CF172 to CF197 and enable loggregator on CF197. But for application deployed to CF197 (with loggregator enabled), we DO NOT
see stdout.log and stderr.log anymore in application logs directory anymore. We can see logs/stdout.log and logs/stderr.log in CF172.

CF197:
cf file <app> logs
Getting file contents... OK

staging_task.log                          1.3K

Can you tell us what setting in CF 197 can affect stdout.log and stderr.log show up or not? How to let logs/stdout.log and logs/stderr.log show up?

Thanks,
Tina Zhang

attachment.html

Re: UAA with external openid connect

Mike Youngstrom

#29

Sounds good, thanks Sree!

Mike

Show quoted text

On Tue, May 5, 2015 at 5:00 PM, Sree Tummidi <stummidi(a)pivotal.io> wrote:

Hi Mike,
You would need to use SAML for this integration. UAA can act as an OpenID
Connect Identity provider but not Relying Party which is required in this
case.
We support SAML 2.0 Identity Providers.


Thanks,
Sree Tummidi
Sr. Product Manager
Identity - Pivotal Cloud Foundry


On Tue, May 5, 2015 at 3:52 PM, Mike Youngstrom <youngm(a)gmail.com> wrote:

We are looking at revamping the UAA integrations we are using for our
deployments.

Our enterprise authentication provider supports SAML and OpenID Connect
for single signon.  I know that UAA supports SAML.  Is it possible to use
our enterprise OpenID Connect provider to authenticate for UAA OAuth?  Or
should I just use SAML?

Mike

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

attachment.html