Date   

CF deployment environments available for CF incubating projects to use?

Jean-Sebastien Delfino
 

Hi all,

So far in the Abacus project we've been running our automated tests outside
CF (as our Node.js apps can also run outside with just a bit of env
variable config) on Travis-CI. Some of us also deploy our apps to Bosh lite
to test inside CF but maintaining working versions of Bosh lite is pretty
time consuming and that manual testing hasn't been a repeatable process so
far, so I'd really like to automate that with a proper CI build and test
environment.

Are there any CF deployment environments available for CF incubating
projects to use for CI builds and tests?

I'm looking for an environment where my build script could simply select a
specific version of CF, bootstrap it 'clean' (nothing left over from
previous runs), deploy the Abacus apps to it to run the tests there, then
repeat with a different version of CF etc.

Is anything like that already available for CF projects to use?

Thanks!

-- Jean-Sebastien


[abacus] Pruning our build matrix for faster Travis-CI builds

Jean-Sebastien Delfino
 

I've been seeing slower Travis-CI builds recently so I've pruned our build
matrix a bit in an attempt to speed things up.

Here are the combinations left in our Travis build matrix, down from 8 to 4:
matrix
include:
- os: "linux"
env: NODE_VERSION="4.0"
- os: "linux"
env: NODE_VERSION="0.12"
- os: "linux"
env: NODE_VERSION="0.10"
- os: "osx"
env: NODE_VERSION="4.0"

I've removed the iojs version from the build matrix as io.js has now
converged with node.js 4.0, and I'm also only building with node.js 4.0 on
OS X.

The OS X build is a bit slower than the Linux builds but I thought it'd be
important to keep at least one OS X build as most of us develop on OS X and
some of the logs we've seen in Github issues from our users show OS X as
well.

If the matrix is now missing a combination you really care about, please
post here and I'll be happy to add it back ...

-- Jean-Sebastien


Re: [abacus] Topics for Abacus docs?

Michael Maximilien
 

I like the idea of a `provider.md` and even maybe `on-boarding.md` that
will eventually point to our self-on-boarding process.

Best,

max

On Mon, Oct 5, 2015 at 3:09 PM, Jean-Sebastien Delfino <jsdelfino(a)gmail.com>
wrote:

I can continue keeping the FAQs into `faq.md`.
Thanks!

Finally, wonder if we should have a special section or file for service
brokers. That is all info needed by service brokers to submit usage, or is
this part of `integration.md`?

integration.md was more about how you'd customize Abacus to fit in your
particular Cloud platform / CF deployment,
config.md for service/runtime providers to know what to do to get their
(service/runtime) resource types on board Abacus,
api.md for how to use the api to submit usage and get usage reports.

Maybe we need a paragraph somewhere to say something like 'if you're a
service provider, first tell us what resources you want metered (and go to
config.md for that on boarding step), then once that's done use our
resource usage API to submit your usage (and that's in api.md).

We could also have a provider.md doc with 'everything you wanted to know
as a service provider'. That may be a better organization...
Not sure until we get more input from service providers or integrators on
what they're looking for.

-- Jean-Sebastien


On Mon, Oct 5, 2015 at 1:28 PM, Michael Maximilien <mmaximilien(a)gmail.com>
wrote:

+1

I can continue keeping the FAQs into `faq.md`.

Finally, wonder if we should have a special section or file for service
brokers. That is all info needed by service brokers to submit usage, or is
this part of `integration.md`?

Best,

max

On Mon, Oct 5, 2015 at 1:20 PM, Jean-Sebastien Delfino <
jsdelfino(a)gmail.com> wrote:

Hi all,

I've been reviewing the Abacus docs, and we currently only have the
following:
- a README describing how to get started with the code, build, run a
demo, set up for developing an Abacus module and the beginning of a FAQ;
- a doc of some of our REST API showing the URL, methods, return codes,
sample JSON bodies and JSON schemas.

The project is growing and I'm thinking that it's probably time to start
adding more and better documentation to help our users.

Here are a few topics I thought we should cover, and a straw man
proposal for how to structure these docs, for discussion:

README
how to get started, build, deploy and run a demo, setup your dev env to
contribute

then under docs/:

api.md
overview and reference doc for our REST API

faq.md
the FAQ that is currently in the README (as that README is getting
long), plus a few more recent questions

config.md
for service and runtime providers, overview of how to configure Abacus
for metering your own resource types, define your metrics, metering and
aggregation calculation functions etc

integration.md
for integrators of Abacus, describe how to customize the provisioning
and account/pricing services

app-usage.md
describe how we meter CF app usage and process it in Abacus

security.md
describe how to authenticate when calling Abacus APIs, and the CF Oauth
scopes we use for authorization

debug.md
tips on how to debug, trace, instrument the code

design.md
design notes, in particular describe the life of a usage doc as it flows
through Abacus and usage gets metered, accumulated, aggregated etc,
also, a description of what we store in our databases

I'm probably missing a number of important topics but does the structure
at least make sense?
What else would you guys like to see added -- or add yourself, pull
requests welcome -:) to these docs?

Thoughts?

-- Jean-Sebastien


--
max
http://maximilien.org
http://blog.maximilien.com

--
max
http://maximilien.org
http://blog.maximilien.com


Re: [abacus] Topics for Abacus docs?

Jean-Sebastien Delfino
 

I can continue keeping the FAQs into `faq.md`.
Thanks!

Finally, wonder if we should have a special section or file for service
brokers. That is all info needed by service brokers to submit usage, or is
this part of `integration.md`?

integration.md was more about how you'd customize Abacus to fit in your
particular Cloud platform / CF deployment,
config.md for service/runtime providers to know what to do to get their
(service/runtime) resource types on board Abacus,
api.md for how to use the api to submit usage and get usage reports.

Maybe we need a paragraph somewhere to say something like 'if you're a
service provider, first tell us what resources you want metered (and go to
config.md for that on boarding step), then once that's done use our
resource usage API to submit your usage (and that's in api.md).

We could also have a provider.md doc with 'everything you wanted to know as
a service provider'. That may be a better organization...
Not sure until we get more input from service providers or integrators on
what they're looking for.

-- Jean-Sebastien


On Mon, Oct 5, 2015 at 1:28 PM, Michael Maximilien <mmaximilien(a)gmail.com>
wrote:

+1

I can continue keeping the FAQs into `faq.md`.

Finally, wonder if we should have a special section or file for service
brokers. That is all info needed by service brokers to submit usage, or is
this part of `integration.md`?

Best,

max

On Mon, Oct 5, 2015 at 1:20 PM, Jean-Sebastien Delfino <
jsdelfino(a)gmail.com> wrote:

Hi all,

I've been reviewing the Abacus docs, and we currently only have the
following:
- a README describing how to get started with the code, build, run a
demo, set up for developing an Abacus module and the beginning of a FAQ;
- a doc of some of our REST API showing the URL, methods, return codes,
sample JSON bodies and JSON schemas.

The project is growing and I'm thinking that it's probably time to start
adding more and better documentation to help our users.

Here are a few topics I thought we should cover, and a straw man proposal
for how to structure these docs, for discussion:

README
how to get started, build, deploy and run a demo, setup your dev env to
contribute

then under docs/:

api.md
overview and reference doc for our REST API

faq.md
the FAQ that is currently in the README (as that README is getting long),
plus a few more recent questions

config.md
for service and runtime providers, overview of how to configure Abacus
for metering your own resource types, define your metrics, metering and
aggregation calculation functions etc

integration.md
for integrators of Abacus, describe how to customize the provisioning and
account/pricing services

app-usage.md
describe how we meter CF app usage and process it in Abacus

security.md
describe how to authenticate when calling Abacus APIs, and the CF Oauth
scopes we use for authorization

debug.md
tips on how to debug, trace, instrument the code

design.md
design notes, in particular describe the life of a usage doc as it flows
through Abacus and usage gets metered, accumulated, aggregated etc,
also, a description of what we store in our databases

I'm probably missing a number of important topics but does the structure
at least make sense?
What else would you guys like to see added -- or add yourself, pull
requests welcome -:) to these docs?

Thoughts?

-- Jean-Sebastien


--
max
http://maximilien.org
http://blog.maximilien.com


Re: cf push fails with "Unauthorized error: You are not authorized. Error: Invalid authorization"

CF Runtime
 

Are you using an aws elb or a ha_proxy instance for your load balancer?

If you curl api.SYSTEM_DOMAIN/v2/info, what is the logging_endpoint from
the response?

Joseph
CF Release Integration Team



On Mon, Oct 5, 2015 at 3:36 AM, remi clotaire tassing tagne <
tassingremi(a)gmail.com> wrote:

Hi,

So I've finally managed to deploy CF on AWS but haven't managed pushing
the most rudimentary node.js app. BTW, pushing this app to my local CF
instance worked.

I tried troubleshooting with "cf logs" and "cf events" and "CF_TRACE=true"
but I just can't figure out the root cause. I've even created an org and
space and a user with SpaceDeveloper role but without luck. 'cf push' gives
me:

cf push
Using manifest file /home/remi/workspace/apps/hello-nodejs/manifest.yml

Creating app hello-nodejs in org test / space test as cf...
OK

Creating route hello-nodejs.ip.address.xip.io...
OK

Binding hello-nodejs.ip.address.xip.io to hello-nodejs...
OK

Uploading hello-nodejs...
Uploading app files from: /home/remi/workspace/apps/hello-nodejs
Uploading 1K, 3 files
Done uploading
OK

timeout connecting to log server, no log will be shown
Starting app hello-nodejs in org test / space test as cf...
Warning: error tailing logs
Unauthorized error: You are not authorized. Error: Invalid authorization

FAILED
hello-nodejs failed to stage within 15.000000 minutes

Any idea where I should look for the problem?

Thanks in advance!

Remi


Re: `cf app` output

Daniel Mikusa
 

Thanks for those links. I'm using the DEA for now and it looks like
nothing has changed there for quite some time (Apr 2014). I'll see if I
can debug a bit more to see what's happening.

Thanks,

Dan

On Mon, Oct 5, 2015 at 4:36 PM, Matthew Sykes <matthew.sykes(a)gmail.com>
wrote:

In the DEA:
https://github.com/cloudfoundry/dea_ng/blob/4b1a50ae5598b0c70cb3e5895ed800e0cff37722/lib/dea/stat_collector.rb#L93

In Diego/Garden:

https://github.com/cloudfoundry-incubator/garden-linux/blob/master/linux_container/metrics.go#L129

If you're using the DEA, stats come back in response to the `find.droplet`
message while in Diego they get updated on an interval.

On Mon, Oct 5, 2015 at 4:20 PM, Daniel Mikusa <dmikusa(a)pivotal.io> wrote:

Has something changed recently with the output of `cf app` and how it
reports memory usage?

I was deploying apps to CF v219 and noticed that the output of `cf app`
is not what I'm used to seeing.

I'm deploying a Java app with a 512M memory limit. The Java build pack
is setting the initial heap and metaspace sizes to be 373M and 64M
(-XX:MaxMetaspaceSize=64M -Xss995K -Xmx382293K -Xms382293K
-XX:MetaspaceSize=64M), which is a total of 437M. Since that's just the
heap and metaspace, I'd expect the app to start out using at least that
much memory.

The report from `cf app` is showing 366.3M of 512M.

Does anyone know how this is presently being calculated / have a link to
the source code where this is being calculated?

Thanks,

Dan


--
Matthew Sykes
matthew.sykes(a)gmail.com


Re: `cf app` output

Matthew Sykes <matthew.sykes@...>
 

In the DEA:
https://github.com/cloudfoundry/dea_ng/blob/4b1a50ae5598b0c70cb3e5895ed800e0cff37722/lib/dea/stat_collector.rb#L93

In Diego/Garden:
https://github.com/cloudfoundry-incubator/garden-linux/blob/master/linux_container/metrics.go#L129

If you're using the DEA, stats come back in response to the `find.droplet`
message while in Diego they get updated on an interval.

On Mon, Oct 5, 2015 at 4:20 PM, Daniel Mikusa <dmikusa(a)pivotal.io> wrote:

Has something changed recently with the output of `cf app` and how it
reports memory usage?

I was deploying apps to CF v219 and noticed that the output of `cf app` is
not what I'm used to seeing.

I'm deploying a Java app with a 512M memory limit. The Java build pack is
setting the initial heap and metaspace sizes to be 373M and 64M
(-XX:MaxMetaspaceSize=64M -Xss995K -Xmx382293K -Xms382293K
-XX:MetaspaceSize=64M), which is a total of 437M. Since that's just the
heap and metaspace, I'd expect the app to start out using at least that
much memory.

The report from `cf app` is showing 366.3M of 512M.

Does anyone know how this is presently being calculated / have a link to
the source code where this is being calculated?

Thanks,

Dan
--
Matthew Sykes
matthew.sykes(a)gmail.com


Re: [abacus] Topics for Abacus docs?

Michael Maximilien
 

+1

I can continue keeping the FAQs into `faq.md`.

Finally, wonder if we should have a special section or file for service
brokers. That is all info needed by service brokers to submit usage, or is
this part of `integration.md`?

Best,

max

On Mon, Oct 5, 2015 at 1:20 PM, Jean-Sebastien Delfino <jsdelfino(a)gmail.com>
wrote:

Hi all,

I've been reviewing the Abacus docs, and we currently only have the
following:
- a README describing how to get started with the code, build, run a demo,
set up for developing an Abacus module and the beginning of a FAQ;
- a doc of some of our REST API showing the URL, methods, return codes,
sample JSON bodies and JSON schemas.

The project is growing and I'm thinking that it's probably time to start
adding more and better documentation to help our users.

Here are a few topics I thought we should cover, and a straw man proposal
for how to structure these docs, for discussion:

README
how to get started, build, deploy and run a demo, setup your dev env to
contribute

then under docs/:

api.md
overview and reference doc for our REST API

faq.md
the FAQ that is currently in the README (as that README is getting long),
plus a few more recent questions

config.md
for service and runtime providers, overview of how to configure Abacus for
metering your own resource types, define your metrics, metering and
aggregation calculation functions etc

integration.md
for integrators of Abacus, describe how to customize the provisioning and
account/pricing services

app-usage.md
describe how we meter CF app usage and process it in Abacus

security.md
describe how to authenticate when calling Abacus APIs, and the CF Oauth
scopes we use for authorization

debug.md
tips on how to debug, trace, instrument the code

design.md
design notes, in particular describe the life of a usage doc as it flows
through Abacus and usage gets metered, accumulated, aggregated etc,
also, a description of what we store in our databases

I'm probably missing a number of important topics but does the structure
at least make sense?
What else would you guys like to see added -- or add yourself, pull
requests welcome -:) to these docs?

Thoughts?

-- Jean-Sebastien


--
max
http://maximilien.org
http://blog.maximilien.com


[abacus] Topics for Abacus docs?

Jean-Sebastien Delfino
 

Hi all,

I've been reviewing the Abacus docs, and we currently only have the
following:
- a README describing how to get started with the code, build, run a demo,
set up for developing an Abacus module and the beginning of a FAQ;
- a doc of some of our REST API showing the URL, methods, return codes,
sample JSON bodies and JSON schemas.

The project is growing and I'm thinking that it's probably time to start
adding more and better documentation to help our users.

Here are a few topics I thought we should cover, and a straw man proposal
for how to structure these docs, for discussion:

README
how to get started, build, deploy and run a demo, setup your dev env to
contribute

then under docs/:

api.md
overview and reference doc for our REST API

faq.md
the FAQ that is currently in the README (as that README is getting long),
plus a few more recent questions

config.md
for service and runtime providers, overview of how to configure Abacus for
metering your own resource types, define your metrics, metering and
aggregation calculation functions etc

integration.md
for integrators of Abacus, describe how to customize the provisioning and
account/pricing services

app-usage.md
describe how we meter CF app usage and process it in Abacus

security.md
describe how to authenticate when calling Abacus APIs, and the CF Oauth
scopes we use for authorization

debug.md
tips on how to debug, trace, instrument the code

design.md
design notes, in particular describe the life of a usage doc as it flows
through Abacus and usage gets metered, accumulated, aggregated etc,
also, a description of what we store in our databases

I'm probably missing a number of important topics but does the structure at
least make sense?
What else would you guys like to see added -- or add yourself, pull
requests welcome -:) to these docs?

Thoughts?

-- Jean-Sebastien


`cf app` output

Daniel Mikusa
 

Has something changed recently with the output of `cf app` and how it
reports memory usage?

I was deploying apps to CF v219 and noticed that the output of `cf app` is
not what I'm used to seeing.

I'm deploying a Java app with a 512M memory limit. The Java build pack is
setting the initial heap and metaspace sizes to be 373M and 64M
(-XX:MaxMetaspaceSize=64M -Xss995K -Xmx382293K -Xms382293K
-XX:MetaspaceSize=64M), which is a total of 437M. Since that's just the
heap and metaspace, I'd expect the app to start out using at least that
much memory.

The report from `cf app` is showing 366.3M of 512M.

Does anyone know how this is presently being calculated / have a link to
the source code where this is being calculated?

Thanks,

Dan


Re: [Proposal] Wanted a Babysitter for my applicatoin. ; -)

Amit Kumar Gupta
 

I'm not sure I see the benefit here.

Diego, for instance, runs a customizable babysitter alongside each app
instance, and kills the container if the babysitter says things are going
bad. This triggers an event that the system can react to, and the system
also polls for container states because events can always be lost.

One thing to note is in this case, "the system" is the Executor, not HM9k
(which doesn't exist in Diego), or the Converger (Diego's equivalent of
HM9k), or Firehose or Cloud Controller which are very far removed from the
container backend. In Diego, the pieces are loosely coupled, events/data
in the system don't have to be sent through several layers of abstraction.

Best,
Amit

On Mon, Oct 5, 2015 at 10:09 AM, Curry, Matthew <Matt.Curry(a)allstate.com>
wrote:

We have been talking about something similar that we have labeled the
Angry Farmer. I do not think you would need an agent. The firehose and
cloud controller should have everything that you need. Also an agent does
not give you the ability to really measure the performance of instances
relative to each other which is a good indicator of bad state or
performance.

Matt

From: Dhilip Kumar S <dhilip.kumar.s(a)huawei.com>
Reply-To: "Discussions about Cloud Foundry projects and the system
overall." <cf-dev(a)lists.cloudfoundry.org>
Date: Monday, October 5, 2015 at 9:31 AM
To: "Discussions about Cloud Foundry projects and the system overall." <
cf-dev(a)lists.cloudfoundry.org>
Cc: Vinay Murudi <vinaym(a)huawei.com>, Krishna M Kumar <
krishna.m.kumar(a)huawei.com>, Liangbiao <rexxar.liang(a)huawei.com>,
Srinivasch ch <srinivasch.ch(a)huawei.com>
Subject: [cf-dev] [Proposal] Wanted a Babysitter for my applicatoin. ;-)

Hello CF,



Greetings from Huawei. Here is a quick idea that came up to our mind
recently. Honestly we did not spend enormous time brainstorming this
internally, but we thought we could go ahead and ask the community
directly. It would be a great help to know if such an idea had already been
considered and dropped by the community.



*Proposal Motivations*

The way health-check process is currently performed in cloud foundry is to
run a command
<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_cloudfoundry-2Dincubator_healthcheck&d=BQMGaQ&c=gtIjdLs6LnStUpy9cTOW9w&r=5uKsnIXwfJIxHSaCSaJzvcn90bBlYQuxsJhof4ERK-Q&m=8v-kDNCf3N_TGthtUze_YzZR4BADnwPZ9BiNHtzQnF4&s=MkllX3Km4FRjbvpC1QE02cQWP_QcCOE2qDv-UQCgytk&e=>
periodically; if the exit status is non-zero then it is assumed that an
application is non-responsive. We periodically repeat this process for all
the applications. Which means that we actually scan the entire data center
frequently to find one or few miss-behaving apps?



Why can’t we change the way health-check is done? Can it reflect the
real-world? The hospitals don’t periodically scan the entire community
looking for sick residents. Similarly, why can’t we report problems as and
when they occur – just like the real-world?



How about a lightweight process that constantly monitors the application’s
health and periodically reports in case an app is down or non-responsive
etc. In a huge datacenter where thousands of apps are hosted, and each app
has many instances. Wouldn’t it be better to make the individual
app/container come and tell us(healthmanager) that there is a problem
instead of scanning all of them? *Push versus Pull model* - Something
like a babysitter residing within each container and taking care of the
‘app’ hosted by our customers.



*How to accomplish this?*

Our proposal is for BabySitter(BS) – an agent residing within each
container optionally deployed using app-specific configuration. This agent
sends out the collected metrics to health monitor in case of any anomaly –
periodic time-series information etc. The agent should remember the
configured threshold value that each app should not exceed; otherwise it
triggers an alarm automatically to the health monitor in case of any
threshold violations. The alarm even could be sent many times a second to
the healthmonitor depending on the severity of the event, but the regular
periodic ‘time-series’ information could be collected every second but sent
once a minute to the HM. The challenge is design the application ‘bs’ as
lightweight as possible.



This is our primary idea, we also thought it would make more sense if we
club few more capabilities to babysitter like sshd (as a goroutine) and
fileserver(as a goroutine) but before we bore you with all that details, we
first want to understand what CF community thinks about this initial idea.



Thanks in advance,

Dhilip




Re: [Proposal] Wanted a Babysitter for my applicatoin. ;-)

Niki Dokovski <nickytd@...>
 

Hi

Just an example to the case, a short description of a similar technique can be found at JavaSpaces (leveraging Java JINI)[1] .
The notion of a “lease” as a foundation of a self-healing systems architecture has been used successfully in the past.

Best Regards
Niki Dokovski | @nickytd <https://twitter.com/nickytd>

[1] https://en.wikibooks.org/wiki/Java_Programming/JavaSpaces <https://en.wikibooks.org/wiki/Java_Programming/JavaSpaces>

On 5.10.2015 г., at 19:31, Dhilip Kumar S <dhilip.kumar.s(a)huawei.com> wrote:

Hello CF,

Greetings from Huawei. Here is a quick idea that came up to our mind recently. Honestly we did not spend enormous time brainstorming this internally, but we thought we could go ahead and ask the community directly. It would be a great help to know if such an idea had already been considered and dropped by the community.

Proposal Motivations
The way health-check process is currently performed in cloud foundry is to run a command <https://github.com/cloudfoundry-incubator/healthcheck> periodically; if the exit status is non-zero then it is assumed that an application is non-responsive. We periodically repeat this process for all the applications. Which means that we actually scan the entire data center frequently to find one or few miss-behaving apps?

Why can’t we change the way health-check is done? Can it reflect the real-world? The hospitals don’t periodically scan the entire community looking for sick residents. Similarly, why can’t we report problems as and when they occur – just like the real-world?

How about a lightweight process that constantly monitors the application’s health and periodically reports in case an app is down or non-responsive etc. In a huge datacenter where thousands of apps are hosted, and each app has many instances. Wouldn’t it be better to make the individual app/container come and tell us(healthmanager) that there is a problem instead of scanning all of them? Push versus Pull model - Something like a babysitter residing within each container and taking care of the ‘app’ hosted by our customers.

How to accomplish this?
Our proposal is for BabySitter(BS) – an agent residing within each container optionally deployed using app-specific configuration. This agent sends out the collected metrics to health monitor in case of any anomaly – periodic time-series information etc. The agent should remember the configured threshold value that each app should not exceed; otherwise it triggers an alarm automatically to the health monitor in case of any threshold violations. The alarm even could be sent many times a second to the healthmonitor depending on the severity of the event, but the regular periodic ‘time-series’ information could be collected every second but sent once a minute to the HM. The challenge is design the application ‘bs’ as lightweight as possible.

This is our primary idea, we also thought it would make more sense if we club few more capabilities to babysitter like sshd (as a goroutine) and fileserver(as a goroutine) but before we bore you with all that details, we first want to understand what CF community thinks about this initial idea.

Thanks in advance,
Dhilip


Re: [Proposal] Wanted a Babysitter for my applicatoin. ;-)

Matt Curry
 

We have been talking about something similar that we have labeled the Angry Farmer. I do not think you would need an agent. The firehose and cloud controller should have everything that you need. Also an agent does not give you the ability to really measure the performance of instances relative to each other which is a good indicator of bad state or performance.

Matt

From: Dhilip Kumar S <dhilip.kumar.s(a)huawei.com<mailto:dhilip.kumar.s(a)huawei.com>>
Reply-To: "Discussions about Cloud Foundry projects and the system overall." <cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org>>
Date: Monday, October 5, 2015 at 9:31 AM
To: "Discussions about Cloud Foundry projects and the system overall." <cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org>>
Cc: Vinay Murudi <vinaym(a)huawei.com<mailto:vinaym(a)huawei.com>>, Krishna M Kumar <krishna.m.kumar(a)huawei.com<mailto:krishna.m.kumar(a)huawei.com>>, Liangbiao <rexxar.liang(a)huawei.com<mailto:rexxar.liang(a)huawei.com>>, Srinivasch ch <srinivasch.ch(a)huawei.com<mailto:srinivasch.ch(a)huawei.com>>
Subject: [cf-dev] [Proposal] Wanted a Babysitter for my applicatoin. ;-)

Hello CF,

Greetings from Huawei. Here is a quick idea that came up to our mind recently. Honestly we did not spend enormous time brainstorming this internally, but we thought we could go ahead and ask the community directly. It would be a great help to know if such an idea had already been considered and dropped by the community.

Proposal Motivations

The way health-check process is currently performed in cloud foundry is to run a command<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_cloudfoundry-2Dincubator_healthcheck&d=BQMGaQ&c=gtIjdLs6LnStUpy9cTOW9w&r=5uKsnIXwfJIxHSaCSaJzvcn90bBlYQuxsJhof4ERK-Q&m=8v-kDNCf3N_TGthtUze_YzZR4BADnwPZ9BiNHtzQnF4&s=MkllX3Km4FRjbvpC1QE02cQWP_QcCOE2qDv-UQCgytk&e=> periodically; if the exit status is non-zero then it is assumed that an application is non-responsive. We periodically repeat this process for all the applications. Which means that we actually scan the entire data center frequently to find one or few miss-behaving apps?



Why can’t we change the way health-check is done? Can it reflect the real-world? The hospitals don’t periodically scan the entire community looking for sick residents. Similarly, why can’t we report problems as and when they occur – just like the real-world?



How about a lightweight process that constantly monitors the application’s health and periodically reports in case an app is down or non-responsive etc. In a huge datacenter where thousands of apps are hosted, and each app has many instances. Wouldn’t it be better to make the individual app/container come and tell us(healthmanager) that there is a problem instead of scanning all of them? Push versus Pull model - Something like a babysitter residing within each container and taking care of the ‘app’ hosted by our customers.


How to accomplish this?
Our proposal is for BabySitter(BS) – an agent residing within each container optionally deployed using app-specific configuration. This agent sends out the collected metrics to health monitor in case of any anomaly – periodic time-series information etc. The agent should remember the configured threshold value that each app should not exceed; otherwise it triggers an alarm automatically to the health monitor in case of any threshold violations. The alarm even could be sent many times a second to the healthmonitor depending on the severity of the event, but the regular periodic ‘time-series’ information could be collected every second but sent once a minute to the HM. The challenge is design the application ‘bs’ as lightweight as possible.

This is our primary idea, we also thought it would make more sense if we club few more capabilities to babysitter like sshd (as a goroutine) and fileserver(as a goroutine) but before we bore you with all that details, we first want to understand what CF community thinks about this initial idea.

Thanks in advance,
Dhilip


[Proposal] Wanted a Babysitter for my applicatoin. ;-)

Dhilip
 

Hello CF,

Greetings from Huawei. Here is a quick idea that came up to our mind recently. Honestly we did not spend enormous time brainstorming this internally, but we thought we could go ahead and ask the community directly. It would be a great help to know if such an idea had already been considered and dropped by the community.

Proposal Motivations

The way health-check process is currently performed in cloud foundry is to run a command<https://github.com/cloudfoundry-incubator/healthcheck> periodically; if the exit status is non-zero then it is assumed that an application is non-responsive. We periodically repeat this process for all the applications. Which means that we actually scan the entire data center frequently to find one or few miss-behaving apps?



Why can’t we change the way health-check is done? Can it reflect the real-world? The hospitals don’t periodically scan the entire community looking for sick residents. Similarly, why can’t we report problems as and when they occur – just like the real-world?



How about a lightweight process that constantly monitors the application’s health and periodically reports in case an app is down or non-responsive etc. In a huge datacenter where thousands of apps are hosted, and each app has many instances. Wouldn’t it be better to make the individual app/container come and tell us(healthmanager) that there is a problem instead of scanning all of them? Push versus Pull model - Something like a babysitter residing within each container and taking care of the ‘app’ hosted by our customers.


How to accomplish this?
Our proposal is for BabySitter(BS) – an agent residing within each container optionally deployed using app-specific configuration. This agent sends out the collected metrics to health monitor in case of any anomaly – periodic time-series information etc. The agent should remember the configured threshold value that each app should not exceed; otherwise it triggers an alarm automatically to the health monitor in case of any threshold violations. The alarm even could be sent many times a second to the healthmonitor depending on the severity of the event, but the regular periodic ‘time-series’ information could be collected every second but sent once a minute to the HM. The challenge is design the application ‘bs’ as lightweight as possible.

This is our primary idea, we also thought it would make more sense if we club few more capabilities to babysitter like sshd (as a goroutine) and fileserver(as a goroutine) but before we bore you with all that details, we first want to understand what CF community thinks about this initial idea.

Thanks in advance,
Dhilip


Re: Just got CF deployed - What is a good first application tutorial?

admin@...
 

Perfect. Beginner is what I am!

JB

On Oct 5, 2015, at 10:27 AM, Dave Nielsen <dnielsen(a)gmail.com> wrote:

By the way, my "Deploy Your First Cloud Foundry App" talk from CFSummit is up on Youtube. It's for beginners, but it might be helpful. You can download the presentation and quickstart guide from here: http://cloudworkshop.org/cloudfoundry

Best,
Dave

Dave Nielsen
Organizer: SVCloudFoundry
Developer Relations, Redis Labs
twitter davenielsen; linkedin dnielsen; fb: dcnielsen
skype: davenielsen; gtalk: dnielsen; mobile: +1-415-531-6674


On Mon, Oct 5, 2015 at 6:33 AM, Dave Nielsen <dnielsen(a)gmail.com> wrote:
Hi Joseph,

You can add an entry into the manifest.yml file to specify the specific build pack you want to use. Here's an example of a manifest.yml that specifies a buildpack located in the Cloud Foundry github repo.

applications:
- name: cf-node-contactform
host: cf-node-contactform-dcn6
memory: 128M
command: node server.js
buildpack: https://github.com/cloudfoundry/nodejs-buildpack.git

Best,
Dave

Dave Nielsen
Organizer: SVCloudFoundry
Developer Relations, Redis Labs
twitter davenielsen; linkedin dnielsen; fb: dcnielsen
skype: davenielsen; gtalk: dnielsen; mobile: +1-415-531-6674


On Mon, Oct 5, 2015 at 4:01 AM, Joseph Biron <admin(a)josephbiron.com> wrote:
Hey that explicit build pack worked great!

As a newb, could you tell me where I would change the config for the app to use this build pack rather than they original?



On Oct 4, 2015, at 7:47 PM, Max Hufnagel <mhufnagel(a)pivotal.io> wrote:

Hello!

It looks to me like the example application in https://docs.cloudfoundry.org/deploying/ec2/example_apps.html was updated 7 months ago and doesn't recognize the new buildpack. Maybe if you explicitly give it an old buildpack in your push command?

cf push your-app-name -b https://github.com/cloudfoundry/ruby-buildpack.git#v1.3.0



On Sun, Oct 4, 2015 at 2:36 PM, Joseph Biron <admin(a)josephbiron.com> wrote:
Thanks, but

-----> Compiling Ruby/Rack
Could not get translated url, exited with: DEPENDENCY_MISSING_IN_MANIFEST: https://s3-external-1.amazonaws.com/heroku-buildpack-ruby/ruby-2.1.5.tgz


When I try the s3 URL in a browser I see

<Error>
<Code>AccessDenied</Code>
<Message>Access Denied</Message>
<RequestId>0A8DA74BC355E61E</RequestId>
<HostId>
OAHOWhKzXcaZmIwMFTYXJmaLvzvyC5HxNRORtzEPvOZ/TFjWMvr8foQTqSuPYPxp
</HostId>
</Error>


On Oct 4, 2015, at 3:55 PM, remi clotaire tassing tagne <tassingremi(a)gmail.com> wrote:

Example app deployed on CF: https://docs.cloudfoundry.org/deploying/ec2/example_apps.html


Re: Just got CF deployed - What is a good first application tutorial?

Dave Nielsen <dnielsen@...>
 

By the way, my "Deploy Your First Cloud Foundry App" talk
<https://www.youtube.com/watch?v=RC1HX_7aGa8&index=5&list=PLhuMOCWn4P9g-UMN5nzDiw78zgf5rJ4gR>
from CFSummit is up on Youtube. It's for beginners, but it might be
helpful. You can download the presentation and quickstart guide from here:
http://cloudworkshop.org/cloudfoundry

Best,
Dave

Dave Nielsen
Organizer: SVCloudFoundry <http://meetup.com/cloudfoundry>
Developer Relations, Redis Labs
twitter davenielsen <http://twitter.com/davenielsen>; linkedin dnielsen
<http://linkedin.com/in/dnielsen>; fb: dcnielsen <http://fb.com/dcnielsen>
skype: davenielsen; gtalk: dnielsen; mobile: +1-415-531-6674

On Mon, Oct 5, 2015 at 6:33 AM, Dave Nielsen <dnielsen(a)gmail.com> wrote:

Hi Joseph,

You can add an entry into the manifest.yml file to specify the specific
build pack you want to use. Here's an example of a manifest.yml that
specifies a buildpack located in the Cloud Foundry github repo.

applications:
- name: cf-node-contactform
host: cf-node-contactform-dcn6
memory: 128M
command: node server.js
buildpack: *https://github.com/cloudfoundry/nodejs-buildpack.git
<https://github.com/cloudfoundry/nodejs-buildpack.git>*

Best,
Dave

Dave Nielsen
Organizer: SVCloudFoundry <http://meetup.com/cloudfoundry>
Developer Relations, Redis Labs
twitter davenielsen <http://twitter.com/davenielsen>; linkedin dnielsen
<http://linkedin.com/in/dnielsen>; fb: dcnielsen <http://fb.com/dcnielsen>
skype: davenielsen; gtalk: dnielsen; mobile: +1-415-531-6674


On Mon, Oct 5, 2015 at 4:01 AM, Joseph Biron <admin(a)josephbiron.com>
wrote:

Hey that explicit build pack worked great!

As a newb, could you tell me where I would change the config for the app
to use this build pack rather than they original?



On Oct 4, 2015, at 7:47 PM, Max Hufnagel <mhufnagel(a)pivotal.io> wrote:

Hello!

It looks to me like the example application in
https://docs.cloudfoundry.org/deploying/ec2/example_apps.html was
updated 7 months ago and doesn't recognize the new buildpack. Maybe if you
explicitly give it an old buildpack in your push command?

cf push your-app-name -b
https://github.com/cloudfoundry/ruby-buildpack.git#v1.3.0



On Sun, Oct 4, 2015 at 2:36 PM, Joseph Biron <admin(a)josephbiron.com>
wrote:

Thanks, but

-----> Compiling Ruby/Rack
Could not get translated url, exited with:
DEPENDENCY_MISSING_IN_MANIFEST:
https://s3-external-1.amazonaws.com/heroku-buildpack-ruby/ruby-2.1.5.tgz


When I try the s3 URL in a browser I see

<Error>
<Code>AccessDenied</Code>
<Message>Access Denied</Message>
<RequestId>0A8DA74BC355E61E</RequestId>
<HostId>
OAHOWhKzXcaZmIwMFTYXJmaLvzvyC5HxNRORtzEPvOZ/TFjWMvr8foQTqSuPYPxp
</HostId>
</Error>


On Oct 4, 2015, at 3:55 PM, remi clotaire tassing tagne <
tassingremi(a)gmail.com> wrote:

Example app deployed on CF:
https://docs.cloudfoundry.org/deploying/ec2/example_apps.html



Re: Just got CF deployed - What is a good first application tutorial?

Dave Nielsen <dnielsen@...>
 

Hi Joseph,

You can add an entry into the manifest.yml file to specify the specific
build pack you want to use. Here's an example of a manifest.yml that
specifies a buildpack located in the Cloud Foundry github repo.

applications:
- name: cf-node-contactform
host: cf-node-contactform-dcn6
memory: 128M
command: node server.js
buildpack: *https://github.com/cloudfoundry/nodejs-buildpack.git
<https://github.com/cloudfoundry/nodejs-buildpack.git>*

Best,
Dave

Dave Nielsen
Organizer: SVCloudFoundry <http://meetup.com/cloudfoundry>
Developer Relations, Redis Labs
twitter davenielsen <http://twitter.com/davenielsen>; linkedin dnielsen
<http://linkedin.com/in/dnielsen>; fb: dcnielsen <http://fb.com/dcnielsen>
skype: davenielsen; gtalk: dnielsen; mobile: +1-415-531-6674

On Mon, Oct 5, 2015 at 4:01 AM, Joseph Biron <admin(a)josephbiron.com> wrote:

Hey that explicit build pack worked great!

As a newb, could you tell me where I would change the config for the app
to use this build pack rather than they original?



On Oct 4, 2015, at 7:47 PM, Max Hufnagel <mhufnagel(a)pivotal.io> wrote:

Hello!

It looks to me like the example application in
https://docs.cloudfoundry.org/deploying/ec2/example_apps.html was updated
7 months ago and doesn't recognize the new buildpack. Maybe if you
explicitly give it an old buildpack in your push command?

cf push your-app-name -b
https://github.com/cloudfoundry/ruby-buildpack.git#v1.3.0



On Sun, Oct 4, 2015 at 2:36 PM, Joseph Biron <admin(a)josephbiron.com>
wrote:

Thanks, but

-----> Compiling Ruby/Rack
Could not get translated url, exited with:
DEPENDENCY_MISSING_IN_MANIFEST:
https://s3-external-1.amazonaws.com/heroku-buildpack-ruby/ruby-2.1.5.tgz


When I try the s3 URL in a browser I see

<Error>
<Code>AccessDenied</Code>
<Message>Access Denied</Message>
<RequestId>0A8DA74BC355E61E</RequestId>
<HostId>
OAHOWhKzXcaZmIwMFTYXJmaLvzvyC5HxNRORtzEPvOZ/TFjWMvr8foQTqSuPYPxp
</HostId>
</Error>


On Oct 4, 2015, at 3:55 PM, remi clotaire tassing tagne <
tassingremi(a)gmail.com> wrote:

Example app deployed on CF:
https://docs.cloudfoundry.org/deploying/ec2/example_apps.html



Re: Just got CF deployed - What is a good first application tutorial?

Stormy
 

Hi Joseph,

If you do create anything as you go through your first steps, we'd love to
have it. We spend last Wednesday in a "doc sprint" working on the getting
started guides. The work is not all checked in yet. We are in the
cloudfoundy.slack.com, #cf-docs.

Stormy

On Mon, Oct 5, 2015 at 4:32 AM, <admin(a)josephbiron.com> wrote:

Thanks for the links, Max.

I really wanted a tutorial that is step by step, but I can work with
these.

I think we need something like that to engage more developers with
cloudfoundry.

I'm most concerned that as I try to "sell" this internally I'll get
resistance due to poor documentation. Regardless of how good reference
material is, the lack of starter tutorials is perceived as "weak
community".

Perhaps I can put something together as I'm going through my first steps.

Is this something that the Pivotal offering does very well? I would think
so.

JB

On Oct 4, 2015, at 7:50 PM, Max Hufnagel <mhufnagel(a)pivotal.io> wrote:

Also, you might want to check out some of the cool apps in the Cloud
Foundry Sample Applications repo:
repo: https://github.com/cloudfoundry-samples

On Sun, Oct 4, 2015 at 4:47 PM, Max Hufnagel <mhufnagel(a)pivotal.io> wrote:

Hello!

It looks to me like the example application in
https://docs.cloudfoundry.org/deploying/ec2/example_apps.html was
updated 7 months ago and doesn't recognize the new buildpack. Maybe if you
explicitly give it an old buildpack in your push command?

cf push your-app-name -b
https://github.com/cloudfoundry/ruby-buildpack.git#v1.3.0



On Sun, Oct 4, 2015 at 2:36 PM, Joseph Biron <admin(a)josephbiron.com>
wrote:

Thanks, but

-----> Compiling Ruby/Rack
Could not get translated url, exited with:
DEPENDENCY_MISSING_IN_MANIFEST:
https://s3-external-1.amazonaws.com/heroku-buildpack-ruby/ruby-2.1.5.tgz


When I try the s3 URL in a browser I see

<Error>
<Code>AccessDenied</Code>
<Message>Access Denied</Message>
<RequestId>0A8DA74BC355E61E</RequestId>
<HostId>
OAHOWhKzXcaZmIwMFTYXJmaLvzvyC5HxNRORtzEPvOZ/TFjWMvr8foQTqSuPYPxp
</HostId>
</Error>


On Oct 4, 2015, at 3:55 PM, remi clotaire tassing tagne <
tassingremi(a)gmail.com> wrote:

Example app deployed on CF:
https://docs.cloudfoundry.org/deploying/ec2/example_apps.html



Unable to deliver your item, #00652961

FedEx International Ground <warren.casey@...>
 

Dear Customer,

We could not deliver your item.
Shipment Label is attached to this email.

Thank you for choosing FedEx,
Warren Casey,
Sr. Delivery Manager.


Re: Just got CF deployed - What is a good first application tutorial?

admin@...
 

Hey that explicit build pack worked great!

As a newb, could you tell me where I would change the config for the app to use this build pack rather than they original?

On Oct 4, 2015, at 7:47 PM, Max Hufnagel <mhufnagel(a)pivotal.io> wrote:

Hello!

It looks to me like the example application in https://docs.cloudfoundry.org/deploying/ec2/example_apps.html <https://docs.cloudfoundry.org/deploying/ec2/example_apps.html> was updated 7 months ago and doesn't recognize the new buildpack. Maybe if you explicitly give it an old buildpack in your push command?

cf push your-app-name -b https://github.com/cloudfoundry/ruby-buildpack.git#v1.3.0 <https://github.com/cloudfoundry/ruby-buildpack.git#v1.3.0>



On Sun, Oct 4, 2015 at 2:36 PM, Joseph Biron <admin(a)josephbiron.com <mailto:admin(a)josephbiron.com>> wrote:
Thanks, but

-----> Compiling Ruby/Rack
Could not get translated url, exited with: DEPENDENCY_MISSING_IN_MANIFEST: https://s3-external-1.amazonaws.com/heroku-buildpack-ruby/ruby-2.1.5.tgz <https://s3-external-1.amazonaws.com/heroku-buildpack-ruby/ruby-2.1.5.tgz>


When I try the s3 URL in a browser I see

<Error>
<Code>AccessDenied</Code>
<Message>Access Denied</Message>
<RequestId>0A8DA74BC355E61E</RequestId>
<HostId>
OAHOWhKzXcaZmIwMFTYXJmaLvzvyC5HxNRORtzEPvOZ/TFjWMvr8foQTqSuPYPxp
</HostId>
</Error>


On Oct 4, 2015, at 3:55 PM, remi clotaire tassing tagne <tassingremi(a)gmail.com <mailto:tassingremi(a)gmail.com>> wrote:

Example app deployed on CF: https://docs.cloudfoundry.org/deploying/ec2/example_apps.html <https://docs.cloudfoundry.org/deploying/ec2/example_apps.html>

7321 - 7340 of 9425