Date   

Re: Initialization script for SSHFS

Daniel Mikusa
 

There's a branch of that repo that uses a .profile.d script.


https://github.com/dmikusa-pivotal/cf-ex-wordpress/blob/pcf-sshfs-example/.profile.d/setup.sh

It's a little different than the Python one as it pulls the connection info
from a bound service, but you don't have to do that.


https://github.com/dmikusa-pivotal/cf-ex-wordpress/blob/pcf-sshfs-example/.profile.d/setup.sh#L33-L36

Some additional comments inline...


#!/bin/bash
mv $HOME/app/.ssh $HOME/
chmod 644 $HOME/.ssh/*
chmod 600 $HOME/.ssh/sshfs_rsa
mv $HOME/app/main.rb /tmp/
mkdir -p $HOME/app/SSHFS/
sshfs root(a)192.168.1.15:

Can you access 192.168.1.15 from the container? You might try adding a
`ping -c 5 192.168.1.15` earlier in the script to test that. If it doesn't
work check your security groups and make sure that you can route to that
network.



/root/ssh_target/ $HOME/app/SSHFS -o IdentityFile=$HOME/.ssh/sshfs_rsa -o
StrictHostKeyChecking=yes -o UserKnownHostsFile=$HOME/.ssh/known_hosts -o
idmap=user -o cache=yes -o kernel_cache -o compression=no -o large_read
Try running this command from your PC and make sure it works and that it
connects without prompting for a password. That last bit is critical or
you need to change the command so that the password is piped in like this.

https://github.com/dmikusa-pivotal/cf-ex-wordpress/blob/pcf-sshfs-example/.profile.d/setup.sh#L48-L56



mv /tmp/main.rb $HOME/app/SSHFS/
fusermount -uz $HOME/app/SSHFS
The Python script is run by the build pack, so it runs during staging.
This is why it unmounts the volume. A .profile.d script runs in the
runtime container prior to the application starting. You probably don't
want to unmount the drive or it won't be accessible. You also want to make
sure that it runs in the background or your script will just hang.

Dan


Re: [abacus] Usage submission authorization

Jean-Sebastien Delfino
 

Hi Piotr,

In some cases it may not be possible or viable to create new scope for
each resource id e.g. short lived resources.

Why wouldn't that be possible? What type of short-lived resources did you
have in mind?

The typical use case I've seen is for a Cloud platform to decide to offer a
new type of database or analytics or messaging service, or a new type of
runtime for example. Before that new resource is offered on the platform,
their resource provider needs to get on board, get a user id, auth
credentials defined in UAA etc... You probably also need to define how
you're going to meter that new resource and the pricing for it.

Couldn't a scope be created in UAA at that time along all these other on
boarding steps?

Another reason why I'm not sure about short lived resources, is that
although you may decide to stop offering a type a resource at some point,
once you've metered it, and sent a bill for it to a customer, I don't think
you can really 'forget' about its existence anymore... So in that sense I'm
not sure how it can be 'short lived'.

Some flexibility would also help to accommodate changes related to
grouping resources by type as discussed in [1].

We discussed two options in [1]:
a) support a resource_type in addition to resource_id for grouping many
resource_ids under a single type
b) a common resource_id for several resources (something like 'node' for
all your versions of Node.js build packs for example)

Since option (a) is not implemented at this point and Issue #38 is actually
assigned to a 'future' milestone, AIUI resource providers need to use
option (b) with a common resource_id for multiple resources. Is creating a
scope for that common id still too much of a burden then?

[1] - https://github.com/cloudfoundry-incubator/cf-abacus/issues/38

Thoughts?

- Jean-Sebastien

On Wed, Oct 7, 2015 at 5:51 PM, Piotr Przybylski <piotrp(a)us.ibm.com> wrote:

Hi Sebastien,

That OAuth token should include:
- a user id uniquely identifying that resource provider;
- an OAuth scope named like abacus.usage.<resource_id>.write
What kind of customization of the above do you plan to expose? In some
cases it may not be possible or viable to create new scope for each
resource id e.g. short lived resources. The ability to either configure
scope to use for validation or provide scope 'mapping' would help to adapt
it to existing deployments. Some flexibility would also help to accommodate
changes related to grouping resources by type as discussed in [1].

[1] - https://github.com/cloudfoundry-incubator/cf-abacus/issues/38


Piotr



[image: Inactive hide details for Jean-Sebastien Delfino ---10/07/2015
12:30:05 AM---Hi Piotr, > what kind of authorization is required]Jean-Sebastien
Delfino ---10/07/2015 12:30:05 AM---Hi Piotr, > what kind of authorization
is required to submit usage to Abacus ?

From: Jean-Sebastien Delfino <jsdelfino(a)gmail.com>
To: "Discussions about Cloud Foundry projects and the system overall." <
cf-dev(a)lists.cloudfoundry.org>
Date: 10/07/2015 12:30 AM
Subject: [cf-dev] Re: [abacus] Usage submission authorization
------------------------------



Hi Piotr,

what kind of authorization is required to submit usage to Abacus ?
Is the oauth token used for submission [1] required to have particular
scope, specific to resource or resource provider ?

A resource provider is expected to present an OAuth token with the usage
it submits for a (service or runtime) resource.

That OAuth token should include:
- a user id uniquely identifying that resource provider;
- an OAuth scope named like abacus.usage.<resource_id>.write.

The precise naming syntax for that scope may still evolve in the next few
days as we progress with the implementation of user story 101703426 [1].

Is there a different scope required to submit runtimes usage (like cf
bridge) versus other services or its possible to use single scope for all
the submissions

I'd like to handle runtimes and services consistently as they're basically
just different types of resources, i.e. one scope per 'service' resource,
one scope per 'runtime' resource.

We're still working on the detailed design and implementation, but I'm not
sure we'd want to share scopes across (service and runtime) resource
providers as that'd allow a resource provider to submit usage for resources
owned by another...

@assk / @sasrin, anything I missed? Thoughts?

-- Jean-Sebastien


On Tue, Oct 6, 2015 at 6:29 PM, Piotr Przybylski <*piotrp(a)us.ibm.com*
<piotrp(a)us.ibm.com>> wrote:

Hi,
what kind of authorization is required to submit usage to Abacus ?
Is the oauth token used for submission [1] required to have particular
scope, specific to resource or resource provider ? Is there a different
scope required to submit runtimes usage (like cf bridge) versus other
services or its possible to use single scope for all the submissions ?


[1] - *https://www.pivotaltracker.com/story/show/101703426*
<https://www.pivotaltracker.com/story/show/101703426>

Piotr





Re: CF v205 / Pushing an app

Dieu Cao <dcao@...>
 

We generally recommend to use a separate system domain from the shared apps
domain.
That could look like
Our test environment for example uses
a1.cf-app.com as the system domain and a1-app.cf-app.com as the default
shared apps domain.
This is because system components bypass cloud controller when registering
routes with the gorouter.
If you wish to use the same domain for system components and apps, a work
around is to create a space that squats on routes that would be used by
system components.

-Dieu
CF CAPI PM

On Thu, Oct 8, 2015 at 10:10 AM, Jim Park <spark(a)pivotal.io> wrote:

The cf-release templates allow for a "system_domain" (
https://github.com/cloudfoundry/cf-release/blob/master/templates/cf-jobs.yml#L616),
this allows for a separate namespace for non-app hostnames("app_domain").
Hope this helps

Thanks,


Jim

On Thu, Oct 8, 2015 at 2:49 AM Sylvain Gibier <sylvain(a)munichconsulting.de>
wrote:

Hi,

Bug ?

Pushing an application using a manifest.yml file with an host entry set
to 'api', you are able to override the default route api.<<mycf.domain>>
with the application - preventing further cf commands to work correctly, as
it redirects traffic to new application.
I would have expect an error indicating the api.<<mycf.domain>> is
already used to prevent anyone to override api, uaa and login endpoint with
a custom app. Or am I missing something?

Is it a way - to make application/host like api, uaa, login ... not been
able to be overrided by custom application binding?

Sylvain


Re: CF deployment environments available for CF incubating projects to use?

Jean-Sebastien Delfino
 

It is ready to be given a try, and it is still under change and
development.

OK thanks so much for all the detailed info you've included here. I'll dig
into it and will report back here for any questions or issues.
Thanks!

I would love to get to a world that deploying CF is so easy that the
public documentation suffices for any individual team to set up their own
CI easily.

+1 from me :) as IMO that's key for growth and traction in a public and
open community.

- Jean-Sebastien

On Wed, Oct 7, 2015 at 8:32 PM, Amit Gupta <agupta(a)pivotal.io> wrote:

Hi Jean-Sebastien.

Yes and yes. It is ready to be given a try, and it is still under change
and development. There are several tools in mega-ci:

1. provision AWS for BOSH and Concourse (and optionally CF), and deploy
BOSH.
2. deploy Concourse into that environment
3. provision AWS for BOSH and CF (without Concourse), and deploy BOSH.

The "provision AWS" parts are done via CloudFormation templates. I'm not
sure how to characterize the dependency on AWS API there, but the
CloudFormation templates have a format version, and we currently use the
only valid format, which has been stable for 5 years:


http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/format-version-structure.html

Tools 1 (
https://github.com/cloudfoundry/mega-ci/#provisioning-aws-for-bosh-and-concourse-and-deploying-bosh)
and 2 (https://github.com/cloudfoundry/mega-ci/#deploying-concourse)
mentioned above get you a concourse ready to run builds that deploy CF.
Depending on how well you understand AWS, BOSH, Concourse, and CF, the
pipeline config ymls and CI scripts in the same mega-ci repo can be
modified and used to continuously deploy CF to your environment. But they
are not ideal for the onboarding experience.

As for your point about whether other projects have to figure out
individually how to operate CI builds, that's a very poignant question.
Currently, the answer is yes. There is a dojo program designed to allow
people in the foundation to pair with the core teams to learn the system,
CI practices, etc. I would love to get to a world that deploying CF is so
easy that the public documentation suffices for any individual team to set
up their own CI easily.

Among the 20 or so core development teams, distributed around the world,
comprised of 100+ developers from several different companies, there
actually is a fairly robust, well integrated set of tools and practices
around CI and how we consume and produce assets for other pipelines. As
the foundation grows, the onboarding story will need to be improved. But
this is no small problem.

There are 100s of v219+ builds happening every day (and next week, there
will be v220+ builds). For now, I'd recommend opening up mailing list
issues or Github issues (e.g. here:
https://github.com/cloudfoundry/cf-release/issues) with the specific
errors you're encountering, the core development teams are happy to help
the community past the initial technical and conceptual bootstrapping
hurdles.

And to address your question about the financials, I don't know the
complete picture, but the question "who foots the bills for all the current
CIs?" and "who should foot the bill for a publicly available pool of CI
environments?" probably have different answers.

Cheers,
Amit, OSS Release Integration PM

On Wed, Oct 7, 2015 at 3:48 PM, Jean-Sebastien Delfino <
jsdelfino(a)gmail.com> wrote:

Hi Amit,

We're working on making deploying CF itself less burdensome, but that
will take more time.

Is that tooling [1] ready for use? In other words is it ready for me to
give it a try, or is still under dev?
How strong is the dependency on the AWS API?

As for having someone actually maintain CF environments for the
community or dev teams to checkout and use as a service, the question is,
who foots the bill?

There's obviously a lot that I don't know yet about the CF community as a
new kid on the block here, but I see that we have a common CF mailing list
infra, common Github org and repos, some Web sites, some governance. So for
me, naively, the next logical thing to look for as a resource operated by
and for the CF community, was... a common build infra for the CF projects
to share :) Who foots the bill for the other existing CF common resources?

[1] https://github.com/cloudfoundry/mega-ci

-- Jean-Sebastien

On Tue, Oct 6, 2015 at 9:03 AM, Amit Gupta <agupta(a)pivotal.io> wrote:

The CF OSS Release Integration team (aka MEGA) is working on tooling to
fully bootstrap an AWS VPC with BOSH and Concourse running, and sufficient
AWS resources (eg subnets, ELB) to deploy CF:
https://github.com/cloudfoundry/mega-ci

We're working on making deploying CF itself less burdensome, but that
will take more time.

As for having someone actually maintain CF environments for the
community or dev teams to checkout and use as a service, the question is,
who foots the bill?

Amit

On Tuesday, October 6, 2015, Michael Maximilien <mmaximilien(a)gmail.com>
wrote:

Chiming in with some info and to add to Sebastien's request.

While most of the teams in CF have switched to using concourse for
their pipelines. And each pipeline instantiation has to be built and manage
individually. The issue mentioned here, needing to provision latest CF
releases to use for testing apps, services, brokers, etc is a real one. And
likely one many solve also individually.

Begs the question whether it should be offered as a service for all to
use. Almost like "CF as a Service" type of service broker... Just thinking
out loud here. Cloud capacity and managing resources would be the long
running costs after initial investment.

Best,

Max


Sent from Mailbox <https://www.dropbox.com/mailbox>


On Mon, Oct 5, 2015 at 7:27 PM, Jean-Sebastien Delfino <
jsdelfino(a)gmail.com> wrote:

Hi all,

So far in the Abacus project we've been running our automated tests
outside CF (as our Node.js apps can also run outside with just a bit of env
variable config) on Travis-CI. Some of us also deploy our apps to Bosh lite
to test inside CF but maintaining working versions of Bosh lite is pretty
time consuming and that manual testing hasn't been a repeatable process so
far, so I'd really like to automate that with a proper CI build and test
environment.

Are there any CF deployment environments available for CF incubating
projects to use for CI builds and tests?

I'm looking for an environment where my build script could simply
select a specific version of CF, bootstrap it 'clean' (nothing left over
from previous runs), deploy the Abacus apps to it to run the tests there,
then repeat with a different version of CF etc.

Is anything like that already available for CF projects to use?

Thanks!

-- Jean-Sebastien


Re: Multi-Line Loggregator events and the new Splunk "HTTP Event Collector" API

Mike Youngstrom
 

Thanks for the response Rohit. I hope this is the beginning of a good long
discussion on the topic. :)

Before going too deep with the '\' proposal are you aware if the
loggregator team considered any other possible ways an application could
hint to the agent that this line should wait for future lines before
sending the event? I'm not necessarily in love with the '\' approach just
throwing an idea out to start a discussion.

Mike

On Wed, Oct 7, 2015 at 7:58 PM, Rohit Kumar <rokumar(a)pivotal.io> wrote:

Hi Mike,

As Erik mentioned in the last thread, multi-line logging is something
which the loggregator team would like to solve. But there are a few
questions to answer before we can come up with a clean solution. We want a
design which solves the problem while not breaking existing apps which do
not require this functionality. Before implementing a solution we would
also want to answer if we want to do it for both runtimes or just Diego,
since the way log lines are sent to Metron differ based on the runtime.

If we were to implement the solution which you described, where newlines
are escaped with a '\', I guess the expectation is that loggregator would
internally remove the escape character. This has performance implications
because now some part of loggregator will need to inspect the log message
and coalesce the message with the succeeding ones. We will need to do this
in a way which respects multi-tenancy. That means now we are storing
additional state related to log lines per app. We will also need to decide
how long loggregator needs to wait for the next lines in a multi-line log,
before deciding to send the line which it received. To me that's not a
simple change.

I am happy to continue this discussion and hear your thoughts on the
existing proposal or any other design alternatives.

Thanks,
Rohit

On Wed, Oct 7, 2015 at 10:45 AM, Mike Youngstrom <youngm(a)gmail.com> wrote:

Splunk recently released its new "HTTP Event Collector" that greatly
simplifies how data can be streamed directly into Splunk without going to
an intermediate log file. It would be great to utilize this to efficiently
stream Loggregator information into Splunk.

For the most part loggregator appears to be very compatible with this API
with the exception of multi-line log messages.

The problem is that using this API splunk takes every request as an
independent splunk event. This completely eliminates anything splunk did
in the past to attempt to detect multi-line log messages.

Wouldn't it be great if a single loggregator event could contain multiple
log lines then these events could be easily streamed directly into Splunk
using this new api multiple lines preserved and all?

The previous attempt to bring up this topic fizzled [0]. With a new LAMB
PM coming I thought I'd ask my previous questions again.

In the previous thread [0] Erik mentioned a lot of work that he thought
would lead to multi-line log messages. But, it seems to me that the main
issue is simply how can a client actually communicate an multi-line event
to an agent? I don't think this issue is about breaking apart and then
combining log event rather how can I just I as a client hint to loggregator
that it should include multiple lines included into a single event?

Could it be as simple as escaping new lines with a '\' to notify the
agent to not end that event?

This problem cannot be solved without some help from loggregator.

Mike

[0]
https://lists.cloudfoundry.org/archives/list/cf-dev%40lists.cloudfoundry.org/thread/O6NDVGV44IBMVKZQXWOFIYOIC6CDU27G/


Initialization script for SSHFS

Cory Jett
 

I am looking for a way to push an application (ruby/node/java) and have a script run prior to the application starting that will setup SSHFS and move some of the content onto the share before the application starts. I was able to get the sample wordpress application working which includes this script that does exactly that but it is written in Python https://github.com/dmikusa-pivotal/cf-ex-wordpress/blob/master/.extensions/wordpress/extension.py. ideally, I would have a generic shell script that would run and set up SSHFS on deployment.

I attempted to accomplish this using a shell script in .profile.d but havent been able to get it working. If I get into a container and run the shell script it works fine. This is the script (which follows the same pattern as the wordpress python script, just in bash):


#!/bin/bash
mv $HOME/app/.ssh $HOME/
chmod 644 $HOME/.ssh/*
chmod 600 $HOME/.ssh/sshfs_rsa
mv $HOME/app/main.rb /tmp/
mkdir -p $HOME/app/SSHFS/
sshfs root(a)192.168.1.15:/root/ssh_target/ $HOME/app/SSHFS -o IdentityFile=$HOME/.ssh/sshfs_rsa -o StrictHostKeyChecking=yes -o UserKnownHostsFile=$HOME/.ssh/known_hosts -o idmap=user -o cache=yes -o kernel_cache -o compression=no -o large_read
mv /tmp/main.rb $HOME/app/SSHFS/
fusermount -uz $HOME/app/SSHFS

Any ideas what I am doing wrong or if there is a better way to accomplish this?


Re: [abacus] Usage processing authorization, was: Usage submission authorization

Jean-Sebastien Delfino
 

OK that confirms what I thought. Thanks!

Assk, any thoughts as well? Did that make sense to you?

-- Jean-Sebastien

On Wed, Oct 7, 2015 at 6:09 PM, Piotr Przybylski <piotrp(a)us.ibm.com> wrote:

Sebastien

So, I'm wondering if it still makes sense to use the resource provider's
token inside our *asynchronous* usage *processing* pipeline. Shouldn't we
require the individual processing steps to obtain their own tokens instead?

It seems natural that the Abacus should use its own token - or tokens to
authenticate and authorize steps in the pipeline. After the submission
Abacus 'takes custody' of the submitted data so it should be solely
responsible for authrizing its processing.

Piotr

Piotr Przybylski | IBM Bluemix | piotrp(a)us.ibm.com | 650-645-8213


[image: Inactive hide details for Jean-Sebastien Delfino ---10/07/2015
09:21:05 AM---Hi all, A few more thoughts on a different, but re]Jean-Sebastien
Delfino ---10/07/2015 09:21:05 AM---Hi all, A few more thoughts on a
different, but related subject.

From: Jean-Sebastien Delfino <jsdelfino(a)gmail.com>
To: "Discussions about Cloud Foundry projects and the system overall." <
cf-dev(a)lists.cloudfoundry.org>
Date: 10/07/2015 09:21 AM
Subject: [cf-dev] Re: [abacus] Usage processing authorization, was: Usage
submission authorization
------------------------------



Hi all,

A few more thoughts on a different, but related subject.

We're using the resource provider's token and scopes to authorize usage
*submission* to our usage collector service. I agree with that and have
confirmed it in my answer to Piotr's question below.

It also looks like we're using the resource provider's token as well to
flow usage data through our usage *processing* pipeline (after we've
authorized the usage submission, validated that usage and taken
responsibility for it). I'm wondering if we couldn't find a better
approach, as we will run into a number of issues with this:

- processing delays in the pipeline can cause the token to expire, at that
point the resource provider is out of the loop, can't do anything about it,
and it won't make much sense anyway to ask the resource provider for a new
token way after it has submitted its usage;

- when restarting an Abacus service and recovering after a processing
interruption, we don't have a valid resource provider token either;

- more generally, I find a bit odd to use the resource provider's token in
usage processing steps down the Abacus pipeline, as they're really just
processing usage passed to them by the previous Abacus processing step (or
could have just read that usage from one of our usage DBs, and again the
resource provider wouldn't be so relevant as it wouldn't even have written
that usage to that DB itself.)

So, I'm wondering if it still makes sense to use the resource provider's
token inside our *asynchronous* usage *processing* pipeline. Shouldn't we
require the individual processing steps to obtain their own tokens instead?

Thoughts?

-- Jean-Sebastien


On Tue, Oct 6, 2015 at 10:36 PM, Jean-Sebastien Delfino <
*jsdelfino(a)gmail.com* <jsdelfino(a)gmail.com>> wrote:

Hi Piotr,

> what kind of authorization is required to submit usage to Abacus ?
> Is the oauth token used for submission [1] required to have
particular scope, specific to resource or resource provider ?

A resource provider is expected to present an OAuth token with the
usage it submits for a (service or runtime) resource.

That OAuth token should include:
- a user id uniquely identifying that resource provider;
- an OAuth scope named like abacus.usage.<resource_id>.write.

The precise naming syntax for that scope may still evolve in the next
few days as we progress with the implementation of user story 101703426 [1].

> Is there a different scope required to submit runtimes usage (like
cf bridge) versus other services or its possible to use single scope for
all the submissions

I'd like to handle runtimes and services consistently as they're
basically just different types of resources, i.e. one scope per 'service'
resource, one scope per 'runtime' resource.

We're still working on the detailed design and implementation, but I'm
not sure we'd want to share scopes across (service and runtime) resource
providers as that'd allow a resource provider to submit usage for resources
owned by another...

@assk / @sasrin, anything I missed? Thoughts?

-- Jean-Sebastien


On Tue, Oct 6, 2015 at 6:29 PM, Piotr Przybylski <*piotrp(a)us.ibm.com*
<piotrp(a)us.ibm.com>> wrote:
Hi,
what kind of authorization is required to submit usage to Abacus ?
Is the oauth token used for submission [1] required to have particular
scope, specific to resource or resource provider ? Is there a different
scope required to submit runtimes usage (like cf bridge) versus other
services or its possible to use single scope for all the submissions ?


[1] - *https://www.pivotaltracker.com/story/show/101703426*
<https://www.pivotaltracker.com/story/show/101703426>

Piotr





Re: CF v205 / Pushing an app

Jim Park
 

The cf-release templates allow for a "system_domain" (
https://github.com/cloudfoundry/cf-release/blob/master/templates/cf-jobs.yml#L616),
this allows for a separate namespace for non-app hostnames("app_domain").
Hope this helps

Thanks,


Jim

On Thu, Oct 8, 2015 at 2:49 AM Sylvain Gibier <sylvain(a)munichconsulting.de>
wrote:

Hi,

Bug ?

Pushing an application using a manifest.yml file with an host entry set to
'api', you are able to override the default route api.<<mycf.domain>> with
the application - preventing further cf commands to work correctly, as it
redirects traffic to new application.
I would have expect an error indicating the api.<<mycf.domain>> is already
used to prevent anyone to override api, uaa and login endpoint with a
custom app. Or am I missing something?

Is it a way - to make application/host like api, uaa, login ... not been
able to be overrided by custom application binding?

Sylvain


Re: cloud_controller_ng performance degrades slowly over time

Amit Kumar Gupta
 

We've seen issues on some environments where requests to cc that involve cc
making a request to uaa or hm9k have a 5s delay while the local consul
agent fails to resolves the DNS for uaa/hm9k, before moving on to a
different resolver.

The expected behavior observed in almost all environments is that the DNS
request to consul agent fails fast and moves on to the next resolver, we
haven't figured out why a couple envs exhibit different behavior. The
impact is a 5 or 10s delay (5 or 10, not 5 to 10). It doesn't explain your
1:20 delay though. Are you always seeing delays that long?

Amit

On Thursday, October 8, 2015, Zach Robinson <zrobinson(a)pivotal.io> wrote:

Hey Matt,

I'm trying to think of other things that would affect only the endpoints
that interact with UAA and would be fixed after a CC restart. I'm
wondering if it's possible there are a large number of connections being
kept-alive, or stuck in a wait state or something. Could you take a look
at the netstat information on the CC and UAA next time this happens?

-Zach and Swetha


Re: cloud_controller_ng performance degrades slowly over time

Zach Robinson
 

Hey Matt,

I'm trying to think of other things that would affect only the endpoints that interact with UAA and would be fixed after a CC restart. I'm wondering if it's possible there are a large number of connections being kept-alive, or stuck in a wait state or something. Could you take a look at the netstat information on the CC and UAA next time this happens?

-Zach and Swetha


Re: UAA not sending routes registration and updates

Amit Kumar Gupta
 

What version of cf-release? If it's recent, see the "Important" section of
the release notes about colocating the new route_registrar for v217 and
v218.

https://github.com/cloudfoundry/cf-release/releases/tag/v218

Amit

On Thursday, October 8, 2015, Haitao Jiang <jianghaitao(a)gmail.com> wrote:

I filed a GitHub issue:
https://github.com/cloudfoundry/cf-registrar/issues/7

What is happening was that
- NATS receives route registration from CC and Traffic Controller
- UAA not sending route registration to NATS, so UAA's routes are missing
from gorouter
- UAA's cf registrar stuck after following (instead of sending route
registration messages):
Connected to NATS - varz registration
Announcing start up vcap.component.announce

bosh vms, bosh cck, and monit all saying that everything running.

Any suggestion on how to troubleshoot this? What are the possible reasons
cf-registrar not sending route.register messages?


Java Buildpack v3.3

Christopher Frost
 

I'm pleased to announce the release of the java-buildpack, version 3.3. This
release contains updates to various dependencies.

- When processing Java Options the $ and \ characters are no longer
escaped to allow environment properties to be used. (see the
documentation
<https://github.com/cloudfoundry/java-buildpack/blob/master/docs/framework-java_opts.md#escaping-strings>
)
- Improved Luna Security Provider HA Support
- Improved configuration of the DynaTrace agent. (via Tom Collings
<https://github.com/cloudfoundry/java-buildpack/pull/235>)
- Better AppDynamics code comments. (via Nikhil Katre
<https://github.com/cloudfoundry/java-buildpack/pull/229>)
- Better documentation of the Oracle JRE support. (via Dominik Bartholdi
<https://github.com/cloudfoundry/java-buildpack/pull/230>)

For a more detailed look at the changes in 3.3, please take a look at
the commit
log <https://github.com/cloudfoundry/java-buildpack/compare/v3.2...v3.3>.
Packaged versions of the buildpack, suitable for use with create-buildpack
and update-buildpack, can be found attached to this release
<https://github.com/cloudfoundry/java-buildpack/releases/tag/v3.3>.
*Packaged Dependencies*

- AppDynamics Agent: 4.1.4_2
- GemFire 8.0.0
- GemFire Modules 8.0.0.1
- GemFire Modules Tomcat7 8.0.0.1
- GemFire Security 8.0.0
- Groovy: 2.4.5
- JRebel 6.2.5
- MariaDB JDBC: 1.2.2
- Memory Calculator (mountainlion): 2.0.0.RELEASE
- Memory Calculator (precise): 2.0.0.RELEASE
- Memory Calculator (trusty): 2.0.0.RELEASE
- New Relic Agent: 3.21.0
- OpenJDK JRE (mountainlion): 1.8.0_60
- OpenJDK JRE (precise): 1.8.0_60
- OpenJDK JRE (trusty): 1.8.0_60
- Play Framework JPA Plugin: 1.10.0.RELEASE
- PostgreSQL JDBC: 9.4.1203
- RedisStore: 1.2.0_RELEASE
- Spring Auto-reconfiguration: 1.10.0_RELEASE
- Spring Boot CLI: 1.2.6_RELEASE
- Tomcat Access Logging Support: 2.4.0_RELEASE
- Tomcat Lifecycle Support: 2.4.0_RELEASE
- Tomcat Logging Support: 2.4.0_RELEASE
- Tomcat: 8.0.27


Christopher Frost - Pivotal UK
Java Buildpack Team


UAA not sending routes registration and updates

Haitao Jiang
 

I filed a GitHub issue: https://github.com/cloudfoundry/cf-registrar/issues/7

What is happening was that
- NATS receives route registration from CC and Traffic Controller
- UAA not sending route registration to NATS, so UAA's routes are missing from gorouter
- UAA's cf registrar stuck after following (instead of sending route registration messages):
Connected to NATS - varz registration
Announcing start up vcap.component.announce

bosh vms, bosh cck, and monit all saying that everything running.

Any suggestion on how to troubleshoot this? What are the possible reasons cf-registrar not sending route.register messages?


Re: "bosh ssh" times out

Amit Kumar Gupta
 

You can use your bosh director as a gateway:

bosh ssh --gateway-host ADDRESS-OF-DIRECTOR --gateway-user vcap

On Thursday, October 8, 2015, Daniel Mikusa <dmikusa(a)pivotal.io> wrote:

I was under the impression that you need to be able to connect directly,
but I can't say that definitively. Perhaps someone else can confirm.

Dan


On Thu, Oct 8, 2015 at 11:43 AM, Remi Tassing <tassingremi(a)gmail.com
<javascript:_e(%7B%7D,'cvml','tassingremi(a)gmail.com');>> wrote:

Hi Daniel,
10.0.16.103 is the internal address of that particular VM within the VPC
(I've deployed CF in AWS). So I can't access it directly.
I thought that was the point of using "bosh ssh", i.e., connecting to the
ha_proxy (which has a public address) first and then to the VM.

I have a feeling I've completed missed the point

Remi


Re: "bosh ssh" times out

Jim Park
 

You can proxy through the director or whatever else if you'd like with
`bosh ssh --gateway_host director.example.com --gateway_user vcap
--gateway_identity_file /path/to/bosh_directors_key`.

BOSH director only manages creating a one-time use user login with sudo
privileges and passes it back to bosh_cli.

We use a bastion host to perform BOSH-ey things because of this.

Thanks,


Jim

On Thu, Oct 8, 2015 at 8:49 AM Daniel Mikusa <dmikusa(a)pivotal.io> wrote:

I was under the impression that you need to be able to connect directly,
but I can't say that definitively. Perhaps someone else can confirm.

Dan


On Thu, Oct 8, 2015 at 11:43 AM, Remi Tassing <tassingremi(a)gmail.com>
wrote:

Hi Daniel,
10.0.16.103 is the internal address of that particular VM within the VPC
(I've deployed CF in AWS). So I can't access it directly.
I thought that was the point of using "bosh ssh", i.e., connecting to the
ha_proxy (which has a public address) first and then to the VM.

I have a feeling I've completed missed the point

Remi


Re: "bosh ssh" times out

Daniel Mikusa
 

I was under the impression that you need to be able to connect directly,
but I can't say that definitively. Perhaps someone else can confirm.

Dan

On Thu, Oct 8, 2015 at 11:43 AM, Remi Tassing <tassingremi(a)gmail.com> wrote:

Hi Daniel,
10.0.16.103 is the internal address of that particular VM within the VPC
(I've deployed CF in AWS). So I can't access it directly.
I thought that was the point of using "bosh ssh", i.e., connecting to the
ha_proxy (which has a public address) first and then to the VM.

I have a feeling I've completed missed the point

Remi


Re: "bosh ssh" times out

Remi Tassing
 

Hi Daniel,
10.0.16.103 is the internal address of that particular VM within the VPC (I've deployed CF in AWS). So I can't access it directly.
I thought that was the point of using "bosh ssh", i.e., connecting to the ha_proxy (which has a public address) first and then to the VM.

I have a feeling I've completed missed the point

Remi


Re: "bosh ssh" times out

Daniel Mikusa
 

Have you checked that the connection is not being blocked by a firewall?
What happens if you SSH directly to that IP?

Dan

On Thu, Oct 8, 2015 at 11:22 AM, Remi Tassing <tassingremi(a)gmail.com> wrote:

Hi,
I was trying "bosh ssh" in the interactive mode and after choosing the VM
it hangs for a bit then times out. I was following this tutorial:
http://docs.pivotal.io/pivotalcf/customizing/trouble-advanced.html

Console snippet:
....
13. stats_z1/0
Choose an instance: 1
Acting as user 'admin' on deployment 'cf' on 'microbosh'
Enter password (use it to sudo on remote host): *
Target deployment is `cf'

Setting up ssh artifacts

Director task 43

Task 43 done
Starting interactive shell on job nats_z1/0
ssh: connect to host 10.0.16.103 port 22: Connection timed out
...

Has anyone encountered this problem? Is there other alternative?

Remi


"bosh ssh" times out

Remi Tassing
 

Hi,
I was trying "bosh ssh" in the interactive mode and after choosing the VM it hangs for a bit then times out. I was following this tutorial: http://docs.pivotal.io/pivotalcf/customizing/trouble-advanced.html

Console snippet:
....
13. stats_z1/0
Choose an instance: 1
Acting as user 'admin' on deployment 'cf' on 'microbosh'
Enter password (use it to sudo on remote host): *
Target deployment is `cf'

Setting up ssh artifacts

Director task 43

Task 43 done
Starting interactive shell on job nats_z1/0
ssh: connect to host 10.0.16.103 port 22: Connection timed out
...

Has anyone encountered this problem? Is there other alternative?

Remi


Re: Metron: Timed out talking to store

Kyle Havlovitz (kyhavlov)
 

I only have one ETCD node, and it's the correct address in the metron config.

From: Rohit Kumar <rokumar(a)pivotal.io<mailto:rokumar(a)pivotal.io>>
Reply-To: "Discussions about Cloud Foundry projects and the system overall." <cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org>>
Date: Wednesday, October 7, 2015 at 8:24 PM
To: "Discussions about Cloud Foundry projects and the system overall." <cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org>>
Subject: [cf-dev] Re: Metron: Timed out talking to store

Hi Kyle,

How many nodes do you have in your ETCD cluster? Also can you check whether the ETCD servers listed in the metron config match the IP addresses of the machines in your cluster. The config file is located in /var/vcap/jobs/metron_agent/config/metron_agent.json and you should look for the "EtcdUrls" field.

Rohit

On Wed, Oct 7, 2015 at 3:12 PM, Kyle Havlovitz (kyhavlov) <kyhavlov(a)cisco.com<mailto:kyhavlov(a)cisco.com>> wrote:
I'm seeing this error in the Metron logs repeatedly: "ServerAddressList.Run: Timed out talking to store; will try again soon." Metron seems to be able to connect to ETCD just fine, and I can curl /v2/keys/healthstatus/doppler from the machine. I'm not sure what would cause this error and can't think of anything else to try. Later, when it gets logs from the dea agent, it gives the error "can't forward message: loggregator client pool is empty", presumably because of the previous error.

I can't figure out what the problem is with this; the zone property in the logging config files matches and it doesn't seem like a firewall problem, could anyone give advice? CF version is 217.

7241 - 7260 of 9422