Thanks for the announcement.

To be clear is this announcement to cease support for the old v1 brokers
or is this to eliminate support for the v1 api in the CC? Does the v1 CC
code depend on NATS? None of my custom v1 brokers depend on NATS.

Mike

On Tue, Sep 22, 2015 at 6:01 PM, Dieu Cao <dcao(a)pivotal.io> wrote:

Hello all,

We plan to remove support for v1 service brokers in about 3 months, in a
cf-release following 12/31/2015.
We are working towards removing CF's dependency on NATS and the v1
service brokers are still dependent on NATS.
Please let me know if you have questions/concerns about this timeline.

I'll be working on verifying a set of steps that you can find here [1]
that document how to migrate your service broker from v1 to v2 and what is
required in order to persist user data and will get that posted to the
service broker api docs officially.

-Dieu
CF CAPI PM

[1]
https://docs.google.com/document/d/1Pl1o7mxtn3Iayq2STcMArT1cJsKkvi4Ey1-d3TB_Nhs/edit?usp=sharing

Please check the file collector/collector.log, it's in a subdirectory of
the unpacked log tarball.

On Wed, Sep 23, 2015 at 12:01 AM, Guangcai Wang <guangcai.wang(a)gmail.com>
wrote:

Actually, I checked the two files in status_z1 job VM. I did not find any
clues. Attached for reference.

On Wed, Sep 23, 2015 at 4:54 PM, Amit Gupta <agupta(a)pivotal.io> wrote:

If you do "bosh logs stats_z1 0 --job" you will get a tarball of all
the logs for the relevant processes running on the stats_z1/0 VM. You will
likely find some error messages in the collectors stdout or stderr logs.

On Tue, Sep 22, 2015 at 11:30 PM, Guangcai Wang <guangcai.wang(a)gmail.com

wrote:

It does not help.

I always see the "collector" process bouncing between "running" and
"does not exit" when I use "monit summary" in a while loop.

Who knows how to get the real error when the "collector" process is not
failed? Thanks.

On Wed, Sep 23, 2015 at 4:11 PM, Tony <Tonyl(a)fast.au.fujitsu.com>
wrote:

My approach is to login on the stats vm and sudo, then
run "monit status" and restart the failed processes or simply restart
all
processes by running "monit restart all"

wait for a while(5~10 minutes at most)
If there is still some failed process, e.g. collector
then run ps -ef | grep collector
and kill the processes in the list(may be you need to run kill -9
sometimes)

then "monit restart all"

Normally, it will fix the issue "Failed: `XXX' is not running after
update"



--
View this message in context:
http://cf-dev.70369.x6.nabble.com/cf-dev-Error-400007-stats-z1-0-is-not-running-after-update-tp1901p1902.html
Sent from the CF Dev mailing list archive at Nabble.com.

If you do "bosh logs stats_z1 0 --job" you will get a tarball of all the
logs for the relevant processes running on the stats_z1/0 VM. You will
likely find some error messages in the collectors stdout or stderr logs.

On Tue, Sep 22, 2015 at 11:30 PM, Guangcai Wang <guangcai.wang(a)gmail.com>
wrote:

It does not help.

I always see the "collector" process bouncing between "running" and "does
not exit" when I use "monit summary" in a while loop.

Who knows how to get the real error when the "collector" process is not
failed? Thanks.

On Wed, Sep 23, 2015 at 4:11 PM, Tony <Tonyl(a)fast.au.fujitsu.com> wrote:

My approach is to login on the stats vm and sudo, then
run "monit status" and restart the failed processes or simply restart all
processes by running "monit restart all"

wait for a while(5~10 minutes at most)
If there is still some failed process, e.g. collector
then run ps -ef | grep collector
and kill the processes in the list(may be you need to run kill -9
sometimes)

then "monit restart all"

Normally, it will fix the issue "Failed: `XXX' is not running after
update"



--
View this message in context:
http://cf-dev.70369.x6.nabble.com/cf-dev-Error-400007-stats-z1-0-is-not-running-after-update-tp1901p1902.html
Sent from the CF Dev mailing list archive at Nabble.com.

My approach is to login on the stats vm and sudo, then
run "monit status" and restart the failed processes or simply restart all
processes by running "monit restart all"

wait for a while(5~10 minutes at most)
If there is still some failed process, e.g. collector
then run ps -ef | grep collector
and kill the processes in the list(may be you need to run kill -9
sometimes)

then "monit restart all"

Normally, it will fix the issue "Failed: `XXX' is not running after update"



--
View this message in context:
http://cf-dev.70369.x6.nabble.com/cf-dev-Error-400007-stats-z1-0-is-not-running-after-update-tp1901p1902.html
Sent from the CF Dev mailing list archive at Nabble.com.

Hello all,

We plan to remove support for v1 service brokers in about 3 months, in a
cf-release following 12/31/2015.
We are working towards removing CF's dependency on NATS and the v1 service
brokers are still dependent on NATS.
Please let me know if you have questions/concerns about this timeline.

I'll be working on verifying a set of steps that you can find here [1]
that document how to migrate your service broker from v1 to v2 and what is
required in order to persist user data and will get that posted to the
service broker api docs officially.

-Dieu
CF CAPI PM

[1]
https://docs.google.com/document/d/1Pl1o7mxtn3Iayq2STcMArT1cJsKkvi4Ey1-d3TB_Nhs/edit?usp=sharing

On 23.09.2015, at 05:26, Anthony <lee.apc(a)gmail.com> wrote:

Yep. --recent works. Other cf commands and cf curl also works.

It definitely is the websockets for loggregator. Just not sure what the right config for F5 (version 10.4) should be.

Regards,
Anthony

On Sep 22, 2015, at 9:53 PM, Rohit Kumar <rokumar(a)pivotal.io> wrote:

Does `cf logs --recent` work for you? The recent logs request goes over HTTP. If that goes through that means only the websocket requests to loggregator servers are a problem.

Rohit

On Tue, Sep 22, 2015 at 8:18 PM, Anthony <lee.apc(a)gmail.com> wrote:
Thanks Mike! Unfortunately, upgrading is not an option since its a really loaded enterprise device. The interesting part is that there is a similarly set up websockets vip (plain old server i think .net) that is working on the same device.

We'll work with our network folks to find other devices with newer software we can use.

Would appreciate if anyone has other ideas?

Regards,
Anthony

On Sep 22, 2015, at 7:49 PM, Mike Youngstrom <youngm(a)gmail.com> wrote:

We are running 11.4 and 11.6. I'd give an upgrade a try before digging too much deeper.

Mike

On Sep 22, 2015 6:36 PM, "Anthony" <lee.apc(a)gmail.com> wrote:
The version we are testing in is 10.4.

Regards,
Anthony

On Sep 22, 2015, at 6:41 PM, Mike Youngstrom <youngm(a)gmail.com> wrote:

What version of F5 software are you running?

Mike

On Tue, Sep 22, 2015 at 5:20 PM, Anthony Lee <lee.apc(a)gmail.com> wrote:
Does any one have any experience configuring F5 load balancers in front of the CF routers? We have configured F5 and app https and cf push requests are working fine. However, the connectivity with loggregator is not working. Taking a look at the documentation, it requires "websocket support" on the load balancer. We've done the configuration specified here:

https://support.f5.com/kb/en-us/solutions/public/14000/800/sol14814.html

With the following irule basically, applying the default TCP profile if it detects websocket traffic:

when HTTP_REQUEST {
if { [string tolower [HTTP::header Upgrade]] contains "websocket" }{
HTTP::disable
}
}

However, we are running into errors. Doing `cf logs myapp1` yields:

Error dialing loggregator server: read tcp <ip redacted>:443: connection reset by peer.
Please ask your Cloud Foundry Operator to check the platform configuration (loggregator endpoint is wss://loggregator.<sys domain redacted>:443).

Does anyone have a clue?

Thanks!
Anthony

On Tue, Sep 22, 2015 at 8:18 PM, Anthony <lee.apc(a)gmail.com> wrote:
Thanks Mike! Unfortunately, upgrading is not an option since its a really loaded enterprise device. The interesting part is that there is a similarly set up websockets vip (plain old server i think .net) that is working on the same device.

We'll work with our network folks to find other devices with newer software we can use.

Would appreciate if anyone has other ideas?

Regards,
Anthony

On Sep 22, 2015, at 7:49 PM, Mike Youngstrom <youngm(a)gmail.com> wrote:

We are running 11.4 and 11.6. I'd give an upgrade a try before digging too much deeper.

Mike

On Sep 22, 2015 6:36 PM, "Anthony" <lee.apc(a)gmail.com> wrote:
The version we are testing in is 10.4.

Regards,
Anthony

On Sep 22, 2015, at 6:41 PM, Mike Youngstrom <youngm(a)gmail.com> wrote:

What version of F5 software are you running?

Mike

On Tue, Sep 22, 2015 at 5:20 PM, Anthony Lee <lee.apc(a)gmail.com> wrote:
Does any one have any experience configuring F5 load balancers in front of the CF routers? We have configured F5 and app https and cf push requests are working fine. However, the connectivity with loggregator is not working. Taking a look at the documentation, it requires "websocket support" on the load balancer. We've done the configuration specified here:

https://support.f5.com/kb/en-us/solutions/public/14000/800/sol14814.html

With the following irule basically, applying the default TCP profile if it detects websocket traffic:

when HTTP_REQUEST {
if { [string tolower [HTTP::header Upgrade]] contains "websocket" }{
HTTP::disable
}
}

However, we are running into errors. Doing `cf logs myapp1` yields:

Error dialing loggregator server: read tcp <ip redacted>:443: connection reset by peer.
Please ask your Cloud Foundry Operator to check the platform configuration (loggregator endpoint is wss://loggregator.<sys domain redacted>:443).

Does anyone have a clue?

Thanks!
Anthony

Thanks Mike! Unfortunately, upgrading is not an option since its a really
loaded enterprise device. The interesting part is that there is a
similarly set up websockets vip (plain old server i think .net) that is
working on the same device.

We'll work with our network folks to find other devices with newer
software we can use.

Would appreciate if anyone has other ideas?

Regards,
Anthony

On Sep 22, 2015, at 7:49 PM, Mike Youngstrom <youngm(a)gmail.com> wrote:

We are running 11.4 and 11.6. I'd give an upgrade a try before digging
too much deeper.

Mike
On Sep 22, 2015 6:36 PM, "Anthony" <lee.apc(a)gmail.com> wrote:

The version we are testing in is 10.4.

Regards,
Anthony

On Sep 22, 2015, at 6:41 PM, Mike Youngstrom <youngm(a)gmail.com> wrote:

What version of F5 software are you running?

Mike

On Tue, Sep 22, 2015 at 5:20 PM, Anthony Lee <lee.apc(a)gmail.com> wrote:

Does any one have any experience configuring F5 load balancers in front
of the CF routers? We have configured F5 and app https and cf push requests
are working fine. However, the connectivity with loggregator is not
working. Taking a look at the documentation, it requires "websocket
support" on the load balancer. We've done the configuration specified here:

https://support.f5.com/kb/en-us/solutions/public/14000/800/sol14814.html

With the following irule basically, applying the default TCP profile if
it detects websocket traffic:

when HTTP_REQUEST {
if { [string tolower [HTTP::header Upgrade]] contains "websocket" }{
HTTP::disable
}
}

However, we are running into errors. Doing `cf logs myapp1` yields:

Error dialing loggregator server: read tcp <ip redacted>:443: connection
reset by peer.
Please ask your Cloud Foundry Operator to check the platform
configuration (loggregator endpoint is wss://loggregator.<sys domain
redacted>:443).

Does anyone have a clue?

Thanks!
Anthony

On Sep 22, 2015 6:36 PM, "Anthony" <lee.apc(a)gmail.com> wrote:
The version we are testing in is 10.4.

Regards,
Anthony

On Sep 22, 2015, at 6:41 PM, Mike Youngstrom <youngm(a)gmail.com> wrote:

What version of F5 software are you running?

Mike

On Tue, Sep 22, 2015 at 5:20 PM, Anthony Lee <lee.apc(a)gmail.com> wrote:
Does any one have any experience configuring F5 load balancers in front of the CF routers? We have configured F5 and app https and cf push requests are working fine. However, the connectivity with loggregator is not working. Taking a look at the documentation, it requires "websocket support" on the load balancer. We've done the configuration specified here:

https://support.f5.com/kb/en-us/solutions/public/14000/800/sol14814.html

With the following irule basically, applying the default TCP profile if it detects websocket traffic:

when HTTP_REQUEST {
if { [string tolower [HTTP::header Upgrade]] contains "websocket" }{
HTTP::disable
}
}

However, we are running into errors. Doing `cf logs myapp1` yields:

Error dialing loggregator server: read tcp <ip redacted>:443: connection reset by peer.
Please ask your Cloud Foundry Operator to check the platform configuration (loggregator endpoint is wss://loggregator.<sys domain redacted>:443).

Does anyone have a clue?

Thanks!
Anthony

Yes Amit. Thanks

I'm trying the 2 approaches since the both have their pros and cons.

is your compute environment a multi-tenant one that will be running

multiple different workloads?

Yes. dev can push their own spark-based apps and non-spark apps. The
spark-based apps would rely on the existing Spark cluster.

it's also likely to be a more efficient use of resources, since a BOSH VM

can only run one of these spark-job-processors,

I think a Spark cluster(using YARN) of BOSH VMs should be able to run
multiple spark jobs concurrently.

With the app deployment approach, I did setup a UPS for the Spark cluster
and I've been able to submit Spark jobs to the cluster programmatically
through the Spark API. I'll stay with app deployment for now until I get a
stronger use case for a boshrelease.

On Tue, Sep 22, 2015 at 12:21 AM, Amit Gupta <agupta(a)pivotal.io> wrote:

Hey Kayode,

Were you able to make any progress with the deployments you were trying
to do?

Best,
Amit

On Wed, Sep 16, 2015 at 12:48 PM, Amit Gupta <agupta(a)pivotal.io> wrote:

My very limited understanding is that NFS writes to the actual
filesystem, and achieves persistence by having centralized NFS servers
where it writes to a real mounted device, whereas the clients write to an
ephemeral nfs-mount.

My very limited understanding of HDFS is that it's all userland FS, does
not write to the actual filesystem, and relies on replication to other
nodes in the HDFS cluster. Being a userland FS, you don't have to worry
about the data being wiped when a container is shut down, if you were to
run it as an app.

I think one main issue is going to be ensuring that you never lose too
many instances (whether they are containers or VMs), since you might then
lose all replicas of a given data shard. Whether you go with apps or BOSH
VMs doesn't make a big difference here.

Deploying as an app may be a better way to go, it's simpler right now to
configure and deploy and app, than to configure and deploy a full BOSH
release. It's also likely to be a more efficient use of resources, since a
BOSH VM can only run one of these spark-job-processors, but a CF
container-runner can run lots of other things. That actually brings up a
different question: is your compute environment a multi-tenant one that
will be running multiple different workloads? E.g. could someone also use
the CF to push their own apps? Or is the whole thing just for your spark
jobs, in which case you might only be running one container per VM anyways?

Assuming you can make use of the VMs for other workloads, I think this
would be an ideal use case for Diego. You probably don't need all the
extra logic around apps, like staging and routing, you just need Diego to
efficiently schedule containers for you.

On Wed, Sep 16, 2015 at 1:13 PM, Kayode Odeyemi <dreyemi(a)gmail.com>
wrote:

Thanks Dmitriy,

Just for clarity, are you saying multiple instances of a VM cannot
share a single shared filesystem?

On Wed, Sep 16, 2015 at 6:59 PM, Dmitriy Kalinin <dkalinin(a)pivotal.io>
wrote:

BOSH allocates a persistent disk per instance. It never shares
persistent disks between multiple instances at the same time.

If you need a shared file system, you will have to use some kind of a
release for it. It's not any different from what people do with nfs
server/client.

On Wed, Sep 16, 2015 at 7:09 AM, Amit Gupta <agupta(a)pivotal.io> wrote:

The shared file system aspect is an interesting wrinkle to the
problem. Unless you use some network layer to how you write to the shared
file system, e.g. SSHFS, I think apps will not work because they get
isolated to run in a container, they're given a chroot "jail" for their
file system, and it gets blown away whenever the app is stopped or
restarted (which will commonly happen, e.g. during a rolling deploy of the
container-runner VMs).

Do you have something that currently works? How do your VMs
currently access this shared FS? I'm not sure BOSH has the abstractions
for choosing a shared, already-existing "persistent disk" to be attached to
multiple VMs. I also don't know what happens when you scale your VMs down,
because BOSH would generally destroy the associated persistent disk, but
you don't want to destroy the shared data.

Dmitriy, any idea how BOSH can work with a shared filesystem (e.g.
HDFS)?

Amit

On Wed, Sep 16, 2015 at 6:54 AM, Kayode Odeyemi <dreyemi(a)gmail.com>
wrote:

On Wed, Sep 16, 2015 at 3:44 PM, Amit Gupta <agupta(a)pivotal.io>
wrote:

Are the spark jobs tasks that you expect to end, or apps that you
expect to run forever?

They are tasks that run forever. The jobs are subscribers to
RabbitMQ queues that process
messages in batches.

Do your jobs need to write to the file system, or do they access a
shared/distributed file system somehow?

The jobs write to shared filesystem.

Do you need things like a static IP allocated to your jobs?

No.

Are your spark jobs serving any web traffic?

No.

Hello all,

We plan to remove support for v1 service brokers in about 3 months, in a
cf-release following 12/31/2015.
We are working towards removing CF's dependency on NATS and the v1 service
brokers are still dependent on NATS.
Please let me know if you have questions/concerns about this timeline.

I'll be working on verifying a set of steps that you can find here [1]
that document how to migrate your service broker from v1 to v2 and what is
required in order to persist user data and will get that posted to the
service broker api docs officially.

-Dieu
CF CAPI PM

[1]
https://docs.google.com/document/d/1Pl1o7mxtn3Iayq2STcMArT1cJsKkvi4Ey1-d3TB_Nhs/edit?usp=sharing