It's just the staticfile buildpack from
https://github.com/cloudfoundry/staticfile-buildpack.git, I can try git
cloning it from inside the warden container though

On Wed, Sep 16, 2015 at 9:30 AM, Mike Dalessio <mdalessio(a)pivotal.io>
wrote:

Worth noting that the git repo also needs to allow anonymous access. If
it's a private repo, then the 'git clone' is going to fail.

Can you verify that you can download the buildpack from your repo without
authenticating?

On Tue, Sep 15, 2015 at 7:43 PM, CF Runtime <cfruntime(a)gmail.com> wrote:

It's not something we've ever seen before.

In theory, the warden container needs the git binary, which I think it
gets from the cflinuxfs2 stack; and internet access to wherever the git
repo lives.

If the warden container has both of those things, I can't think of any
reason why it wouldn't work.

Joseph
OSS Release Integration Team

On Tue, Sep 15, 2015 at 2:06 PM, kyle havlovitz <kylehav(a)gmail.com>
wrote:

I tried deploying via uploading a buildpack to the CC (had to set up
nginx first, I didnt have it running/configured before) and that worked! So
that's awesome, but I'm not sure what the problem with using a remote
buildpack is. Even with nginx, I still get the exact same error as before
when pushing using a remote buildpack from git.

On Tue, Sep 15, 2015 at 6:57 AM, CF Runtime <cfruntime(a)gmail.com>
wrote:

Looking at the logs, we can see it finishing downloading the app
package. The next step should be to download and run the buildpack. Since
you mention there is no output after this, I'm guessing it doesn't get that
far.

It might be having trouble downloading the buildpack from the remote
git url. Could you try uploading the buildpack to Cloud Controller and then
having it use that buildpack to see if that makes a difference?


http://apidocs.cloudfoundry.org/217/buildpacks/creates_an_admin_buildpack.html

http://apidocs.cloudfoundry.org/217/buildpacks/upload_the_bits_for_an_admin_buildpack.html

Joseph
OSS Release Integration Team

On Mon, Sep 14, 2015 at 5:37 PM, kyle havlovitz <kylehav(a)gmail.com>
wrote:

Here's the full dea_ng and warden debug logs:
https://gist.github.com/MrEnzyme/6dcc74174482ac62c1cf

Are there any other places I should look for logs?

On Mon, Sep 14, 2015 at 8:14 PM, CF Runtime <cfruntime(a)gmail.com>
wrote:

That's not an error we normally get. It's not clear if the
staging_info.yml error is the source of the problem or an artifact of it.
Having more logs would allow us to speculate more.

Joseph & Dan
OSS Release Integration Team

On Mon, Sep 14, 2015 at 2:24 PM, kyle havlovitz <kylehav(a)gmail.com>
wrote:

I have the cloudfoundry components built, configured and running on
one VM (not in BOSH), and when I push an app I'm getting a generic 'FAILED
StagingError' message after '-----> Downloaded app package (460K)'.

There's nothing in the logs for the dea/warden that seems suspect
other than these 2 things:

{

"timestamp": 1441985105.8883495,

"message": "Exited with status 1 (35.120s):

[[\"/opt/cloudfoundry/warden/warden/src/closefds/closefds\",
\"/opt/cloudfoundry/warden/warden/src/closefds/closefds\"],
\"/var/warden/containers/18vf956il5v/bin/iomux-link\", \"-w\",
\"/var/warden/containers/18vf956il5v/jobs/8/cursors\",
\"/var/warden/containers/18vf956il5v/jobs/8\"]",

"log_level": "warn",

"source": "Warden::Container::Linux",

"data": {

"handle": "18vf956il5v",

"stdout": "",

"stderr": ""

},

"thread_id": 69890836968240,

"fiber_id": 69890849112480,

"process_id": 17063,

"file":

"/opt/cloudfoundry/warden/warden/lib/warden/container/spawn.rb",

"lineno": 135,

"method": "set_deferred_success"

}

{

"timestamp": 1441985105.94083,

"message": "Exited with status 23 (0.023s):

[[\"/opt/cloudfoundry/warden/warden/src/closefds/closefds\",
\"/opt/cloudfoundry/warden/warden/src/closefds/closefds\"], \"rsync\",
\"-e\", \"/var/warden/containers/18vf956il5v/bin/wsh --socket
/var/warden/containers/18vf956il5v/run/wshd.sock --rsh\", \"-r\", \"-p\",
\"--links\", \"vcap(a)container:/tmp/staged/staging_info.yml\",
\"/tmp/dea_ng/staging/d20150911-17093-1amg6y8\"]",

"log_level": "warn",

"source": "Warden::Container::Linux",

"data": {

"handle": "18vf956il5v",

"stdout": "",

"stderr": "rsync: link_stat

\"/tmp/staged/staging_info.yml\" failed: No such file or directory
(2)\nrsync error: some files/attrs were not transferred (see previous
errors) (code 23) at main.c(1655) [Receiver=3.1.0]\nrsync: [Receiver] write
error: Broken pipe (32)\n"

},

"thread_id": 69890836968240,

"fiber_id": 69890849112480,

"process_id": 17063,

"file":

"/opt/cloudfoundry/warden/warden/lib/warden/container/spawn.rb",

"lineno": 135,

"method": "set_deferred_success"

}


And I think the second error is just during cleanup, only failing
because the staging process didn't get far enough in to create the
'staging_info.yml'. The one about iomux-link exiting with status 1 is
pretty mysterious though and I have no idea what caused it. Does anyone
know why this might be happening?

Thanks for the feedback Mike!

Can you tell us more specifically what sort of extensions you need? It
would be great if cf-deployment provided an interface that could serve the
needs of essentially all operators of CF.

Thanks,
Amit

On Tue, Sep 15, 2015 at 4:02 PM, Mike Youngstrom <youngm(a)gmail.com> wrote:

This is great stuff! My organization currently maintains our own custom
ways to generate manifests, include secure properties, and manage release
versions.

We would love to base the next generation of our solution on
cf-deployment. Have you put any thought into how others might customize or
extend cf-deployment? Our needs are very similar to yours just sometimes a
little different.

Perhaps a private fork periodically merged with a known good release
combination (tag) might be appropriate? Or some way to include the same
tools into a wholly private repo?

Mike


On Tue, Sep 8, 2015 at 1:22 PM, Amit Gupta <agupta(a)pivotal.io> wrote:

Hi all,

The CF OSS Release Integration team (casually referred to as the "MEGA
team") is trying to solve a lot of tightly interrelated problems, and make
many of said problems less interrelated. It is difficult to address just
one issue without touching the others, so the following proposal addresses
several issues, but the most important ones are:

* decompose cf-release into many independently manageable, independently
testable, independently usable releases
* separate manifest generation strategies from the release source,
paving the way for Diego to be part of the standard deployment

This proposal will outline a picture of how manifest generation will
work in a unified manner in development, test, and integration
environments. It will also outline a picture of what each release’s test
pipelines will look like, how they will feed into a common integration
environment, and how feedback from the integration environment will feed
back into the test environments. Finally, it will propose a picture for
what the integration environment will look like, and how we get from the
current integration environment to where we want to be.

For further details, please feel free to view and comment here:


https://docs.google.com/document/d/1Viga_TzUB2nLxN_ILqksmUiILM1hGhq7MBXxgLaUOkY

Thanks,
Amit, CF OSS Release Integration team

+1 We'd love a metadata service as well when the dust settles.

Mike

On Wed, Sep 16, 2015 at 1:17 PM, Christopher B Ferris <chrisfer(a)us.ibm.com

wrote:

Onsi,

I can tell you that we (IBM) would welcome a secure metadata service for
retrieving credentials, etc. I'm sure we'd help make that happen.

Cheers,

Christopher Ferris
IBM Distinguished Engineer, CTO Open Technology
IBM Cloud, Open Technologies
email: chrisfer(a)us.ibm.com
twitter: @christo4ferris
blog: http://thoughtsoncloud.com/index.php/author/cferris/
phone: +1 508 667 0402



----- Original message -----
From: Onsi Fakhouri <ofakhouri(a)pivotal.io>
To: "Discussions about Cloud Foundry projects and the system overall." <
cf-dev(a)lists.cloudfoundry.org>
Cc:
Subject: [cf-dev] Re: Environment variable names (was RE: Environment
variable changes in DIEGO)
Date: Wed, Sep 16, 2015 11:40 AM


Would if we could! Environment variables are part of the surface area
that constitutes the contract between the platform and the application.
While we could have duplicates with CF_* and eventually deprecate VCAP_*
the new runtime needs to support droplets staged by the legacy runtime
seamlessly.

Once the dust settles I'd sooner see us step back and reconsider this
particular abstraction. For example, instead of env vars an authenticated
metadata service a la AWS's http://169.254.169.254/ would give us more
flexibility, allow us to dynamically control metadata made available to
apps, and even version metadata sanely.

Onsi

Sent from my iPad

On Sep 16, 2015, at 8:30 AM, john mcteague <john.mcteague(a)gmail.com>
wrote:


On a related, but slightly off, topic, while renaming the VCAP_* vars
would have a big impact, is it not time we thought about renaming these to
CF_* ?

John.
On 16 Sep 2015 16:09, "Matthew Sykes" <matthew.sykes(a)gmail.com> wrote:

The changes, in general, were intentional. The `application_uris` data
was always broken as it didn't reflect route changes. I can't speak
directly to the time stamp data.

The host is present still so I don't know why you don't see it.

We also have a migration guide [1]. If you think additional information
is needed there, pull requests would be welcome.

[1]:
https://github.com/cloudfoundry-incubator/diego-design-notes/blob/master/migrating-to-diego.md

On Wed, Sep 16, 2015 at 10:19 AM, Jack Cai <greensight(a)gmail.com> wrote:

I notice the below changes in the environment variables of DIEGO:
1. VCAP_APP_HOST & VCAP_APP_PORT are removed.
2. These fields are removed from VCAP_APPLICATION value:
application_uris, started_at, start, started_at_timestamp, host,
state_timestamp

I suppose #1 is intentional. How about #2?

Jack




--
Matthew Sykes
matthew.sykes(a)gmail.com

Thanks Dmitriy,

Just for clarity, are you saying multiple instances of a VM cannot share a
single shared filesystem?

On Wed, Sep 16, 2015 at 6:59 PM, Dmitriy Kalinin <dkalinin(a)pivotal.io>
wrote:

BOSH allocates a persistent disk per instance. It never shares persistent
disks between multiple instances at the same time.

If you need a shared file system, you will have to use some kind of a
release for it. It's not any different from what people do with nfs
server/client.

On Wed, Sep 16, 2015 at 7:09 AM, Amit Gupta <agupta(a)pivotal.io> wrote:

The shared file system aspect is an interesting wrinkle to the problem.
Unless you use some network layer to how you write to the shared file
system, e.g. SSHFS, I think apps will not work because they get isolated to
run in a container, they're given a chroot "jail" for their file system,
and it gets blown away whenever the app is stopped or restarted (which will
commonly happen, e.g. during a rolling deploy of the container-runner VMs).

Do you have something that currently works? How do your VMs currently
access this shared FS? I'm not sure BOSH has the abstractions for choosing
a shared, already-existing "persistent disk" to be attached to multiple
VMs. I also don't know what happens when you scale your VMs down, because
BOSH would generally destroy the associated persistent disk, but you don't
want to destroy the shared data.

Dmitriy, any idea how BOSH can work with a shared filesystem (e.g. HDFS)?

Amit

On Wed, Sep 16, 2015 at 6:54 AM, Kayode Odeyemi <dreyemi(a)gmail.com>
wrote:

On Wed, Sep 16, 2015 at 3:44 PM, Amit Gupta <agupta(a)pivotal.io> wrote:

Are the spark jobs tasks that you expect to end, or apps that you
expect to run forever?

They are tasks that run forever. The jobs are subscribers to RabbitMQ
queues that process
messages in batches.

Do your jobs need to write to the file system, or do they access a
shared/distributed file system somehow?

The jobs write to shared filesystem.

Do you need things like a static IP allocated to your jobs?

No.

Are your spark jobs serving any web traffic?

No.

----- Original message -----
From: Onsi Fakhouri <ofakhouri@...>
To: "Discussions about Cloud Foundry projects and the system overall." <cf-dev@...>
Cc:
Subject: [cf-dev] Re: Environment variable names (was RE: Environment variable changes in DIEGO)
Date: Wed, Sep 16, 2015 11:40 AM
 

 

Would if we could!  Environment variables are part of the surface area that constitutes the contract between the platform and the application.  While we could have duplicates with CF_* and eventually deprecate VCAP_* the new runtime needs to support droplets staged by the legacy runtime seamlessly.

 

Once the dust settles I'd sooner see us step back and reconsider this particular abstraction.  For example, instead of env vars an authenticated metadata service a la AWS's http://169.254.169.254/ would give us more flexibility, allow us to dynamically control metadata made available to apps, and even version metadata sanely.

 

Onsi

Sent from my iPad

On Sep 16, 2015, at 8:30 AM, john mcteague <john.mcteague@...> wrote:
 

On a related, but slightly off, topic, while renaming the VCAP_* vars would have a big impact, is it not time we thought about renaming these to CF_*  ?

John.

On 16 Sep 2015 16:09, "Matthew Sykes" <matthew.sykes@...> wrote:

The changes, in general, were intentional. The `application_uris` data was always broken as it didn't reflect route changes. I can't speak directly to the time stamp data.

 

The host is present still so I don't know why you don't see it.

 

We also have a migration guide [1]. If you think additional information is needed there, pull requests would be welcome.

 

[1]: https://github.com/cloudfoundry-incubator/diego-design-notes/blob/master/migrating-to-diego.md

 

On Wed, Sep 16, 2015 at 10:19 AM, Jack Cai <greensight@...> wrote:

I notice the below changes in the environment variables of DIEGO:
1. VCAP_APP_HOST & VCAP_APP_PORT are removed.
2. These fields are removed from VCAP_APPLICATION value: application_uris, started_at, start, started_at_timestamp, host, state_timestamp
 

I suppose #1 is intentional. How about #2?
 

Jack
 

 

 

--

Matthew Sykes
matthew.sykes@...

Some of the proposed changes to the Routing API are backward incompatible.
We don't believe anyone is using it yet, as adoption has generally be
blocked on securing connections to Consul, but we'd like to confirm.

Please raise your hand if you're using the routing API.

Thank you!

Shannon Coen
Product Manager, Cloud Foundry
Pivotal, Inc.

On Wed, Sep 9, 2015 at 12:10 PM, Shannon Coen <scoen(a)pivotal.io> wrote:

We currently have two routing APIs in CF.
1. HTTP Routing API in cf-release:
https://github.com/cloudfoundry-incubator/routing-api
2. TCP Routing API in cf-routing-release:
https://github.com/cloudfoundry-incubator/cf-routing-release

The TCP Routing API is quite basic and we want to extend it for high
availability, authentication, etc. However, instead of enhancing the
existing TCP Routing API, we plan to add support for TCP route registration
to the Routing API in cf-release, as it already supports many of the
desired features. We'll get rid of the current API in cf-routing-release
and submodule in the Routing API from cf-release. Eventually we'll move the
Routing API (along with GoRouter and HAProxy) from cf-release into
cf-routing-release and submodule them into cf-release.

This consolidation, along with our not having any API consumer besides
GoRouter yet, gives us the opportunity to consider a common behavior for
routing API endpoints. We welcome your comments in our design doc:


https://docs.google.com/document/d/1v941oy3Y7RRI80gmLfhPZlaahElK_Q0C3pCQewK8Z3g/edit?usp=sharing

Thank you,

Shannon Coen
Product Manager, Cloud Foundry
Pivotal, Inc.

The shared file system aspect is an interesting wrinkle to the problem.
Unless you use some network layer to how you write to the shared file
system, e.g. SSHFS, I think apps will not work because they get isolated to
run in a container, they're given a chroot "jail" for their file system,
and it gets blown away whenever the app is stopped or restarted (which will
commonly happen, e.g. during a rolling deploy of the container-runner VMs).

Do you have something that currently works? How do your VMs currently
access this shared FS? I'm not sure BOSH has the abstractions for choosing
a shared, already-existing "persistent disk" to be attached to multiple
VMs. I also don't know what happens when you scale your VMs down, because
BOSH would generally destroy the associated persistent disk, but you don't
want to destroy the shared data.

Dmitriy, any idea how BOSH can work with a shared filesystem (e.g. HDFS)?

Amit

On Wed, Sep 16, 2015 at 6:54 AM, Kayode Odeyemi <dreyemi(a)gmail.com> wrote:

On Wed, Sep 16, 2015 at 3:44 PM, Amit Gupta <agupta(a)pivotal.io> wrote:

Are the spark jobs tasks that you expect to end, or apps that you expect
to run forever?

They are tasks that run forever. The jobs are subscribers to RabbitMQ
queues that process
messages in batches.

Do your jobs need to write to the file system, or do they access a
shared/distributed file system somehow?

The jobs write to shared filesystem.

Do you need things like a static IP allocated to your jobs?

No.

Are your spark jobs serving any web traffic?

No.

On Wed, Sep 16, 2015 at 4:09 PM, Amit Gupta <agupta(a)pivotal.io> wrote:

How do your VMs currently access this shared FS?

This is the big ???? . What I planned on using is HDFS over http or
SWIFTFS. Would have to see what this gives.

I also don't know what happens when you scale your VMs down, because BOSH

would generally destroy the associated persistent disk

Not until you mentioned this, my understanding is this wasn't going to be
an issue for BOSH.

I don't have something useful yet since I'm still experimenting. Would
push it somewhere once I'm able to package it as a boshrelease.

Worth noting that the git repo also needs to allow anonymous access. If
it's a private repo, then the 'git clone' is going to fail.

Can you verify that you can download the buildpack from your repo without
authenticating?

On Tue, Sep 15, 2015 at 7:43 PM, CF Runtime <cfruntime(a)gmail.com> wrote:

It's not something we've ever seen before.

In theory, the warden container needs the git binary, which I think it
gets from the cflinuxfs2 stack; and internet access to wherever the git
repo lives.

If the warden container has both of those things, I can't think of any
reason why it wouldn't work.

Joseph
OSS Release Integration Team

On Tue, Sep 15, 2015 at 2:06 PM, kyle havlovitz <kylehav(a)gmail.com>
wrote:

I tried deploying via uploading a buildpack to the CC (had to set up
nginx first, I didnt have it running/configured before) and that worked! So
that's awesome, but I'm not sure what the problem with using a remote
buildpack is. Even with nginx, I still get the exact same error as before
when pushing using a remote buildpack from git.

On Tue, Sep 15, 2015 at 6:57 AM, CF Runtime <cfruntime(a)gmail.com> wrote:

Looking at the logs, we can see it finishing downloading the app
package. The next step should be to download and run the buildpack. Since
you mention there is no output after this, I'm guessing it doesn't get that
far.

It might be having trouble downloading the buildpack from the remote
git url. Could you try uploading the buildpack to Cloud Controller and then
having it use that buildpack to see if that makes a difference?


http://apidocs.cloudfoundry.org/217/buildpacks/creates_an_admin_buildpack.html

http://apidocs.cloudfoundry.org/217/buildpacks/upload_the_bits_for_an_admin_buildpack.html

Joseph
OSS Release Integration Team

On Mon, Sep 14, 2015 at 5:37 PM, kyle havlovitz <kylehav(a)gmail.com>
wrote:

Here's the full dea_ng and warden debug logs:
https://gist.github.com/MrEnzyme/6dcc74174482ac62c1cf

Are there any other places I should look for logs?

On Mon, Sep 14, 2015 at 8:14 PM, CF Runtime <cfruntime(a)gmail.com>
wrote:

That's not an error we normally get. It's not clear if the
staging_info.yml error is the source of the problem or an artifact of it.
Having more logs would allow us to speculate more.

Joseph & Dan
OSS Release Integration Team

On Mon, Sep 14, 2015 at 2:24 PM, kyle havlovitz <kylehav(a)gmail.com>
wrote:

I have the cloudfoundry components built, configured and running on
one VM (not in BOSH), and when I push an app I'm getting a generic 'FAILED
StagingError' message after '-----> Downloaded app package (460K)'.

There's nothing in the logs for the dea/warden that seems suspect
other than these 2 things:

{

"timestamp": 1441985105.8883495,

"message": "Exited with status 1 (35.120s):

[[\"/opt/cloudfoundry/warden/warden/src/closefds/closefds\",
\"/opt/cloudfoundry/warden/warden/src/closefds/closefds\"],
\"/var/warden/containers/18vf956il5v/bin/iomux-link\", \"-w\",
\"/var/warden/containers/18vf956il5v/jobs/8/cursors\",
\"/var/warden/containers/18vf956il5v/jobs/8\"]",

"log_level": "warn",

"source": "Warden::Container::Linux",

"data": {

"handle": "18vf956il5v",

"stdout": "",

"stderr": ""

},

"thread_id": 69890836968240,

"fiber_id": 69890849112480,

"process_id": 17063,

"file":

"/opt/cloudfoundry/warden/warden/lib/warden/container/spawn.rb",

"lineno": 135,

"method": "set_deferred_success"

}

{

"timestamp": 1441985105.94083,

"message": "Exited with status 23 (0.023s):

[[\"/opt/cloudfoundry/warden/warden/src/closefds/closefds\",
\"/opt/cloudfoundry/warden/warden/src/closefds/closefds\"], \"rsync\",
\"-e\", \"/var/warden/containers/18vf956il5v/bin/wsh --socket
/var/warden/containers/18vf956il5v/run/wshd.sock --rsh\", \"-r\", \"-p\",
\"--links\", \"vcap(a)container:/tmp/staged/staging_info.yml\",
\"/tmp/dea_ng/staging/d20150911-17093-1amg6y8\"]",

"log_level": "warn",

"source": "Warden::Container::Linux",

"data": {

"handle": "18vf956il5v",

"stdout": "",

"stderr": "rsync: link_stat \"/tmp/staged/staging_info.yml\"

failed: No such file or directory (2)\nrsync error: some files/attrs were
not transferred (see previous errors) (code 23) at main.c(1655)
[Receiver=3.1.0]\nrsync: [Receiver] write error: Broken pipe (32)\n"

},

"thread_id": 69890836968240,

"fiber_id": 69890849112480,

"process_id": 17063,

"file":

"/opt/cloudfoundry/warden/warden/lib/warden/container/spawn.rb",

"lineno": 135,

"method": "set_deferred_success"

}


And I think the second error is just during cleanup, only failing
because the staging process didn't get far enough in to create the
'staging_info.yml'. The one about iomux-link exiting with status 1 is
pretty mysterious though and I have no idea what caused it. Does anyone
know why this might be happening?

On Sep 16, 2015, at 8:30 AM, john mcteague <john.mcteague(a)gmail.com> wrote:

On a related, but slightly off, topic, while renaming the VCAP_* vars would have a big impact, is it not time we thought about renaming these to CF_* ?

John.

On 16 Sep 2015 16:09, "Matthew Sykes" <matthew.sykes(a)gmail.com> wrote:
The changes, in general, were intentional. The `application_uris` data was always broken as it didn't reflect route changes. I can't speak directly to the time stamp data.

The host is present still so I don't know why you don't see it.

We also have a migration guide [1]. If you think additional information is needed there, pull requests would be welcome.

[1]: https://github.com/cloudfoundry-incubator/diego-design-notes/blob/master/migrating-to-diego.md

On Wed, Sep 16, 2015 at 10:19 AM, Jack Cai <greensight(a)gmail.com> wrote:
I notice the below changes in the environment variables of DIEGO:
1. VCAP_APP_HOST & VCAP_APP_PORT are removed.
2. These fields are removed from VCAP_APPLICATION value: application_uris, started_at, start, started_at_timestamp, host, state_timestamp

I suppose #1 is intentional. How about #2?

Jack

--
Matthew Sykes
matthew.sykes(a)gmail.com

The changes, in general, were intentional. The `application_uris` data was
always broken as it didn't reflect route changes. I can't speak directly to
the time stamp data.

The host is present still so I don't know why you don't see it.

We also have a migration guide [1]. If you think additional information is
needed there, pull requests would be welcome.

[1]:
https://github.com/cloudfoundry-incubator/diego-design-notes/blob/master/migrating-to-diego.md

On Wed, Sep 16, 2015 at 10:19 AM, Jack Cai <greensight(a)gmail.com> wrote:

I notice the below changes in the environment variables of DIEGO:
1. VCAP_APP_HOST & VCAP_APP_PORT are removed.
2. These fields are removed from VCAP_APPLICATION value:
application_uris, started_at, start, started_at_timestamp, host,
state_timestamp

I suppose #1 is intentional. How about #2?

Jack

--
Matthew Sykes
matthew.sykes(a)gmail.com

I notice the below changes in the environment variables of DIEGO:
1. VCAP_APP_HOST & VCAP_APP_PORT are removed.
2. These fields are removed from VCAP_APPLICATION value: application_uris,
started_at, start, started_at_timestamp, host, state_timestamp

I suppose #1 is intentional. How about #2?

Jack

On Wed, Sep 16, 2015 at 3:44 PM, Amit Gupta <agupta(a)pivotal.io> wrote:

Are the spark jobs tasks that you expect to end, or apps that you expect
to run forever?

They are tasks that run forever. The jobs are subscribers to RabbitMQ
queues that process
messages in batches.

Do your jobs need to write to the file system, or do they access a
shared/distributed file system somehow?

The jobs write to shared filesystem.

Do you need things like a static IP allocated to your jobs?

No.

Are your spark jobs serving any web traffic?

No.