(No subject)


Arbi Akhina
 

I removed static assets as well as many classes that were not used,
I left classes related to jetty, javax (managment, ..), apache commons io,
and the package (with all classes) where the main class is located.

I didn't try on public cfs like PWS (what is this one?) and BlueMix.

On Thu, Jun 11, 2015 at 4:42 PM, Daniel Mikusa <dmikusa(a)pivotal.io> wrote:

On Thu, Jun 11, 2015 at 10:31 AM, Arbi Akhina <arbi.akhina(a)gmail.com>
wrote:

Hi, I had to reduce the size of the jar to ~ 51M to get the simple class
that runs jetty to work on bosh-lite!!!
Interesting. Out of curiosity what did you remove to make it smaller?
Static assets? Also have you tried a larger CF install like PWS / BlueMix,
just to see if this is an issue specific to bosh-lite.


Is there any kind of size restriction for executable jars??
The system can put a limit on how big of an application you can upload. I
don't know about bosh-lite specifically, but the default is 1G. Also, when
you exceed that you'll get a message from the cf cli saying your app is too
big. It won't silently fail, which seems to be what happened here.

Dan



Here is the system information of the warden container on which the jar
is running:
Java version : 1.8.0_45-
Java Spec version : 1.8
JVM version : 25.45-b02
JVM vendor : Oracle Corporation
JVM name : OpenJDK 64-Bit Server VM
Os Name : Linux
Os Architecture : amd64
Os version : 3.13.0-44-generic
Number of processors : 4
Max memory : 3087007744
Total memory : 3087007744
Available memory : 2958157616
Free work disk space : 17898352640
VM Args : [-Djava.io.tmpdir=/home/vcap/tmp,
-XX:OnOutOfMemoryError=/home/vcap/app/.java-buildpack/open_jdk_jre/bin/killjava.sh,
-Xss1M, -Xmx3G, -Xms3G, -XX:MaxMetaspaceSize=419430K,
-XX:MetaspaceSize=419430K]

On Tue, Jun 9, 2015 at 7:01 PM, Arbi Akhina <arbi.akhina(a)gmail.com>
wrote:

no I meant the source is for the company I work for, not my personal
project.

On Tue, Jun 9, 2015 at 6:58 PM, Daniel Mikusa <dmikusa(a)pivotal.io>
wrote:

On Tue, Jun 9, 2015 at 12:43 PM, Arbi Akhina <arbi.akhina(a)gmail.com>
wrote:

I don't own the source, but here is the jar file
http://tempsend.com/6D745B7B07/7033/modules.jar
you can try to run it locally with java -jar modules.jar then connect
to localhost:8080
Hmm, OK. I thought you had a minimal example from you last note. I'm
not really up for running untrusted binary code. If you can put a minimal
code sample together, I'll give it a try.



An here is the manifest.yml i'm using:
---
applications:
- name: modules
memory: 4G
disk_quota: 2G
timeout: 180
instances: 1
host: modules-${random-word}
path: modules.jar
buildpack: https://github.com/cloudfoundry/java-buildpack
command: sleep 2 &&
CALCULATED_MEMORY=$($PWD/.java-buildpack/open_jdk_jre/bin/java-buildpack-memory-calculator-1.1.1_RELEASE
-memorySizes=metaspace:64m..
-memoryWeights=heap:75,metaspace:10,stack:5,native:10
-totMemory=$MEMORY_LIMIT) && $PWD/.java-buildpack/open_jdk_jre/bin/java -cp
$PWD/. -Djava.io.tmpdir=$TMPDIR
-XX:OnOutOfMemoryError=$PWD/.java-buildpack/open_jdk_jre/bin/killjava.sh
$CALCULATED_MEMORY com.heavenize.osmoze.kernel.HelloHandler
# env:
# JAVA_OPTS: "$JAVA_OPTS -agentlib:jdwp=transport=dt_socket,address=
192.168.2.8:8000"

I'm having both files in the same folder modules/ and I push with cf
push
That seems OK. Is this app designed to run on CF? By that, I mean is
it configure to log to STDOUT / STDERR and is it going to listen on the
right port (i.e $PORT)?

Dan




Thanks a lot

On Tue, Jun 9, 2015 at 6:25 PM, Daniel Mikusa <dmikusa(a)pivotal.io>
wrote:

I'd be happy to try the demo and see if it works for me. Can you
post a complete project on github so I can just `git clone` & build?

Dan

On Tue, Jun 9, 2015 at 12:21 PM, Arbi Akhina <arbi.akhina(a)gmail.com>
wrote:

I've tried with a simple app that just launches an embedded jetty
that listen on the vcap port and it worked (I can connect to the app url
from curl and chrome)
then I moved this single file into my modules.jar (the real app) and
configure it as main class, but here it doesn't work.

here is the class in question:
public class HelloHandler extends AbstractHandler
{
public static void main(String[] args)
{
int VCAP_APP_PORT =
Integer.parseInt((System.getenv("VCAP_APP_PORT") != null ?
System.getenv("VCAP_APP_PORT") : "8080"));

Server server = new Server(VCAP_APP_PORT);
server.setHandler(new HelloHandler());

try
{
server.start();
server.join();
}
catch (Exception exception)
{
System.out.println("Failed to start embedded jetty server.");
exception.printStackTrace();
}
}

final String greeting;
final String body;

public HelloHandler()
{
this("Hello World");
}

public HelloHandler( String greeting )
{
this(greeting, null);
}

public HelloHandler( String greeting, String body )
{
this.greeting = greeting;
this.body = body;
}

public void handle( String target,
Request baseRequest,
HttpServletRequest request,
HttpServletResponse response ) throws
IOException,
ServletException
{
response.setContentType("text/html; charset=utf-8");
response.setStatus(HttpServletResponse.SC_OK);

PrintWriter out = response.getWriter();

out.println("<h1>" + greeting + "</h1>");
out.println("<p>Current directory: " + getCurrentDirectory() +
"</p>");
out.println("<p> Parent directory: " + getParentDirectory() +
"</p>");
for(File file: getFiles()) {
out.println("<p>" + file.getAbsolutePath() + "</p>");
}

if (body != null)
{
out.println(body);
}

baseRequest.setHandled(true);
}
private List<File> getFiles() {
return Arrays.asList(new File(".").listFiles());
}

private String getCurrentDirectory()
{
String directory = null;
try
{
directory = new File(".").getCanonicalPath();
}
catch (Exception e)
{
e.printStackTrace();
}
return directory;
}

private String getParentDirectory()
{
String directory = null;
try
{
directory = new File("..").getCanonicalPath();
}
catch (Exception e)
{
e.printStackTrace();
}
return directory;
}
}


On Tue, Jun 9, 2015 at 5:34 PM, Daniel Mikusa <dmikusa(a)pivotal.io>
wrote:

Double check that your app is configured to log to STDOUT / STDER.
If it's logging to a file, you won't see the messages since the system only
captures STDOUT & STDERR.

Also, you might try a demo app like spring-music, just to make sure
your install is working OK.

https://github.com/cloudfoundry-samples/spring-music

Dan

On Tue, Jun 9, 2015 at 10:40 AM, Arbi Akhina <arbi.akhina(a)gmail.com
wrote:
Same logs I still can't see what went wrong!

On Tue, Jun 9, 2015 at 3:58 PM, Daniel Mikusa <dmikusa(a)pivotal.io>
wrote:

And so you'd want to set `command` to `sleep 2 &&
CALCULATED_MEMORY=$($PWD/.java-buildpack/open_jdk_jre/
bin/java-buildpack-memory-calculator-1.1.1_RELEASE
-memorySizes=metaspace:64m.. -memoryWeights=heap:75,metaspace:10,stack:5,native:10
-totMemory=$MEMORY_LIMIT) && $PWD/.java-buildpack/open_jdk_jre/bin/java
-cp $PWD/. -Djava.io.tmpdir=$TMPDIR -XX:OnOutOfMemoryError=$PWD/.
java-buildpack/open_jdk_jre/bin/killjava.sh $CALCULATED_MEMORY
-agentlib:jdwp=transport=dt_socket,address=192.168.2.8:8000
com.heavenize.osmoze.kernel.HelloHandler`.

Dan

On Tue, Jun 9, 2015 at 9:29 AM, Arbi Akhina <
arbi.akhina(a)gmail.com> wrote:

Here is the content of *detected_start_command*:
$ CF_TRACE=true cf app modules | grep "detected_start_command"
"detected_start_command":
"CALCULATED_MEMORY=$($PWD/.java-buildpack/open_jdk_jre/bin/java-buildpack-memory-calculator-1.1.1_RELEASE
-memorySizes=metaspace:64m..
-memoryWeights=heap:75,metaspace:10,stack:5,native:10
-totMemory=$MEMORY_LIMIT) && $PWD/.java-buildpack/open_jdk_jre/bin/java -cp
$PWD/. -Djava.io.tmpdir=$TMPDIR
-XX:OnOutOfMemoryError=$PWD/.java-buildpack/open_jdk_jre/bin/killjava.sh
$CALCULATED_MEMORY -agentlib:jdwp=transport=dt_socket,address=
192.168.2.8:8000 com.heavenize.osmoze.kernel.HelloHandler",

{"guid":"2dabf3c6-1736-4eb4-9bb7-40dc58bce246","name":"modules","routes":[{"guid":"014f77ed-e450-4209-be23-40e4a8257cfc","host":"modules-rhythmic-uvulatomy","domain":{"guid":"a77afa99-76d6-4f3f-9e07-27775597709f","name":"
10.244.0.34.xip.io
"}}],"running_instances":0,"services":[],"available_domains":[{"guid":"a77afa99-76d6-4f3f-9e07-27775597709f","name":"
10.244.0.34.xip.io
"}],"name":"modules","production":false,"space_guid":"381707f7-88d4-4f6e-bd7c-80d7f0699b0f","stack_guid":"3431865a-f165-4e75-9221-4f418e9de889","buildpack":"
https://github.com/cloudfoundry/java-buildpack","detected_buildpack":null,"environment_json":{"JAVA_OPTS":"$JAVA_OPTS
-agentlib:jdwp=transport=dt_socket,address=192.168.2.8:8000"},"memory":4096,"instances":1,"disk_quota":2048,"state":"STARTED","version":"e8f0e18e-7c09-4143-a17c-83a6e32eed84","command":null,"console":false,"debug":null,"staging_task_id":"1249276465c64c1884d401452d2365af","package_state":"STAGED","health_check_type":"port","health_check_timeout":180,"staging_failed_reason":null,"diego":false,"docker_image":null,"package_updated_at":"2015-06-08T07:56:44Z","detected_start_command":"CALCULATED_MEMORY=$($PWD/.java-buildpack/open_jdk_jre/bin/java-buildpack-memory-calculator-1.1.1_RELEASE
-memorySizes=metaspace:64m..
-memoryWeights=heap:75,metaspace:10,stack:5,native:10
-totMemory=$MEMORY_LIMIT) && $PWD/.java-buildpack/open_jdk_jre/bin/java -cp
$PWD/. -Djava.io.tmpdir=$TMPDIR
-XX:OnOutOfMemoryError=$PWD/.java-buildpack/open_jdk_jre/bin/killjava.sh
$CALCULATED_MEMORY -agentlib:jdwp=transport=dt_socket,address=
192.168.2.8:8000 com.heavenize.osmoze.kernel.HelloHandler"}


On Tue, Jun 9, 2015 at 1:35 PM, Daniel Mikusa <
dmikusa(a)pivotal.io> wrote:

On Tue, Jun 9, 2015 at 5:05 AM, Arbi Akhina <
arbi.akhina(a)gmail.com> wrote:

Hi, thanks for the hint, I tried to add to the manifest the
following entry:
command: sleep 2 && java -jar modules.jar
The Java build pack does not put Java on the path, which is
what you're seeing in the error below. I will usually run `CF_TRACE=true
cf app <app-name> | grep "detected_start_command"` which shows the command
the build pack detected and then just copy & paste that into my custom
command.



but it looks like it's not appropriate as I see in the logs:
2015-06-08T05:34:03.71+0200 [App/0] ERR bash: java:
command not found

On Mon, Jun 8, 2015 at 6:32 PM, Daniel Mikusa <
dmikusa(a)pivotal.io> wrote:

On Mon, Jun 8, 2015 at 12:03 PM, Arbi Akhina <
arbi.akhina(a)gmail.com> wrote:

I'm trying to push an executable JAR to a bosh-lite
instance, on the logs I see CF trying many times to restart the app and
eventually fail.

I can't find out why the app crashes as there is no app logs
returned by CF. I tried to remotely debug the app (as described in [1]) but
nothing happens on eclipse. Any hint to solve this issue is appreciated.
There's a known issue that occurs when an app starts and
fails in rapid succession and results in the log entries being missed. If
you add a couple second pause into the app, it will give the system enough
time to attach the logger and you'll see the output generated by the
crashing app.

You can do this with a custom start command `cf push -c
'sleep 2 && <normal-cmd>'` or with a `.profile.d` script that sleeps for a
couple seconds.

Dan




1. I'm launching the app with:
*cf push -t 180*

2. Here is the manifest.yml content:
---
applications:
- name: modules
memory: 4G
instances: 1
host: modules-${random-word}
path: modules.jar
buildpack: https://github.com/cloudfoundry/java-buildpack
env:
JAVA_OPTS: "$JAVA_OPTS
-agentlib:jdwp=transport=dt_socket,address=192.168.2.8:8000"

3. Here is the log:
Connecté, le dumping journaux récents pour application
modules en org heavenize / espace dev comme admin...

2015-06-07T12:05:15.69+0200 [API] OUT Created app with
guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:05:16.10+0200 [API] OUT Updated app with
guid 1fbf3378-6512-46de-bae4-02ee30275464
({"route"=>"68e27d8d-4ff6-443b-a3e0-416c40d325d3"})
2015-06-07T12:12:21.55+0200 [DEA] OUT Got staging
request for app with id 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:12:25.40+0200 [API] OUT Updated app with
guid 1fbf3378-6512-46de-bae4-02ee30275464 ({"state"=>"STARTED"})
2015-06-07T12:12:38.08+0200 [STG] OUT -----> Downloaded
app package (163M)
2015-06-07T12:13:40.93+0200 [STG] ERR Cloning into
'/tmp/buildpacks/java-buildpack'...
2015-06-07T12:14:01.67+0200 [STG] OUT -----> Java
Buildpack Version: c862ac8 |
https://github.com/cloudfoundry/java-buildpack#c862ac8
2015-06-07T12:14:13.89+0200 [STG] OUT -----> Downloading
Open Jdk JRE 1.8.0_45 from
https://download.run.pivotal.io/openjdk/lucid/x86_64/openjdk-1.8.0_45.tar.gz
(11.6s)
2015-06-07T12:14:15.34+0200 [STG] OUT Expanding
Open Jdk JRE to .java-buildpack/open_jdk_jre (1.4s)
2015-06-07T12:14:15.89+0200 [STG] OUT -----> Downloading
Open JDK Like Memory Calculator 1.1.1_RELEASE from
https://download.run.pivotal.io/memory-calculator/lucid/x86_64/memory-calculator-1.1.1_RELEASE
(0.5s)
2015-06-07T12:14:15.90+0200 [STG] OUT Memory
Settings: -XX:MaxMetaspaceSize=419430K -XX:MetaspaceSize=419430K -Xss1M
-Xmx3G -Xms3G
2015-06-07T12:15:43.44+0200 [STG] OUT -----> Uploading
droplet (151M)
2015-06-07T12:16:07.51+0200 [DEA] OUT Starting app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:16:29.66+0200 [API] OUT App instance
exited with guid 1fbf3378-6512-46de-bae4-02ee30275464 payload:
{"cc_partition"=>"default",
"droplet"=>"1fbf3378-6512-46de-bae4-02ee30275464",
"version"=>"703f793c-e388-41ac-b1f1-c564b301ca70",
"instance"=>"9d55c5f791324d358bffb4c961a4c7ee", "index"=>0,
"reason"=>"CRASHED", "exit_status"=>-1, "exit_description"=>"failed to
start", "crash_timestamp"=>1433672189}
2015-06-07T12:17:14.18+0200 [DEA] OUT Starting app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:17:31.10+0200 [API] OUT App instance
exited with guid 1fbf3378-6512-46de-bae4-02ee30275464 payload:
{"cc_partition"=>"default",
"droplet"=>"1fbf3378-6512-46de-bae4-02ee30275464",
"version"=>"703f793c-e388-41ac-b1f1-c564b301ca70",
"instance"=>"2ae0c26f33864f40989ee870a1b9e3db", "index"=>0,
"reason"=>"CRASHED", "exit_status"=>-1, "exit_description"=>"failed to
start", "crash_timestamp"=>1433672251}
2015-06-07T12:17:38.48+0200 [DEA] OUT Removing crash for
app with id 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:17:38.48+0200 [DEA] OUT Stopping app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:17:38.48+0200 [DEA] OUT Stopped app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:17:55.31+0200 [DEA] OUT Starting app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:18:11.82+0200 [API] OUT App instance
exited with guid 1fbf3378-6512-46de-bae4-02ee30275464 payload:
{"cc_partition"=>"default",
"droplet"=>"1fbf3378-6512-46de-bae4-02ee30275464",
"version"=>"703f793c-e388-41ac-b1f1-c564b301ca70",
"instance"=>"dc872d38f3324af481c82ba67f0e216c", "index"=>0,
"reason"=>"CRASHED", "exit_status"=>-1, "exit_description"=>"failed to
start", "crash_timestamp"=>1433672291}
2015-06-07T12:18:18.69+0200 [DEA] OUT Removing crash for
app with id 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:18:18.69+0200 [DEA] OUT Stopping app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:18:18.69+0200 [DEA] OUT Stopped app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:18:34.06+0200 [DEA] OUT Starting app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:18:50.98+0200 [API] OUT App instance
exited with guid 1fbf3378-6512-46de-bae4-02ee30275464 payload:
{"cc_partition"=>"default",
"droplet"=>"1fbf3378-6512-46de-bae4-02ee30275464",
"version"=>"703f793c-e388-41ac-b1f1-c564b301ca70",
"instance"=>"623c3af7e3e84801b6fd44eeee9c0a12", "index"=>0,
"reason"=>"CRASHED", "exit_status"=>-1, "exit_description"=>"failed to
start", "crash_timestamp"=>1433672330}
2015-06-07T12:18:58.80+0200 [DEA] OUT Removing crash for
app with id 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:18:58.80+0200 [DEA] OUT Stopping app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:18:58.80+0200 [DEA] OUT Stopped app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:19:34.08+0200 [DEA] OUT Starting app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:19:50.36+0200 [API] OUT App instance
exited with guid 1fbf3378-6512-46de-bae4-02ee30275464 payload:
{"cc_partition"=>"default",
"droplet"=>"1fbf3378-6512-46de-bae4-02ee30275464",
"version"=>"703f793c-e388-41ac-b1f1-c564b301ca70",
"instance"=>"40727eea293146948af197e13443843c", "index"=>0,
"reason"=>"CRASHED", "exit_status"=>-1, "exit_description"=>"failed to
start", "crash_timestamp"=>1433672390}
2015-06-07T12:19:59.01+0200 [DEA] OUT Removing crash for
app with id 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:19:59.01+0200 [DEA] OUT Stopping app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:19:59.01+0200 [DEA] OUT Stopped app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:21:04.12+0200 [DEA] OUT Starting app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:21:20.61+0200 [API] OUT App instance
exited with guid 1fbf3378-6512-46de-bae4-02ee30275464 payload:
{"cc_partition"=>"default",
"droplet"=>"1fbf3378-6512-46de-bae4-02ee30275464",
"version"=>"703f793c-e388-41ac-b1f1-c564b301ca70",
"instance"=>"f7ffff55692a418c847f4f37be574ddf", "index"=>0,
"reason"=>"CRASHED", "exit_status"=>-1, "exit_description"=>"failed to
start", "crash_timestamp"=>1433672480}
2015-06-07T12:21:29.43+0200 [DEA] OUT Removing crash for
app with id 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:21:29.43+0200 [DEA] OUT Stopping app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:21:29.47+0200 [DEA] OUT Stopped app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:23:34.16+0200 [DEA] OUT Starting app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:23:49.97+0200 [API] OUT App instance
exited with guid 1fbf3378-6512-46de-bae4-02ee30275464 payload:
{"cc_partition"=>"default",
"droplet"=>"1fbf3378-6512-46de-bae4-02ee30275464",
"version"=>"703f793c-e388-41ac-b1f1-c564b301ca70",
"instance"=>"4581e97c6b0f4504b8d64a5c69d6787b", "index"=>0,
"reason"=>"CRASHED", "exit_status"=>-1, "exit_description"=>"failed to
start", "crash_timestamp"=>1433672629}
2015-06-07T12:23:50.29+0200 [DEA] OUT Removing crash for
app with id 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:23:50.29+0200 [DEA] OUT Stopping app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:23:50.29+0200 [DEA] OUT Stopped app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:28:14.24+0200 [DEA] OUT Starting app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:28:29.82+0200 [API] OUT App instance
exited with guid 1fbf3378-6512-46de-bae4-02ee30275464 payload:
{"cc_partition"=>"default",
"droplet"=>"1fbf3378-6512-46de-bae4-02ee30275464",
"version"=>"703f793c-e388-41ac-b1f1-c564b301ca70",
"instance"=>"f98749490a6743598f57d3848eb06177", "index"=>0,
"reason"=>"CRASHED", "exit_status"=>-1, "exit_description"=>"failed to
start", "crash_timestamp"=>1433672909}
2015-06-07T12:28:31.73+0200 [DEA] OUT Removing crash for
app with id 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:28:31.73+0200 [DEA] OUT Stopping app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:28:31.73+0200 [DEA] OUT Stopped app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464



[1]
http://docs.cloudfoundry.org/buildpacks/java/java-tips.html#debugging



_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev