Date   

max length with Dropped log message: message too long (>64K ...)

Koper, Dies <diesk@...>
 

Hi,

When my app outputs more than 64K of text, I get a message "Dropped log message: message too long (>64K without a newline).
When my app outputs much less than 64K of text, I get the app's output.
When my app outputs just under 64K of text, nothing is output: neither the error message, nor the app's output.

What is exactly the limit under which output is still guaranteed, and can we update the error message accordingly?

I tried to read the source code. Not sure if the limit is because messages are sent over UDP (UDP max data length is 65,507), or due to other data that is included in the transmission of messages (timestamp, app id, etc.).

Cheers,
Dies Koper


Post: Cloud Foundry Advisory Board Meeting Notes - 2015 August

Phil Whelan
 

Hi,

I've just posted a write-up of last week's Cloud Foundry Advisory Board
Meeting.
http://www.activestate.com/blog/2015/08/cloud-foundry-advisory-board-meeting-2015-august

Topics include...
- Foundation Update
- Diego
- BOSH
- MEGA
- Loggregator
- CAPI
- CLI
- Services Core
- Lattice
- Routing
- Garden
- Greenhouse
- UAA
- Abacus
- "Super Important Question"
- Cloud Foundry v3
- Slack

Please let me know if you spot any inaccuracies.

Thanks,
Phil


Using s3 for blobstore bucket prefix

Rich Wohlstadter
 

Hi,

I'm trying to setup so that cloud controller uses s3 for blobstore in AWS. I can get it working by supplying bucket names in the relevant key fields(resource_directory_key,etc). Want I wanted to know is if there is anyway we can supply prefix's. For instance, I want to use the same bucket for droplets, buildpacks, etc and organize them as so:

mybucket/prod/cc-buildpacks for buildpacks
mybucket/prod/cc-droplets for droplets
mybucket/prod/cc-packages for packages
mybucket/prod/cc-resources for resources

Just cant find a way to specify the prefix's so I don't have to create 4 buckets. Any way to do this?

Thanks,

Rich


Re: Script hangs when updating the cf-release

CF Runtime
 

Hmm, we're not seeing any problems like that, and really we're just running
some git commands.

I would assume that running the script does create a lot of connections in
a short amount of time. Is it possible part of your network infrastructure
is seeing this as some sort of attack and blocking connections after a
certain number is reached?

Joseph
OSS Release Integration Team

On Thu, Aug 20, 2015 at 6:36 AM, Roberto Jimenez Sanchez <roberto(a)de.ibm.com
wrote:
Hi,

I am having exactly the same issue using develop branch and latest commit:

$ git pull origin develop
From https://github.com/cloudfoundry/cf-release
* branch develop -> FETCH_HEAD
Already up-to-date.
$ git checkout develop
M src/etcd-release
M src/github.com/cloudfoundry/cf-acceptance-tests
M src/hm9000
M src/loggregator
M src/smoke-tests
M src/uaa
M src/warden
Already on 'develop'
Your branch is up-to-date with 'origin/develop'.

[roberto(a)oc4828837525 cf-release]$ ./update
....(output ommitted)
Submodule path 'src/etcd-metrics-server': checked out
'90c444c7f93cacb998e45c46f1e06ecf4c8eb9c4'
Submodule path 'src/etcd-release': checked out
'2a1cbbf8772f7c2887ac3b8a75b1114dbcd9d9d6'
Submodule path 'src/etcd-release/src/
github.com/cloudfoundry-incubator/candiedyaml': checked out
'29b4d9cda9fd156adea631d790c95d37ee6ab8e6'
Cloning into 'src/github.com/cloudfoundry-incubator/cf-test-helpers'...
^C
[roberto(a)oc4828837525 cf-release]$ Failed to recurse into submodule path
'src/etcd-release'


Re: Overcommit on Diego Cells

Will Pragnell <wpragnell@...>
 

Apparently my last reply to this thread never made it through. Hope this
one does!

Mike, you're right that there are currently no btrfs metrics being emitted
from garden-linux. There are currently no immediate plans to implement
this, but clearly such metrics are useful, so I'll raise this with the team
and see where we land.

As for your question about btrfs freeing disk space, I'm afraid I don't
know off hand. I'll have to do some investigation and get back to you on
that next week.

On 19 August 2015 at 23:46, Mike Youngstrom <youngm(a)gmail.com> wrote:

Thanks for the response Eric. It was very helpful.

One last question. Any thoughts on what would be the best way to monitor
free ephemeral disk space in my overcommitted situation? If using
btrfs_store_size_mb=-1 will btrfs free ephemeral disk space when less is
being used or does it just grow when it needs more? Looking at firehose
stats in 1398 I don't see any btrfs usage metrics being sent from
garden-linux.

Thanks,
Mike

On Mon, Aug 17, 2015 at 9:14 PM, Eric Malm <emalm(a)pivotal.io> wrote:

Hi, Mike,

Apologies, I emailed this to cf-dev a few days ago, but it seems not to
have gone through. Anyway, thanks for asking about the different
configuration values Diego exposes for disk and memory. Yes, you can use
the 'diego.executor.memory_capacity_mb' and
'diego.executor.disk_capacity_mb' properties to specify overcommits in
absolute terms rather than the relative factors configurable on the DEAs.
The cell reps will advertise those values as their maximum memory and disk
capacity, and subtract memory and disk for allocated containers when
reporting their available capacity during auctions.

The 'btrfs_store_size_mb' property on garden-linux is more of a moving
target as garden-linux settles in on that filesystem as a backing store. As
of garden-linux-release 0.292.0, which diego-release 0.1412.0 and later
consume, that property accepts a '-1' value that allows it to grow up to
the full size of the available disk on the /var/vcap/data ephemeral disk
volume. The btrfs volume itself is sparse, so it will start at effectively
zero size and grow as needed to accommodate the container layers. Since
you're already monitoring disk usage on your VMs carefully and scaling out
when you hit certain limits, this might be a good option for you. This is
also effectively how the DEAs operate today, without an explicit limit on
the total amount of disk they allocate for containers.

If you do want more certainty in the maximum size that the garden-linux
btrfs volume will grow to, or if you're on a version of diego-release
earlier than 0.1412.0, you should set btrfs_store_size_mb to a positive
value, and garden-linux will create the volume to grow only up to that
size. One strategy to determine that value would be to use the maximum size
of the ephemeral disk, less the size of the BOSH-deployed packages (for the
executor, currently around 1.3 GB, including the untarred cflinuxfs2
rootfs), less the size allocated to the executor cache in the
'diego.executor.max_cache_size_in_bytes' property (which currently defaults
to 10GB).

Best,
Eric


Re: I'm getting different x_forwarded_for in my Gorouter access logs depending on what browser/cli-tool I use.

Simon Johansson <simon@...>
 

Chris, I now know why this is happening.

In ServeHTTP in reverseproxy in the httputil package[1] we make a copy of
the incoming request for the outgoing request[2]. We then check[3] if the
incoming request have any "Hop-by-hop"[4] headers are set.

If they are set we assign the outgoing requests headers to a new Header
object[5], deep copy all the headers into it from the incoming request and
delete the Hop-by-hop headers.

After this we update the X-Forwarded-For header[6].

This effectively means that if the incoming request doesn't container any
of the hop-by-hop headers we will in step [6] update the incoming requests
X-Forwarded-For header as outrec.Header is a reference to rec.Header. But
in the case we do have a hop-by-hop header the inrec.Headers,
outrec.Headers will not be a reference and thus we get different logs when
we enter the defer in gorouter/proxy/proxy.go[7]

I have emailed Brad Fitzpatrick who was involved in writing the reverse
proxy to see if this is expected behaviour, to me it seems like a bug?

[1]
https://github.com/golang/go/blob/release-branch.go1.4/src/net/http/httputil/reverseproxy.go
[2]
https://github.com/golang/go/blob/release-branch.go1.4/src/net/http/httputil/reverseproxy.go#L110
[3]
https://github.com/golang/go/blob/release-branch.go1.4/src/net/http/httputil/reverseproxy.go#L125
[4]
https://github.com/golang/go/blob/release-branch.go1.4/src/net/http/httputil/reverseproxy.go#L92
[5]
https://github.com/golang/go/blob/release-branch.go1.4/src/net/http/httputil/reverseproxy.go#L127
[6]
https://github.com/golang/go/blob/release-branch.go1.4/src/net/http/httputil/reverseproxy.go#L142
[7] https://github.com/cloudfoundry/gorouter/blob/master/proxy/proxy.go#L149

On Thu, Aug 20, 2015 at 6:05 PM, Christopher Piraino <cpiraino(a)pivotal.io>
wrote:

Simon,

That is very interesting, nice investigation. I would have to do a little
digging to try and see what exactly is going on, but the fact that the
GoRouter is not setting the "X-Forwarded-For" header when keep-alive is
enabled sounds suspiciously similar to an issue with HaProxy
<http://comments.gmane.org/gmane.comp.web.haproxy/5419> where it was only
adding the x-forwarded-proto on the first request for a connection. Not
sure if this is actually similar though.

I took a quick look through golang's reverseproxy code and nothing jumped
out at me, I added a story in our backlog
<https://www.pivotaltracker.com/story/show/101692478> to investigate
further.

- Chris Piraino

On Thu, Aug 20, 2015 at 7:20 AM, Simon Johansson <simon(a)simonjohansson.com
wrote:
Disregarding F5/HAProxy comletely and going directly(10.230.18.134) to a
Gorouter also yields different results depending on the client.

$ curl -H "host:cf-env.domain.com" 10.230.18.134
$ wget --header=host:cf-env.domain.com -qO- 10.230.18.134

Results in the following gorouter logs

cf-env.domain.com - [20/08/2015:13:32:57 +0000] "GET / HTTP/1.1" 200 0
5576 "-" "curl/7.43.0" 172.21.27.221:41193
x_forwarded_for:"172.21.27.221" ...

cf-env.domain.com - [20/08/2015:13:35:47 +0000] "GET / HTTP/1.1" 200 0
5655 "-" "Wget/1.16.3 (linux-gnu)" 172.21.27.221:41218
x_forwarded_for:"-" ....

I.e when using curl the x_forwarded_for header is set, but when using
wget its not.

Doing a tcpdump on the gorouter we get these headers for the incoming
request.

GET / HTTP/1.1
host:cf-env.domain.com
User-Agent: curl/7.43.0
Accept: */*

GET / HTTP/1.1
User-Agent: Wget/1.16.3 (linux-gnu)
Accept: */*
Accept-Encoding: identity
host: cf-env.domain.com
Connection: Keep-Alive

As we can see there is two(well, three if you cound User-Agent) differing
headers, the wget request have the Accept-Encoding and Connection headers
whereas the curl request does not.

So, when going directly to a gorouter with curl(that previously set the
x_forwarded_for header) and setting the Connection header to Keep-Alive we
get this

$ curl -H "host:cf-env.domain.com" -H "Connection: Keep-Alive"
10.230.18.134

GET / HTTP/1.1
host:cf-env.domain.com
User-Agent: curl/7.43.0
Accept: */*
Connection: Keep-Alive

cf-env.domain.com - [20/08/2015:13:46:25 +0000] "GET /curl-keep-alive
HTTP/1.1" 404 0 18 "-" "curl/7.43.0" 172.21.27.221:41313
x_forwarded_for:"-" ...

So, Keep-Alive seems to be the difference between getting x_forwarded_for
set or not in this case.

Interestingly, when hitting an app with either wget or curl via
HAProxy(that both logs x_forwarded_for: sourceIP, HAProxyIP) we can see the
headers for the requests on the Gorouter have Connection unset.

$ # On my machine
$ curl -H "host:cf-env.domain.com" 10.230.18.68/curl
$ curl -H "host:cf-env.domain.com" -H "Connection: Keep-Alive"
10.230.18.68/curl-keep-alive-set
$ wget --header=host:cf-env.domain.com -qO- 10.230.18.68/wget

$ # Tcpdump on HAProxy
GET /curl HTTP/1.1
host: cf-env.domain.com
User-Agent: curl/7.43.0
Accept: */*

GET /curl-keep-alive-set HTTP/1.1
host: cf-env.domain.com
User-Agent: curl/7.43.0
Accept: */*
Connection: Keep-Alive

GET /wget HTTP/1.1
User-Agent: Wget/1.16.3 (linux-gnu)
Accept: */*
Accept-Encoding: identity
host: cf-env.domain.com
Connection: Keep-Alive


$ # Tcpdump on Gorouter
GET /curl HTTP/1.1
User-Agent: curl/7.43.0
Accept: */*
host: cf-env.domain.com
X-Forwarded-Proto: http
X-Forwarded-For: 172.21.27.221

GET /curl-keep-alive-set HTTP/1.1
User-Agent: curl/7.43.0
Accept: */*
host: cf-env.domain.com
X-Forwarded-Proto: http
X-Forwarded-For: 172.21.27.221

GET /wget HTTP/1.1
User-Agent: Wget/1.16.3 (linux-gnu)
Accept: */*
Accept-Encoding: identity
host: cf-env.domain.com
X-Forwarded-Proto: http
X-Forwarded-For: 172.21.27.221

$ cf logs cf-env
cf-env.domain.com - [20/08/2015:14:10:12 +0000] "GET /curl HTTP/1.1" 404
0 18 "-" "curl/7.43.0" 10.230.18.68:60573
x_forwarded_for:"172.21.27.221, 10.230.18.68" ...
cf-env.domain.com - [20/08/2015:14:10:17 +0000] "GET
/curl-keep-alive-set HTTP/1.1" 404 0 18 "-" "curl/7.43.0"
10.230.18.68:60594 x_forwarded_for:"172.21.27.221, 10.230.18.68" ...
cf-env.domain.com - [20/08/2015:14:10:22 +0000] "GET /wget HTTP/1.1" 404
0 18 "-" "Wget/1.16.3 (linux-gnu)" 10.230.18.68:60615
x_forwarded_for:"172.21.27.221, 10.230.18.68" ...

So it seems HAProxy is stripping the Connection header before forwarding
it to a Gorouter and that is why we dont get different x_forwarded_for ips
depending on client whereas we get different when using F5 as it doesnt
strip the Connection header.


Re: I'm getting different x_forwarded_for in my Gorouter access logs depending on what browser/cli-tool I use.

Christopher Piraino <cpiraino@...>
 

Simon,

That is very interesting, nice investigation. I would have to do a little
digging to try and see what exactly is going on, but the fact that the
GoRouter is not setting the "X-Forwarded-For" header when keep-alive is
enabled sounds suspiciously similar to an issue with HaProxy
<http://comments.gmane.org/gmane.comp.web.haproxy/5419> where it was only
adding the x-forwarded-proto on the first request for a connection. Not
sure if this is actually similar though.

I took a quick look through golang's reverseproxy code and nothing jumped
out at me, I added a story in our backlog
<https://www.pivotaltracker.com/story/show/101692478> to investigate
further.

- Chris Piraino

On Thu, Aug 20, 2015 at 7:20 AM, Simon Johansson <simon(a)simonjohansson.com>
wrote:

Disregarding F5/HAProxy comletely and going directly(10.230.18.134) to a
Gorouter also yields different results depending on the client.

$ curl -H "host:cf-env.domain.com" 10.230.18.134
$ wget --header=host:cf-env.domain.com -qO- 10.230.18.134

Results in the following gorouter logs

cf-env.domain.com - [20/08/2015:13:32:57 +0000] "GET / HTTP/1.1" 200 0
5576 "-" "curl/7.43.0" 172.21.27.221:41193
x_forwarded_for:"172.21.27.221" ...

cf-env.domain.com - [20/08/2015:13:35:47 +0000] "GET / HTTP/1.1" 200 0
5655 "-" "Wget/1.16.3 (linux-gnu)" 172.21.27.221:41218
x_forwarded_for:"-" ....

I.e when using curl the x_forwarded_for header is set, but when using wget
its not.

Doing a tcpdump on the gorouter we get these headers for the incoming
request.

GET / HTTP/1.1
host:cf-env.domain.com
User-Agent: curl/7.43.0
Accept: */*

GET / HTTP/1.1
User-Agent: Wget/1.16.3 (linux-gnu)
Accept: */*
Accept-Encoding: identity
host: cf-env.domain.com
Connection: Keep-Alive

As we can see there is two(well, three if you cound User-Agent) differing
headers, the wget request have the Accept-Encoding and Connection headers
whereas the curl request does not.

So, when going directly to a gorouter with curl(that previously set the
x_forwarded_for header) and setting the Connection header to Keep-Alive we
get this

$ curl -H "host:cf-env.domain.com" -H "Connection: Keep-Alive"
10.230.18.134

GET / HTTP/1.1
host:cf-env.domain.com
User-Agent: curl/7.43.0
Accept: */*
Connection: Keep-Alive

cf-env.domain.com - [20/08/2015:13:46:25 +0000] "GET /curl-keep-alive
HTTP/1.1" 404 0 18 "-" "curl/7.43.0" 172.21.27.221:41313
x_forwarded_for:"-" ...

So, Keep-Alive seems to be the difference between getting x_forwarded_for
set or not in this case.

Interestingly, when hitting an app with either wget or curl via
HAProxy(that both logs x_forwarded_for: sourceIP, HAProxyIP) we can see the
headers for the requests on the Gorouter have Connection unset.

$ # On my machine
$ curl -H "host:cf-env.domain.com" 10.230.18.68/curl
$ curl -H "host:cf-env.domain.com" -H "Connection: Keep-Alive"
10.230.18.68/curl-keep-alive-set
$ wget --header=host:cf-env.domain.com -qO- 10.230.18.68/wget

$ # Tcpdump on HAProxy
GET /curl HTTP/1.1
host: cf-env.domain.com
User-Agent: curl/7.43.0
Accept: */*

GET /curl-keep-alive-set HTTP/1.1
host: cf-env.domain.com
User-Agent: curl/7.43.0
Accept: */*
Connection: Keep-Alive

GET /wget HTTP/1.1
User-Agent: Wget/1.16.3 (linux-gnu)
Accept: */*
Accept-Encoding: identity
host: cf-env.domain.com
Connection: Keep-Alive


$ # Tcpdump on Gorouter
GET /curl HTTP/1.1
User-Agent: curl/7.43.0
Accept: */*
host: cf-env.domain.com
X-Forwarded-Proto: http
X-Forwarded-For: 172.21.27.221

GET /curl-keep-alive-set HTTP/1.1
User-Agent: curl/7.43.0
Accept: */*
host: cf-env.domain.com
X-Forwarded-Proto: http
X-Forwarded-For: 172.21.27.221

GET /wget HTTP/1.1
User-Agent: Wget/1.16.3 (linux-gnu)
Accept: */*
Accept-Encoding: identity
host: cf-env.domain.com
X-Forwarded-Proto: http
X-Forwarded-For: 172.21.27.221

$ cf logs cf-env
cf-env.domain.com - [20/08/2015:14:10:12 +0000] "GET /curl HTTP/1.1" 404
0 18 "-" "curl/7.43.0" 10.230.18.68:60573 x_forwarded_for:"172.21.27.221,
10.230.18.68" ...
cf-env.domain.com - [20/08/2015:14:10:17 +0000] "GET /curl-keep-alive-set
HTTP/1.1" 404 0 18 "-" "curl/7.43.0" 10.230.18.68:60594
x_forwarded_for:"172.21.27.221, 10.230.18.68" ...
cf-env.domain.com - [20/08/2015:14:10:22 +0000] "GET /wget HTTP/1.1" 404
0 18 "-" "Wget/1.16.3 (linux-gnu)" 10.230.18.68:60615
x_forwarded_for:"172.21.27.221, 10.230.18.68" ...

So it seems HAProxy is stripping the Connection header before forwarding
it to a Gorouter and that is why we dont get different x_forwarded_for ips
depending on client whereas we get different when using F5 as it doesnt
strip the Connection header.


Re: I'm getting different x_forwarded_for in my Gorouter access logs depending on what browser/cli-tool I use.

Simon Johansson <simon@...>
 

Disregarding F5/HAProxy comletely and going directly(10.230.18.134) to a Gorouter also yields different results depending on the client.

$ curl -H "host:cf-env.domain.com" 10.230.18.134
$ wget --header=host:cf-env.domain.com -qO- 10.230.18.134

Results in the following gorouter logs

cf-env.domain.com - [20/08/2015:13:32:57 +0000] "GET / HTTP/1.1" 200 0 5576 "-" "curl/7.43.0" 172.21.27.221:41193 x_forwarded_for:"172.21.27.221" ...

cf-env.domain.com - [20/08/2015:13:35:47 +0000] "GET / HTTP/1.1" 200 0 5655 "-" "Wget/1.16.3 (linux-gnu)" 172.21.27.221:41218 x_forwarded_for:"-" ....

I.e when using curl the x_forwarded_for header is set, but when using wget its not.

Doing a tcpdump on the gorouter we get these headers for the incoming request.

GET / HTTP/1.1
host:cf-env.domain.com
User-Agent: curl/7.43.0
Accept: */*

GET / HTTP/1.1
User-Agent: Wget/1.16.3 (linux-gnu)
Accept: */*
Accept-Encoding: identity
host: cf-env.domain.com
Connection: Keep-Alive

As we can see there is two(well, three if you cound User-Agent) differing headers, the wget request have the Accept-Encoding and Connection headers whereas the curl request does not.

So, when going directly to a gorouter with curl(that previously set the x_forwarded_for header) and setting the Connection header to Keep-Alive we get this

$ curl -H "host:cf-env.domain.com" -H "Connection: Keep-Alive" 10.230.18.134

GET / HTTP/1.1
host:cf-env.domain.com
User-Agent: curl/7.43.0
Accept: */*
Connection: Keep-Alive

cf-env.domain.com - [20/08/2015:13:46:25 +0000] "GET /curl-keep-alive HTTP/1.1" 404 0 18 "-" "curl/7.43.0" 172.21.27.221:41313 x_forwarded_for:"-" ...

So, Keep-Alive seems to be the difference between getting x_forwarded_for set or not in this case.

Interestingly, when hitting an app with either wget or curl via HAProxy(that both logs x_forwarded_for: sourceIP, HAProxyIP) we can see the headers for the requests on the Gorouter have Connection unset.

$ # On my machine
$ curl -H "host:cf-env.domain.com" 10.230.18.68/curl
$ curl -H "host:cf-env.domain.com" -H "Connection: Keep-Alive" 10.230.18.68/curl-keep-alive-set
$ wget --header=host:cf-env.domain.com -qO- 10.230.18.68/wget

$ # Tcpdump on HAProxy
GET /curl HTTP/1.1
host: cf-env.domain.com
User-Agent: curl/7.43.0
Accept: */*

GET /curl-keep-alive-set HTTP/1.1
host: cf-env.domain.com
User-Agent: curl/7.43.0
Accept: */*
Connection: Keep-Alive

GET /wget HTTP/1.1
User-Agent: Wget/1.16.3 (linux-gnu)
Accept: */*
Accept-Encoding: identity
host: cf-env.domain.com
Connection: Keep-Alive


$ # Tcpdump on Gorouter
GET /curl HTTP/1.1
User-Agent: curl/7.43.0
Accept: */*
host: cf-env.domain.com
X-Forwarded-Proto: http
X-Forwarded-For: 172.21.27.221

GET /curl-keep-alive-set HTTP/1.1
User-Agent: curl/7.43.0
Accept: */*
host: cf-env.domain.com
X-Forwarded-Proto: http
X-Forwarded-For: 172.21.27.221

GET /wget HTTP/1.1
User-Agent: Wget/1.16.3 (linux-gnu)
Accept: */*
Accept-Encoding: identity
host: cf-env.domain.com
X-Forwarded-Proto: http
X-Forwarded-For: 172.21.27.221

$ cf logs cf-env
cf-env.domain.com - [20/08/2015:14:10:12 +0000] "GET /curl HTTP/1.1" 404 0 18 "-" "curl/7.43.0" 10.230.18.68:60573 x_forwarded_for:"172.21.27.221, 10.230.18.68" ...
cf-env.domain.com - [20/08/2015:14:10:17 +0000] "GET /curl-keep-alive-set HTTP/1.1" 404 0 18 "-" "curl/7.43.0" 10.230.18.68:60594 x_forwarded_for:"172.21.27.221, 10.230.18.68" ...
cf-env.domain.com - [20/08/2015:14:10:22 +0000] "GET /wget HTTP/1.1" 404 0 18 "-" "Wget/1.16.3 (linux-gnu)" 10.230.18.68:60615 x_forwarded_for:"172.21.27.221, 10.230.18.68" ...

So it seems HAProxy is stripping the Connection header before forwarding it to a Gorouter and that is why we dont get different x_forwarded_for ips depending on client whereas we get different when using F5 as it doesnt strip the Connection header.


Re: CF integration with logger and monitoring tools

James Bayer
 

for 1., getting custom metrics from an app container (JMX, some business
foo metric happened, etc) is a goal for the logging team, but it is lower
than existing work to remove the collector.

[image: Inline image 1]

2. Can I start and stop log collection from firehose anytime? the nozzle
clients connect over websocket, so yes i believe you can simply disconnect
them and then reconnect them later.

On Thu, Aug 20, 2015 at 2:32 AM, Swatz bosh <swatzron(a)gmail.com> wrote:

Hi,

So am assuming that - to integrate tool like Introscope, if I go with
firehose approach then would need to write a custom nozzle/forwarder?

I have couple queries -

1. Can I add custom Metric for my java application apart from the one
which is mention here of firehose catalog [1] ? so that on Wily Introscope
UI, I can have more sliced dashboard apart from these 8-10 metrics.

2. Can I start and stop log collection from firehose anytime? from my
Introscope UI. The idea is to reduce the number of API calls from
Introscope to Doppler firehose and giving flexibility to Wily-Introscope
admin to collect data either periodically by running etl's or through UI.
So will firehose provide REST apis to be called directly from Introscope or
while writing custom nozzle for Introscope I should take care of such REST
endpoints for nozzle?
--
Thank you,

James Bayer


Re: Script hangs when updating the cf-release

Roberto Jimenez Sanchez
 

Hi,

I am having exactly the same issue using develop branch and latest commit:

$ git pull origin develop
From https://github.com/cloudfoundry/cf-release
* branch develop -> FETCH_HEAD
Already up-to-date.
$ git checkout develop
M src/etcd-release
M src/github.com/cloudfoundry/cf-acceptance-tests
M src/hm9000
M src/loggregator
M src/smoke-tests
M src/uaa
M src/warden
Already on 'develop'
Your branch is up-to-date with 'origin/develop'.

[roberto(a)oc4828837525 cf-release]$ ./update
....(output ommitted)
Submodule path 'src/etcd-metrics-server': checked out '90c444c7f93cacb998e45c46f1e06ecf4c8eb9c4'
Submodule path 'src/etcd-release': checked out '2a1cbbf8772f7c2887ac3b8a75b1114dbcd9d9d6'
Submodule path 'src/etcd-release/src/github.com/cloudfoundry-incubator/candiedyaml': checked out '29b4d9cda9fd156adea631d790c95d37ee6ab8e6'
Cloning into 'src/github.com/cloudfoundry-incubator/cf-test-helpers'...
^C
[roberto(a)oc4828837525 cf-release]$ Failed to recurse into submodule path 'src/etcd-release'


Re: Security Question --- Securely wipe data on warden container removal / destruction???

James Bayer
 

this is a relevant tracker story to clean up after containers:
https://www.pivotaltracker.com/n/projects/1158420/stories/96458012

On Wed, Aug 19, 2015 at 8:28 AM, Will Pragnell <wpragnell(a)pivotal.io> wrote:

In the Docker image case, the filesystem layer specific to the container
is also deleted immediately when the container stops running (this is the
same for buildpack based apps on Diego/Garden). Lower layers in the image
(i.e. the pre-existing docker image, as pulled from the registry) are not
currently removed, even if not used in any other running containers.

In the coming weeks, we'll define and implement a strategy to remove
unused images, but the details aren't decided yet.

On 19 August 2015 at 14:57, James Bayer <jbayer(a)pivotal.io> wrote:

warden/DEAs keeps container file systems for a configured amount of time,
something like 1 hr before removing the containers, i believe with standard
removal tools.

diego cells and garden removes container file system immediately after
they are stopped by the user or the system. when using docker images, the
container images are cached in the garden graph directory and i'm not quite
sure of their cleanup / garbage collection life cycle.

On Wed, Aug 19, 2015 at 1:08 AM, Chris K <christopherkugler2(a)yahoo.de>
wrote:

Hi,

I have a few questions regarding the way data is removed when an
application is removed and its corresponding warden container is destroyed.
As the Cloud Foundry instance my company is using may be shared with
multiple tenants, this is a very critical question for us to be answered.
From Cloud Foundry's GitHub repository I gathered the following
information regarding the destruction process:

"When a container is destroyed -- either per user request, or
automatically after being idle -- Warden first kills all unprivileged
processes running inside the container. These processes first receive a
TERM signal followed by a KILL if they haven't exited after a couple of
seconds. When these processes have terminated, the root of the container's
process tree is sent a KILL . Once all resources the container used have
been released, its files are removed and it is considered destroyed."
(Quote: https://github.com/cloudfoundry/warden/tree/master/warden)

According to this quote all files of the file system are removed before
the resources can be used again. But how are they removed? Are they
securely wiped, meaning all blocks are set to zero (or randomized)? And how
is data removed from the RAM before it can be assigned to a new warden
(i.e. new application).

In case the data is not being securely wiped, how much access does an
application have towards the available memory? Is it for example possible
to create files of arbitrary size and read / access them?

I'd be thankful for any kind of hints on this topic.

With Regards,
Chris


--
Thank you,

James Bayer
--
Thank you,

James Bayer


Re: no more stdout in app files since upgrade to 214

ramonskie
 

i removed the following

nats:
authorization_timeout: 10
use_gnatsd: true

changed
dea: &dea
to
dea_next:

and removed
dea_next: *dea

and it seems to log App/0 again
please note that because of this change all dea's where reconfigured



--
View this message in context: http://cf-dev.70369.x6.nabble.com/no-more-stdout-in-app-files-since-upgrade-to-214-tp1197p1294.html
Sent from the CF Dev mailing list archive at Nabble.com.


Re: I'm getting different x_forwarded_for in my Gorouter access logs depending on what browser/cli-tool I use.

Simon Johansson <simon@...>
 

Dieu,

We use F5, just tried with the release HAProxy and I get concictent X-Forwarded-For between clients! Dont know how I managed to forget to check that. :)

In that way, I believe it should be consistent, whatever the client.
So in the HAProxy config the header gets updated/inserted with

frontend http-in
....
option forwardfor

In the HTTP profile in F5 we have " Insert X-Forwarded-For " enabled which is basically the same as the HAProxy config option. No irules to modify the header. But I get different results depending on the client.

How curious....


Re: CF integration with logger and monitoring tools

Swatz bosh
 

Hi,

So am assuming that - to integrate tool like Introscope, if I go with firehose approach then would need to write a custom nozzle/forwarder?

I have couple queries -

1. Can I add custom Metric for my java application apart from the one which is mention here of firehose catalog [1] ? so that on Wily Introscope UI, I can have more sliced dashboard apart from these 8-10 metrics.

2. Can I start and stop log collection from firehose anytime? from my Introscope UI. The idea is to reduce the number of API calls from Introscope to Doppler firehose and giving flexibility to Wily-Introscope admin to collect data either periodically by running etl's or through UI. So will firehose provide REST apis to be called directly from Introscope or while writing custom nozzle for Introscope I should take care of such REST endpoints for nozzle?


Re: Overcommit on Diego Cells

Mike Youngstrom
 

Thanks for the response Eric. It was very helpful.

One last question. Any thoughts on what would be the best way to monitor
free ephemeral disk space in my overcommitted situation? If using
btrfs_store_size_mb=-1 will btrfs free ephemeral disk space when less is
being used or does it just grow when it needs more? Looking at firehose
stats in 1398 I don't see any btrfs usage metrics being sent from
garden-linux.

Thanks,
Mike

On Mon, Aug 17, 2015 at 9:14 PM, Eric Malm <emalm(a)pivotal.io> wrote:

Hi, Mike,

Apologies, I emailed this to cf-dev a few days ago, but it seems not to
have gone through. Anyway, thanks for asking about the different
configuration values Diego exposes for disk and memory. Yes, you can use
the 'diego.executor.memory_capacity_mb' and
'diego.executor.disk_capacity_mb' properties to specify overcommits in
absolute terms rather than the relative factors configurable on the DEAs.
The cell reps will advertise those values as their maximum memory and disk
capacity, and subtract memory and disk for allocated containers when
reporting their available capacity during auctions.

The 'btrfs_store_size_mb' property on garden-linux is more of a moving
target as garden-linux settles in on that filesystem as a backing store. As
of garden-linux-release 0.292.0, which diego-release 0.1412.0 and later
consume, that property accepts a '-1' value that allows it to grow up to
the full size of the available disk on the /var/vcap/data ephemeral disk
volume. The btrfs volume itself is sparse, so it will start at effectively
zero size and grow as needed to accommodate the container layers. Since
you're already monitoring disk usage on your VMs carefully and scaling out
when you hit certain limits, this might be a good option for you. This is
also effectively how the DEAs operate today, without an explicit limit on
the total amount of disk they allocate for containers.

If you do want more certainty in the maximum size that the garden-linux
btrfs volume will grow to, or if you're on a version of diego-release
earlier than 0.1412.0, you should set btrfs_store_size_mb to a positive
value, and garden-linux will create the volume to grow only up to that
size. One strategy to determine that value would be to use the maximum size
of the ephemeral disk, less the size of the BOSH-deployed packages (for the
executor, currently around 1.3 GB, including the untarred cflinuxfs2
rootfs), less the size allocated to the executor cache in the
'diego.executor.max_cache_size_in_bytes' property (which currently defaults
to 10GB).

Best,
Eric


Re: Web proxy support in buildpacks

Jack Cai
 

Thanks JT for your answer. Based on our experience, one of the best
practice to share is that it's a good idea to set all the below four
environment variables:

http_proxy, HTTP_PROXY, https_proxy, HTTPS_PROXY

The reason is that it seems different tools/runtimes recognize different
set of them. E.g, curl does not recognize "HTTP_PROXY" according to [1].

Jack

[1] http://curl.haxx.se/docs/manpage.html

On Mon, Aug 17, 2015 at 9:39 AM, JT Archie <jarchie(a)pivotal.io> wrote:

Jack,

For cached buildpacks, it would not be useful to set HTTP proxying. The
dependencies are bundled with the buildpack and are loaded via the local
file system, not HTTP.

Most of the buildpacks use curl to download the dependencies from the HTTP
server. You should be able to set the environment variables HTTP_PROXY or
HTTPS_PROXY to for curl to use the proxy server.
<http://curl.haxx.se/libcurl/c/CURLOPT_PROXY.html> If this works for you
it would be great to hear your feedback.

Kinds Regards,

JT

On Mon, Aug 17, 2015 at 9:26 AM, Jack Cai <greensight(a)gmail.com> wrote:

Currently I see that the Java buildpack and the PHP buildpack explicitly
mentioned in their doc that they can run behind a Web proxy, by setting the
HTTP_PROXY and HTTPS_RPOXY environment variable. And I suppose this is
supported in either the cached version or the uncached one, and for both
the old lucid64 stack and the new cflinuxfs2 stack (which has different
Ruby version). Do other buildpacks support the same? Aka node.js, python,
ruby, go, etc.

Thanks in advance!

Jack


Re: Update on Mailman 3 launch

Eric Searcy <eric@...>
 

We have an open upstream bug about the issue that is causing delays to mail (including
messages you post via the web interface sometimes not showing up). I hope this will have
a proper resolution by the end of the week, and in the mean time we will be monitoring for
this and “unsticking” any messages that get stuck: so if you don’t see any error, then
your message *will* get posted.
(https://gitlab.com/mailman/mailman/issues/138)
We have a workaround in place for the above issue (hack code to disable connection pooling) and it seems to have addressed this problem until we can get a better fix.

We've also just filed a new bug related to digest delivery mode affecting 3 or 4 list members, however we haven't yet determined exactly what the nature of the impact is. Hopefully a quick fix as soon as we discover more about the bug.
(https://gitlab.com/mailman/mailman/issues/141)

Eric


Re: Running Docker private images on CF

dharmi
 

Thanks for the details.
I deployed diego-docker-cache-release and I could run private docker images
now.

One note however. I had to modify the **property-overrides.yml *to add the
IP:<port> of the *docker-cache/0* job among the
*insecure_docker_registry_list* of for it to work. Without which it says
{"timestamp":"1439701925.514369965","source":"garden-linux","message":"garden-linux.pool.umojd9q7s54.provide-rootfs-failed","log_level":2,"data":{"error":"repository_fetcher:
ProvideRegistry: could not fetch image f93137f1-.. from registry
10.250.21.80:8080: Registry 10.250.21.80:8080 is missing from
-insecureDockerRegistryList
([docker-registry.service.cf.internal:8080])","session":"2.13"}}

Consul discovery at fault I suspect, if not, pls suggest.

Another observation on the Docker registry URI while running docker private
images(*, not a Diego issue, I guess*)
Looks like by default (*when I don't mention **docker_login_server*), the
images are pulled using the V1 api

$ cf start myapp
Starting app myapp in org myorg / space default as user...
Creating container
Successfully created container
Staging...
Docker daemon running
Staging process started ...
Caching docker image ...
*Logging to https://index.docker.io/v1/ <https://index.docker.io/v1/> ...*
WARNING: login credentials saved in /root/.dockercfg.
Login Succeeded
Logged in.
Pulling docker image <dockerid>/<image>:latest ...
latest: Pulling from <dockerid>/image
511136ea3c5a: Pulling fs layer
30d39e59ffe2: Pulling fs layer
c90d655b99b2: Pulling fs layer
…..

when I explicitly mention the V2 URI, which is *registry.hub.docker.com
<http://registry.hub.docker.com>* (*correct me if I am wrong*), pulling the
image fails.

$ cf start myapp
Starting app myapp in org myorg / space default as user...
Creating container
Successfully created container
Staging...
Docker daemon running
Staging process started ...
Caching docker image ...
*Logging to https://registry.hub.docker.com/
<https://registry.hub.docker.com/> ...*
WARNING: login credentials saved in /root/.dockercfg.
*Login Succeeded*
Logged in.
Pulling docker image <dockerid>/<image>:latest ...
time="2015-08-19T19:59:44Z" level=error msg=*"Error from V2 registry:
Authentication is required."*
Pulling repository <dockerid>/<image>
Error: image <dockerid>/<image>:latest ... not found

Thanks

On Tue, Aug 11, 2015 at 6:45 PM, Eric Malm <emalm(a)pivotal.io> wrote:

Hi, Dharmi,

In order to run private docker images (that is, ones that require
user/password/email authentication with the registry), you'll have to stage
them into the optional diego-docker-cache deployed alongside Diego. The
BOSH release is located at
https://github.com/cloudfoundry-incubator/diego-docker-cache-release. If
you've already deployed Diego using the spiff-based manifest-generation
templates in diego-release, the deployment for this release should be
similar. If you deploy the caching registry release without TLS enabled or
enabled but with a self-signed certificate, Diego should then be configured
with the URL "docker-registry.service.cf.internal:8080" supplied in the
diego.garden-linux.insecure_docker_registry_list property, and
diego.stager.insecure_docker_registry set to 'true', as you can see in
https://github.com/cloudfoundry-incubator/diego-docker-cache-release/blob/develop/stubs-for-diego-release/bosh-lite-property-overrides.yml
.

Once that release is deployed, you can follow the instructions at
https://github.com/cloudfoundry-incubator/diego-docker-cache-release#caching-docker-image-with-diego
to stage your image into the cache, which should be as simple as setting
the DIEGO_DOCKER_CACHE env var to 'true' on your app before staging it.
When you start the app, Diego will then instruct Garden to pull the image
from the internal caching registry rather than from the remote registry you
staged it from. This has the added benefit of ensuring that you're always
running exactly the Docker image you staged, rather than something that may
have changed in the remote registry.

Thanks,
Eric, CF Runtime Diego PM

On Tue, Aug 11, 2015 at 9:32 AM, dharmi <dharmi(a)gmail.com> wrote:

We have CF v214 with Diego deployed on AWS.

I am able to successfully create apps from Docker public repo, as per the
apidocs <http://apidocs.cloudfoundry.org/214/apps/creating_an_app.html>
,

but, while creating apps from the Docker private repos, I see the below
error from 'cf logs' when starting the app.

[API/0] OUT Updated app with guid bcb8f363-xyz
({"route"=>"5af6948b-xyz"})
[API/0] OUT Updated app with guid bcb8f363-xyz ({"state"=>"STARTED"})
[STG/0] OUT Creating container
[STG/0] OUT Successfully created container
[STG/0] OUT Staging...
[STG/0] OUT Staging process started ...
[STG/0] ERR Staging process failed: Exit trace for group:
[STG/0] ERR builder exited with error: failed to fetch metadata from
[:dockerid/go-app] with tag [latest] and insecure registries [] due to
HTTP
code: 404
[STG/0] OUT Exit status 2
[STG/0] ERR Staging Failed: Exited with status 2
[API/0] ERR Failed to stage application: staging failed


cf curl command for reference.

cf curl /v2/apps -X POST -H "Content-Type: application/json" -H
"Authorization: bearer *accessToken*" -d '
{"name": "myapp",
"space_guid": "71b22eba-xyz",
"docker_image": ":dockerid/go-app",
"diego": true,
"docker_credentials_json":
{"docker_login_server": "https://index.docker.io/v1/",
"docker_user": ":dockerid",
"docker_password": ":dockerpwd",
"docker_email": ":email"
}
}'

Looking at the apidocs, the 'Example value' for 'docker_credentials_json'
indicates a Hash value
(#<RspecApiDocumentation::Views::HtmlExample:0x0000000bb883e0>), but
looking
inside the code, we found the below JSON format.

let(:docker_credentials) do
{
docker_login_server: login_server,
docker_user: user,
docker_password: password,
docker_email: email
}

Pls correct me if I am missing something.

Thanks,
Dharmi



--
View this message in context:
http://cf-dev.70369.x6.nabble.com/Running-Docker-private-images-on-CF-tp1148.html
Sent from the CF Dev mailing list archive at Nabble.com.
--
Wise people learn when they can. Fools learn when they must.” - The Duke of
Ellington


Re: I'm getting different x_forwarded_for in my Gorouter access logs depending on what browser/cli-tool I use.

Dieu Cao <dcao@...>
 

What do you use for your edge load balancer?
You could try to set up your edge load balancer to strip x_forwarded_for
sent in the request and set it again before forwarding on.
In that way, I believe it should be consistent, whatever the client.

-Dieu

On Wed, Aug 19, 2015 at 7:38 AM, Simon Johansson <simon(a)simonjohansson.com>
wrote:

Gist of diff:
https://gist.github.com/simonjohansson/847e972f1459ea4cd65e

Gist of debug output:
https://gist.github.com/simonjohansson/3324b08fc42e5ac7105a

Since it's quite hard to read in the email...


Re: Security Question --- Securely wipe data on warden container removal / destruction???

Will Pragnell <wpragnell@...>
 

In the Docker image case, the filesystem layer specific to the container is
also deleted immediately when the container stops running (this is the same
for buildpack based apps on Diego/Garden). Lower layers in the image (i.e.
the pre-existing docker image, as pulled from the registry) are not
currently removed, even if not used in any other running containers.

In the coming weeks, we'll define and implement a strategy to remove unused
images, but the details aren't decided yet.

On 19 August 2015 at 14:57, James Bayer <jbayer(a)pivotal.io> wrote:

warden/DEAs keeps container file systems for a configured amount of time,
something like 1 hr before removing the containers, i believe with standard
removal tools.

diego cells and garden removes container file system immediately after
they are stopped by the user or the system. when using docker images, the
container images are cached in the garden graph directory and i'm not quite
sure of their cleanup / garbage collection life cycle.

On Wed, Aug 19, 2015 at 1:08 AM, Chris K <christopherkugler2(a)yahoo.de>
wrote:

Hi,

I have a few questions regarding the way data is removed when an
application is removed and its corresponding warden container is destroyed.
As the Cloud Foundry instance my company is using may be shared with
multiple tenants, this is a very critical question for us to be answered.
From Cloud Foundry's GitHub repository I gathered the following
information regarding the destruction process:

"When a container is destroyed -- either per user request, or
automatically after being idle -- Warden first kills all unprivileged
processes running inside the container. These processes first receive a
TERM signal followed by a KILL if they haven't exited after a couple of
seconds. When these processes have terminated, the root of the container's
process tree is sent a KILL . Once all resources the container used have
been released, its files are removed and it is considered destroyed."
(Quote: https://github.com/cloudfoundry/warden/tree/master/warden)

According to this quote all files of the file system are removed before
the resources can be used again. But how are they removed? Are they
securely wiped, meaning all blocks are set to zero (or randomized)? And how
is data removed from the RAM before it can be assigned to a new warden
(i.e. new application).

In case the data is not being securely wiped, how much access does an
application have towards the available memory? Is it for example possible
to create files of arbitrary size and read / access them?

I'd be thankful for any kind of hints on this topic.

With Regards,
Chris


--
Thank you,

James Bayer

8101 - 8120 of 9425