Date   

Re: UAA local run stuck at "cargoRunLocal"

Filip Hanik
 

./gradlew run --info

The --info flag will print out the UAA and Tomcat logs to screen.

the server will then be accessible on http://localhost:8080/uaa

On Fri, Aug 21, 2015 at 2:43 PM, Alex Lam <alex.c.lam(a)intel.com> wrote:

Can someone shed some light on why my UAA local run got stuck at
cargoRunLocal?
I don't think the service is up because I don't see anything listening to
port 8080.

When I did a Ctrl-C, everything ended.

I am using Oracle Java 1.8

thanks,
alex

:cargoRunLocal
Press Ctrl-C to stop the container...
:cargoRunLocal took 254765ms
:run
:run took 0ms

BUILD SUCCESSFUL

Total time: 6 mins 27.502 secs
Task timings:
642ms :cloudfoundry-identity-common:compileJava
690ms :cloudfoundry-identity-common:processResources
1279ms :cloudfoundry-identity-common:jar
19984ms :cloudfoundry-identity-common:javadoc
946ms :cloudfoundry-identity-common:javadocJar
3346ms :cloudfoundry-identity-scim:compileJava
1213ms :cloudfoundry-identity-login:compileJava
4263ms :cloudfoundry-identity-scim:javadoc
4022ms :cloudfoundry-identity-login:javadoc
11868ms :cloudfoundry-identity-uaa:war
3286ms :cloudfoundry-identity-samples:cloudfoundry-identity-api:javadoc
7085ms :cloudfoundry-identity-samples:cloudfoundry-identity-api:war
1969ms :cloudfoundry-identity-samples:cloudfoundry-identity-app:javadoc
6920ms :cloudfoundry-identity-samples:cloudfoundry-identity-app:war
254765ms :cargoRunLocal


UAA local run stuck at "cargoRunLocal"

Alex Lam
 

Can someone shed some light on why my UAA local run got stuck at cargoRunLocal?
I don't think the service is up because I don't see anything listening to port 8080.

When I did a Ctrl-C, everything ended.

I am using Oracle Java 1.8

thanks,
alex

:cargoRunLocal
Press Ctrl-C to stop the container...
:cargoRunLocal took 254765ms
:run
:run took 0ms

BUILD SUCCESSFUL

Total time: 6 mins 27.502 secs
Task timings:
642ms :cloudfoundry-identity-common:compileJava
690ms :cloudfoundry-identity-common:processResources
1279ms :cloudfoundry-identity-common:jar
19984ms :cloudfoundry-identity-common:javadoc
946ms :cloudfoundry-identity-common:javadocJar
3346ms :cloudfoundry-identity-scim:compileJava
1213ms :cloudfoundry-identity-login:compileJava
4263ms :cloudfoundry-identity-scim:javadoc
4022ms :cloudfoundry-identity-login:javadoc
11868ms :cloudfoundry-identity-uaa:war
3286ms :cloudfoundry-identity-samples:cloudfoundry-identity-api:javadoc
7085ms :cloudfoundry-identity-samples:cloudfoundry-identity-api:war
1969ms :cloudfoundry-identity-samples:cloudfoundry-identity-app:javadoc
6920ms :cloudfoundry-identity-samples:cloudfoundry-identity-app:war
254765ms :cargoRunLocal


Re: Using s3 for blobstore bucket prefix

Rich Wohlstadter
 

Thanks for the confirmation. Guess for now we will use 4 buckets. Hopefully this could be an added feature in the future.

-Rich


Re: Is spiff dead?

Amit Kumar Gupta
 

Spiff is not currently replaced by another tool, but it is not the ideal
tool for the job (too many features to shoot yourself in the foot with, not
enough features about BOSH knowledge, and just some general awkward hoops
it makes you jump through). We have it on our roadmap to improve manifest
generation, so we're not investing more activity into spiff that will slow
our progress towards where we eventually want to end up. For now, manifest
generation remains the same, and we will aim to introduce improvements in a
smooth manner.

A great majority of the improvements in manifest generation will come from
BOSH itself. See the bosh-notes for a list of current and upcoming
features (https://github.com/cloudfoundry/bosh-notes). Specifically the
list under the "Deployment configuration" heading in the README. Those
features open up some exciting possibilities for how simple manifest might
become.

Best,
Amit, Release Integration (MEGA) team PM

On Thu, Aug 20, 2015 at 11:29 PM, Kei YAMAZAKI <daydream.yamazaki(a)gmail.com>
wrote:

Hi all,

I consider use the spiff to manage a lot of manifest.
But I found statement that the spiff is not under active development.

https://github.com/cloudfoundry-incubator/spiff/commit/e7af3990e58b7390826b606d6de76ea576d9ad4f

Manifest management is very complex and cf-release has same problem (
diego-release also ).
I think the spiff is unable to resolve this problem completely and I
recognize the mega team is working on this problem.

So I have questions,
- Is spiff replaced by other tool?
- How to manage manifest files in the mega team.

Thanks,


Update Dashboard URL with Asynchronous Provisioning

Robert Moss
 

Hi,

My service broker provisions a service asynchronously but doesn't get the
management url until the service is provisioned. Currently the poll to the
endpoint /{instanceId}/last_operation doesn't return this field in its body
to update the service. Is there any other way to update the service when
this value becomes available?

Robert

--
Cloudsoft Corporation Limited, Registered in Scotland No: SC349230.
Registered Office: 13 Dryden Place, Edinburgh, EH9 1RP

This e-mail message is confidential and for use by the addressee only. If
the message is received by anyone other than the addressee, please return
the message to the sender by replying to it and then delete the message
from your computer. Internet e-mails are not necessarily secure. Cloudsoft
Corporation Limited does not accept responsibility for changes made to this
message after it was sent.

Whilst all reasonable care has been taken to avoid the transmission of
viruses, it is the responsibility of the recipient to ensure that the
onward transmission, opening or use of this message and any attachments
will not adversely affect its systems or data. No responsibility is
accepted by Cloudsoft Corporation Limited in this regard and the recipient
should carry out such virus and other checks as it considers appropriate.


Re: I'm getting different x_forwarded_for in my Gorouter access logs depending on what browser/cli-tool I use.

Simon Johansson <simon@...>
 

I have now asked in golang-nuts[1] . It might take up to 24h for my message to appear, it will be called "Inconsistent X-Forwarded-For when using the httputil.ReverseProxy ?"

[1] https://groups.google.com/forum/#!forum/golang-nuts


Re: CF integration with logger and monitoring tools

Swatz bosh
 

Thanks James.

So for Wily introscope( wily-agent based monitoring tool) integration, if I go with firehose approach, I need to write my own nozzle for Introscope like for datadog ?

https://github.com/cloudfoundry-incubator/datadog-firehose-nozzle

I am assuming that I will be using dea-agent to fetch metrics and logs instead of wily-agent?

My introscope nozzle will call doppler to fetch metrics and Introscope forwarder will forward it to Introscope Enterprise Manager (a server which has web-client to access metrics dashboard) ?


Is spiff dead?

Kei YAMAZAKI
 

Hi all,

I consider use the spiff to manage a lot of manifest.
But I found statement that the spiff is not under active development.
https://github.com/cloudfoundry-incubator/spiff/commit/e7af3990e58b7390826b606d6de76ea576d9ad4f

Manifest management is very complex and cf-release has same problem ( diego-release also ).
I think the spiff is unable to resolve this problem completely and I recognize the mega team is working on this problem.

So I have questions,
- Is spiff replaced by other tool?
- How to manage manifest files in the mega team.

Thanks,


App syslog drain performance improvements

Erik Jasiak
 

Hi CF community,

The Loggregator team has two medium-impact changes as part of CF v215.

* First, we fixed a bug causing slower performance with application
syslog drains. [1][2] The impact is that doppler should stream
application log messages much faster, and by default you should see
fewer “We’ve dropped 100 messages” problems, and related scalability issues.

** Special thanks to Daniel Jones / “EngineerBetter” in the CF community
for identifying the issue - it’s fantastic to see this level of
investigation and participation in the code base, which is what open
source is all about.

* Second, the buffer size for a Doppler is now configurable in bosh for
loggregator dopplers. The number of messages to buffer in doppler is
configurable as:

doppler.message_drain_buffer_size = 100

The default buffer size for dopplers is still set to 100 messages for
now, while we evaluate the effectiveness of fixing the latency bug.
However, we would appreciate feedback from those that do re-configure
their buffer sizes.

Known impacts of reconfiguring buffer size:

1) Memory usage from increased buffer analyzed, please see story[3].
Overhead of upping size appears to be minor, but if you have a different
experience please let us know.
2) When buffer size increased, falling behind a doppler runs the risk of
a much larger number of dropped messages.

Thank you all, and happy loggregating!

Erik Jasiak
PM, Loggregator team

[1] https://www.pivotaltracker.com/n/projects/993188/stories/99494586
[2] https://github.com/cloudfoundry/loggregator/pull/71
[3] https://www.pivotaltracker.com/story/show/100163298


Re: Overcommit on Diego Cells

Mike Youngstrom
 

Thanks Will.

If btrfs does free disk space then I can just use the bosh ephemeral disk
metric to monitor. If it doesn't then I'll need Garden to provide me with
something.

Thanks,
Mike

On Thu, Aug 20, 2015 at 10:58 AM, Will Pragnell <wpragnell(a)pivotal.io>
wrote:

Apparently my last reply to this thread never made it through. Hope this
one does!

Mike, you're right that there are currently no btrfs metrics being emitted
from garden-linux. There are currently no immediate plans to implement
this, but clearly such metrics are useful, so I'll raise this with the team
and see where we land.

As for your question about btrfs freeing disk space, I'm afraid I don't
know off hand. I'll have to do some investigation and get back to you on
that next week.

On 19 August 2015 at 23:46, Mike Youngstrom <youngm(a)gmail.com> wrote:

Thanks for the response Eric. It was very helpful.

One last question. Any thoughts on what would be the best way to monitor
free ephemeral disk space in my overcommitted situation? If using
btrfs_store_size_mb=-1 will btrfs free ephemeral disk space when less is
being used or does it just grow when it needs more? Looking at firehose
stats in 1398 I don't see any btrfs usage metrics being sent from
garden-linux.

Thanks,
Mike

On Mon, Aug 17, 2015 at 9:14 PM, Eric Malm <emalm(a)pivotal.io> wrote:

Hi, Mike,

Apologies, I emailed this to cf-dev a few days ago, but it seems not to
have gone through. Anyway, thanks for asking about the different
configuration values Diego exposes for disk and memory. Yes, you can use
the 'diego.executor.memory_capacity_mb' and
'diego.executor.disk_capacity_mb' properties to specify overcommits in
absolute terms rather than the relative factors configurable on the DEAs.
The cell reps will advertise those values as their maximum memory and disk
capacity, and subtract memory and disk for allocated containers when
reporting their available capacity during auctions.

The 'btrfs_store_size_mb' property on garden-linux is more of a moving
target as garden-linux settles in on that filesystem as a backing store. As
of garden-linux-release 0.292.0, which diego-release 0.1412.0 and later
consume, that property accepts a '-1' value that allows it to grow up to
the full size of the available disk on the /var/vcap/data ephemeral disk
volume. The btrfs volume itself is sparse, so it will start at effectively
zero size and grow as needed to accommodate the container layers. Since
you're already monitoring disk usage on your VMs carefully and scaling out
when you hit certain limits, this might be a good option for you. This is
also effectively how the DEAs operate today, without an explicit limit on
the total amount of disk they allocate for containers.

If you do want more certainty in the maximum size that the garden-linux
btrfs volume will grow to, or if you're on a version of diego-release
earlier than 0.1412.0, you should set btrfs_store_size_mb to a positive
value, and garden-linux will create the volume to grow only up to that
size. One strategy to determine that value would be to use the maximum size
of the ephemeral disk, less the size of the BOSH-deployed packages (for the
executor, currently around 1.3 GB, including the untarred cflinuxfs2
rootfs), less the size allocated to the executor cache in the
'diego.executor.max_cache_size_in_bytes' property (which currently defaults
to 10GB).

Best,
Eric


Re: Using s3 for blobstore bucket prefix

CF Runtime
 

Currently there is no way to use a single bucket with prefixes for the
different blobstores. In theory you could point them all at the same bucket
and nothing would break, but you'd probably regret that some day when you
wanted to clean out the resource matching blobstore or something like that.

Joseph
OSS Release Integration Team

On Thu, Aug 20, 2015 at 11:58 AM, Rich Wohlstadter <lethwin(a)gmail.com>
wrote:

Hi,

I'm trying to setup so that cloud controller uses s3 for blobstore in
AWS. I can get it working by supplying bucket names in the relevant key
fields(resource_directory_key,etc). Want I wanted to know is if there is
anyway we can supply prefix's. For instance, I want to use the same bucket
for droplets, buildpacks, etc and organize them as so:

mybucket/prod/cc-buildpacks for buildpacks
mybucket/prod/cc-droplets for droplets
mybucket/prod/cc-packages for packages
mybucket/prod/cc-resources for resources

Just cant find a way to specify the prefix's so I don't have to create 4
buckets. Any way to do this?

Thanks,

Rich


Re: Post: Cloud Foundry Advisory Board Meeting Notes - 2015 August

Christopher B Ferris <chrisfer@...>
 

thanks, Phil!

Cheers,

Christopher Ferris
IBM Distinguished Engineer, CTO Open Cloud
IBM Software Group, Open Technologies
email: chrisfer(a)us.ibm.com
twitter: @christo4ferris
blog: http://thoughtsoncloud.com/index.php/author/cferris/
phone: +1 508 667 0402

On Aug 20, 2015, at 3:52 PM, Phil Whelan <philw(a)activestate.com> wrote:

Hi,

I've just posted a write-up of last week's Cloud Foundry Advisory Board Meeting.
http://www.activestate.com/blog/2015/08/cloud-foundry-advisory-board-meeting-2015-august

Topics include...
- Foundation Update
- Diego
- BOSH
- MEGA
- Loggregator
- CAPI
- CLI
- Services Core
- Lattice
- Routing
- Garden
- Greenhouse
- UAA
- Abacus
- "Super Important Question"
- Cloud Foundry v3
- Slack

Please let me know if you spot any inaccuracies.

Thanks,
Phil


max length with Dropped log message: message too long (>64K ...)

Koper, Dies <diesk@...>
 

Hi,

When my app outputs more than 64K of text, I get a message "Dropped log message: message too long (>64K without a newline).
When my app outputs much less than 64K of text, I get the app's output.
When my app outputs just under 64K of text, nothing is output: neither the error message, nor the app's output.

What is exactly the limit under which output is still guaranteed, and can we update the error message accordingly?

I tried to read the source code. Not sure if the limit is because messages are sent over UDP (UDP max data length is 65,507), or due to other data that is included in the transmission of messages (timestamp, app id, etc.).

Cheers,
Dies Koper


Post: Cloud Foundry Advisory Board Meeting Notes - 2015 August

Phil Whelan
 

Hi,

I've just posted a write-up of last week's Cloud Foundry Advisory Board
Meeting.
http://www.activestate.com/blog/2015/08/cloud-foundry-advisory-board-meeting-2015-august

Topics include...
- Foundation Update
- Diego
- BOSH
- MEGA
- Loggregator
- CAPI
- CLI
- Services Core
- Lattice
- Routing
- Garden
- Greenhouse
- UAA
- Abacus
- "Super Important Question"
- Cloud Foundry v3
- Slack

Please let me know if you spot any inaccuracies.

Thanks,
Phil


Using s3 for blobstore bucket prefix

Rich Wohlstadter
 

Hi,

I'm trying to setup so that cloud controller uses s3 for blobstore in AWS. I can get it working by supplying bucket names in the relevant key fields(resource_directory_key,etc). Want I wanted to know is if there is anyway we can supply prefix's. For instance, I want to use the same bucket for droplets, buildpacks, etc and organize them as so:

mybucket/prod/cc-buildpacks for buildpacks
mybucket/prod/cc-droplets for droplets
mybucket/prod/cc-packages for packages
mybucket/prod/cc-resources for resources

Just cant find a way to specify the prefix's so I don't have to create 4 buckets. Any way to do this?

Thanks,

Rich


Re: Script hangs when updating the cf-release

CF Runtime
 

Hmm, we're not seeing any problems like that, and really we're just running
some git commands.

I would assume that running the script does create a lot of connections in
a short amount of time. Is it possible part of your network infrastructure
is seeing this as some sort of attack and blocking connections after a
certain number is reached?

Joseph
OSS Release Integration Team

On Thu, Aug 20, 2015 at 6:36 AM, Roberto Jimenez Sanchez <roberto(a)de.ibm.com
wrote:
Hi,

I am having exactly the same issue using develop branch and latest commit:

$ git pull origin develop
From https://github.com/cloudfoundry/cf-release
* branch develop -> FETCH_HEAD
Already up-to-date.
$ git checkout develop
M src/etcd-release
M src/github.com/cloudfoundry/cf-acceptance-tests
M src/hm9000
M src/loggregator
M src/smoke-tests
M src/uaa
M src/warden
Already on 'develop'
Your branch is up-to-date with 'origin/develop'.

[roberto(a)oc4828837525 cf-release]$ ./update
....(output ommitted)
Submodule path 'src/etcd-metrics-server': checked out
'90c444c7f93cacb998e45c46f1e06ecf4c8eb9c4'
Submodule path 'src/etcd-release': checked out
'2a1cbbf8772f7c2887ac3b8a75b1114dbcd9d9d6'
Submodule path 'src/etcd-release/src/
github.com/cloudfoundry-incubator/candiedyaml': checked out
'29b4d9cda9fd156adea631d790c95d37ee6ab8e6'
Cloning into 'src/github.com/cloudfoundry-incubator/cf-test-helpers'...
^C
[roberto(a)oc4828837525 cf-release]$ Failed to recurse into submodule path
'src/etcd-release'


Re: Overcommit on Diego Cells

Will Pragnell <wpragnell@...>
 

Apparently my last reply to this thread never made it through. Hope this
one does!

Mike, you're right that there are currently no btrfs metrics being emitted
from garden-linux. There are currently no immediate plans to implement
this, but clearly such metrics are useful, so I'll raise this with the team
and see where we land.

As for your question about btrfs freeing disk space, I'm afraid I don't
know off hand. I'll have to do some investigation and get back to you on
that next week.

On 19 August 2015 at 23:46, Mike Youngstrom <youngm(a)gmail.com> wrote:

Thanks for the response Eric. It was very helpful.

One last question. Any thoughts on what would be the best way to monitor
free ephemeral disk space in my overcommitted situation? If using
btrfs_store_size_mb=-1 will btrfs free ephemeral disk space when less is
being used or does it just grow when it needs more? Looking at firehose
stats in 1398 I don't see any btrfs usage metrics being sent from
garden-linux.

Thanks,
Mike

On Mon, Aug 17, 2015 at 9:14 PM, Eric Malm <emalm(a)pivotal.io> wrote:

Hi, Mike,

Apologies, I emailed this to cf-dev a few days ago, but it seems not to
have gone through. Anyway, thanks for asking about the different
configuration values Diego exposes for disk and memory. Yes, you can use
the 'diego.executor.memory_capacity_mb' and
'diego.executor.disk_capacity_mb' properties to specify overcommits in
absolute terms rather than the relative factors configurable on the DEAs.
The cell reps will advertise those values as their maximum memory and disk
capacity, and subtract memory and disk for allocated containers when
reporting their available capacity during auctions.

The 'btrfs_store_size_mb' property on garden-linux is more of a moving
target as garden-linux settles in on that filesystem as a backing store. As
of garden-linux-release 0.292.0, which diego-release 0.1412.0 and later
consume, that property accepts a '-1' value that allows it to grow up to
the full size of the available disk on the /var/vcap/data ephemeral disk
volume. The btrfs volume itself is sparse, so it will start at effectively
zero size and grow as needed to accommodate the container layers. Since
you're already monitoring disk usage on your VMs carefully and scaling out
when you hit certain limits, this might be a good option for you. This is
also effectively how the DEAs operate today, without an explicit limit on
the total amount of disk they allocate for containers.

If you do want more certainty in the maximum size that the garden-linux
btrfs volume will grow to, or if you're on a version of diego-release
earlier than 0.1412.0, you should set btrfs_store_size_mb to a positive
value, and garden-linux will create the volume to grow only up to that
size. One strategy to determine that value would be to use the maximum size
of the ephemeral disk, less the size of the BOSH-deployed packages (for the
executor, currently around 1.3 GB, including the untarred cflinuxfs2
rootfs), less the size allocated to the executor cache in the
'diego.executor.max_cache_size_in_bytes' property (which currently defaults
to 10GB).

Best,
Eric


Re: I'm getting different x_forwarded_for in my Gorouter access logs depending on what browser/cli-tool I use.

Simon Johansson <simon@...>
 

Chris, I now know why this is happening.

In ServeHTTP in reverseproxy in the httputil package[1] we make a copy of
the incoming request for the outgoing request[2]. We then check[3] if the
incoming request have any "Hop-by-hop"[4] headers are set.

If they are set we assign the outgoing requests headers to a new Header
object[5], deep copy all the headers into it from the incoming request and
delete the Hop-by-hop headers.

After this we update the X-Forwarded-For header[6].

This effectively means that if the incoming request doesn't container any
of the hop-by-hop headers we will in step [6] update the incoming requests
X-Forwarded-For header as outrec.Header is a reference to rec.Header. But
in the case we do have a hop-by-hop header the inrec.Headers,
outrec.Headers will not be a reference and thus we get different logs when
we enter the defer in gorouter/proxy/proxy.go[7]

I have emailed Brad Fitzpatrick who was involved in writing the reverse
proxy to see if this is expected behaviour, to me it seems like a bug?

[1]
https://github.com/golang/go/blob/release-branch.go1.4/src/net/http/httputil/reverseproxy.go
[2]
https://github.com/golang/go/blob/release-branch.go1.4/src/net/http/httputil/reverseproxy.go#L110
[3]
https://github.com/golang/go/blob/release-branch.go1.4/src/net/http/httputil/reverseproxy.go#L125
[4]
https://github.com/golang/go/blob/release-branch.go1.4/src/net/http/httputil/reverseproxy.go#L92
[5]
https://github.com/golang/go/blob/release-branch.go1.4/src/net/http/httputil/reverseproxy.go#L127
[6]
https://github.com/golang/go/blob/release-branch.go1.4/src/net/http/httputil/reverseproxy.go#L142
[7] https://github.com/cloudfoundry/gorouter/blob/master/proxy/proxy.go#L149

On Thu, Aug 20, 2015 at 6:05 PM, Christopher Piraino <cpiraino(a)pivotal.io>
wrote:

Simon,

That is very interesting, nice investigation. I would have to do a little
digging to try and see what exactly is going on, but the fact that the
GoRouter is not setting the "X-Forwarded-For" header when keep-alive is
enabled sounds suspiciously similar to an issue with HaProxy
<http://comments.gmane.org/gmane.comp.web.haproxy/5419> where it was only
adding the x-forwarded-proto on the first request for a connection. Not
sure if this is actually similar though.

I took a quick look through golang's reverseproxy code and nothing jumped
out at me, I added a story in our backlog
<https://www.pivotaltracker.com/story/show/101692478> to investigate
further.

- Chris Piraino

On Thu, Aug 20, 2015 at 7:20 AM, Simon Johansson <simon(a)simonjohansson.com
wrote:
Disregarding F5/HAProxy comletely and going directly(10.230.18.134) to a
Gorouter also yields different results depending on the client.

$ curl -H "host:cf-env.domain.com" 10.230.18.134
$ wget --header=host:cf-env.domain.com -qO- 10.230.18.134

Results in the following gorouter logs

cf-env.domain.com - [20/08/2015:13:32:57 +0000] "GET / HTTP/1.1" 200 0
5576 "-" "curl/7.43.0" 172.21.27.221:41193
x_forwarded_for:"172.21.27.221" ...

cf-env.domain.com - [20/08/2015:13:35:47 +0000] "GET / HTTP/1.1" 200 0
5655 "-" "Wget/1.16.3 (linux-gnu)" 172.21.27.221:41218
x_forwarded_for:"-" ....

I.e when using curl the x_forwarded_for header is set, but when using
wget its not.

Doing a tcpdump on the gorouter we get these headers for the incoming
request.

GET / HTTP/1.1
host:cf-env.domain.com
User-Agent: curl/7.43.0
Accept: */*

GET / HTTP/1.1
User-Agent: Wget/1.16.3 (linux-gnu)
Accept: */*
Accept-Encoding: identity
host: cf-env.domain.com
Connection: Keep-Alive

As we can see there is two(well, three if you cound User-Agent) differing
headers, the wget request have the Accept-Encoding and Connection headers
whereas the curl request does not.

So, when going directly to a gorouter with curl(that previously set the
x_forwarded_for header) and setting the Connection header to Keep-Alive we
get this

$ curl -H "host:cf-env.domain.com" -H "Connection: Keep-Alive"
10.230.18.134

GET / HTTP/1.1
host:cf-env.domain.com
User-Agent: curl/7.43.0
Accept: */*
Connection: Keep-Alive

cf-env.domain.com - [20/08/2015:13:46:25 +0000] "GET /curl-keep-alive
HTTP/1.1" 404 0 18 "-" "curl/7.43.0" 172.21.27.221:41313
x_forwarded_for:"-" ...

So, Keep-Alive seems to be the difference between getting x_forwarded_for
set or not in this case.

Interestingly, when hitting an app with either wget or curl via
HAProxy(that both logs x_forwarded_for: sourceIP, HAProxyIP) we can see the
headers for the requests on the Gorouter have Connection unset.

$ # On my machine
$ curl -H "host:cf-env.domain.com" 10.230.18.68/curl
$ curl -H "host:cf-env.domain.com" -H "Connection: Keep-Alive"
10.230.18.68/curl-keep-alive-set
$ wget --header=host:cf-env.domain.com -qO- 10.230.18.68/wget

$ # Tcpdump on HAProxy
GET /curl HTTP/1.1
host: cf-env.domain.com
User-Agent: curl/7.43.0
Accept: */*

GET /curl-keep-alive-set HTTP/1.1
host: cf-env.domain.com
User-Agent: curl/7.43.0
Accept: */*
Connection: Keep-Alive

GET /wget HTTP/1.1
User-Agent: Wget/1.16.3 (linux-gnu)
Accept: */*
Accept-Encoding: identity
host: cf-env.domain.com
Connection: Keep-Alive


$ # Tcpdump on Gorouter
GET /curl HTTP/1.1
User-Agent: curl/7.43.0
Accept: */*
host: cf-env.domain.com
X-Forwarded-Proto: http
X-Forwarded-For: 172.21.27.221

GET /curl-keep-alive-set HTTP/1.1
User-Agent: curl/7.43.0
Accept: */*
host: cf-env.domain.com
X-Forwarded-Proto: http
X-Forwarded-For: 172.21.27.221

GET /wget HTTP/1.1
User-Agent: Wget/1.16.3 (linux-gnu)
Accept: */*
Accept-Encoding: identity
host: cf-env.domain.com
X-Forwarded-Proto: http
X-Forwarded-For: 172.21.27.221

$ cf logs cf-env
cf-env.domain.com - [20/08/2015:14:10:12 +0000] "GET /curl HTTP/1.1" 404
0 18 "-" "curl/7.43.0" 10.230.18.68:60573
x_forwarded_for:"172.21.27.221, 10.230.18.68" ...
cf-env.domain.com - [20/08/2015:14:10:17 +0000] "GET
/curl-keep-alive-set HTTP/1.1" 404 0 18 "-" "curl/7.43.0"
10.230.18.68:60594 x_forwarded_for:"172.21.27.221, 10.230.18.68" ...
cf-env.domain.com - [20/08/2015:14:10:22 +0000] "GET /wget HTTP/1.1" 404
0 18 "-" "Wget/1.16.3 (linux-gnu)" 10.230.18.68:60615
x_forwarded_for:"172.21.27.221, 10.230.18.68" ...

So it seems HAProxy is stripping the Connection header before forwarding
it to a Gorouter and that is why we dont get different x_forwarded_for ips
depending on client whereas we get different when using F5 as it doesnt
strip the Connection header.


Re: I'm getting different x_forwarded_for in my Gorouter access logs depending on what browser/cli-tool I use.

Christopher Piraino <cpiraino@...>
 

Simon,

That is very interesting, nice investigation. I would have to do a little
digging to try and see what exactly is going on, but the fact that the
GoRouter is not setting the "X-Forwarded-For" header when keep-alive is
enabled sounds suspiciously similar to an issue with HaProxy
<http://comments.gmane.org/gmane.comp.web.haproxy/5419> where it was only
adding the x-forwarded-proto on the first request for a connection. Not
sure if this is actually similar though.

I took a quick look through golang's reverseproxy code and nothing jumped
out at me, I added a story in our backlog
<https://www.pivotaltracker.com/story/show/101692478> to investigate
further.

- Chris Piraino

On Thu, Aug 20, 2015 at 7:20 AM, Simon Johansson <simon(a)simonjohansson.com>
wrote:

Disregarding F5/HAProxy comletely and going directly(10.230.18.134) to a
Gorouter also yields different results depending on the client.

$ curl -H "host:cf-env.domain.com" 10.230.18.134
$ wget --header=host:cf-env.domain.com -qO- 10.230.18.134

Results in the following gorouter logs

cf-env.domain.com - [20/08/2015:13:32:57 +0000] "GET / HTTP/1.1" 200 0
5576 "-" "curl/7.43.0" 172.21.27.221:41193
x_forwarded_for:"172.21.27.221" ...

cf-env.domain.com - [20/08/2015:13:35:47 +0000] "GET / HTTP/1.1" 200 0
5655 "-" "Wget/1.16.3 (linux-gnu)" 172.21.27.221:41218
x_forwarded_for:"-" ....

I.e when using curl the x_forwarded_for header is set, but when using wget
its not.

Doing a tcpdump on the gorouter we get these headers for the incoming
request.

GET / HTTP/1.1
host:cf-env.domain.com
User-Agent: curl/7.43.0
Accept: */*

GET / HTTP/1.1
User-Agent: Wget/1.16.3 (linux-gnu)
Accept: */*
Accept-Encoding: identity
host: cf-env.domain.com
Connection: Keep-Alive

As we can see there is two(well, three if you cound User-Agent) differing
headers, the wget request have the Accept-Encoding and Connection headers
whereas the curl request does not.

So, when going directly to a gorouter with curl(that previously set the
x_forwarded_for header) and setting the Connection header to Keep-Alive we
get this

$ curl -H "host:cf-env.domain.com" -H "Connection: Keep-Alive"
10.230.18.134

GET / HTTP/1.1
host:cf-env.domain.com
User-Agent: curl/7.43.0
Accept: */*
Connection: Keep-Alive

cf-env.domain.com - [20/08/2015:13:46:25 +0000] "GET /curl-keep-alive
HTTP/1.1" 404 0 18 "-" "curl/7.43.0" 172.21.27.221:41313
x_forwarded_for:"-" ...

So, Keep-Alive seems to be the difference between getting x_forwarded_for
set or not in this case.

Interestingly, when hitting an app with either wget or curl via
HAProxy(that both logs x_forwarded_for: sourceIP, HAProxyIP) we can see the
headers for the requests on the Gorouter have Connection unset.

$ # On my machine
$ curl -H "host:cf-env.domain.com" 10.230.18.68/curl
$ curl -H "host:cf-env.domain.com" -H "Connection: Keep-Alive"
10.230.18.68/curl-keep-alive-set
$ wget --header=host:cf-env.domain.com -qO- 10.230.18.68/wget

$ # Tcpdump on HAProxy
GET /curl HTTP/1.1
host: cf-env.domain.com
User-Agent: curl/7.43.0
Accept: */*

GET /curl-keep-alive-set HTTP/1.1
host: cf-env.domain.com
User-Agent: curl/7.43.0
Accept: */*
Connection: Keep-Alive

GET /wget HTTP/1.1
User-Agent: Wget/1.16.3 (linux-gnu)
Accept: */*
Accept-Encoding: identity
host: cf-env.domain.com
Connection: Keep-Alive


$ # Tcpdump on Gorouter
GET /curl HTTP/1.1
User-Agent: curl/7.43.0
Accept: */*
host: cf-env.domain.com
X-Forwarded-Proto: http
X-Forwarded-For: 172.21.27.221

GET /curl-keep-alive-set HTTP/1.1
User-Agent: curl/7.43.0
Accept: */*
host: cf-env.domain.com
X-Forwarded-Proto: http
X-Forwarded-For: 172.21.27.221

GET /wget HTTP/1.1
User-Agent: Wget/1.16.3 (linux-gnu)
Accept: */*
Accept-Encoding: identity
host: cf-env.domain.com
X-Forwarded-Proto: http
X-Forwarded-For: 172.21.27.221

$ cf logs cf-env
cf-env.domain.com - [20/08/2015:14:10:12 +0000] "GET /curl HTTP/1.1" 404
0 18 "-" "curl/7.43.0" 10.230.18.68:60573 x_forwarded_for:"172.21.27.221,
10.230.18.68" ...
cf-env.domain.com - [20/08/2015:14:10:17 +0000] "GET /curl-keep-alive-set
HTTP/1.1" 404 0 18 "-" "curl/7.43.0" 10.230.18.68:60594
x_forwarded_for:"172.21.27.221, 10.230.18.68" ...
cf-env.domain.com - [20/08/2015:14:10:22 +0000] "GET /wget HTTP/1.1" 404
0 18 "-" "Wget/1.16.3 (linux-gnu)" 10.230.18.68:60615
x_forwarded_for:"172.21.27.221, 10.230.18.68" ...

So it seems HAProxy is stripping the Connection header before forwarding
it to a Gorouter and that is why we dont get different x_forwarded_for ips
depending on client whereas we get different when using F5 as it doesnt
strip the Connection header.


Re: I'm getting different x_forwarded_for in my Gorouter access logs depending on what browser/cli-tool I use.

Simon Johansson <simon@...>
 

Disregarding F5/HAProxy comletely and going directly(10.230.18.134) to a Gorouter also yields different results depending on the client.

$ curl -H "host:cf-env.domain.com" 10.230.18.134
$ wget --header=host:cf-env.domain.com -qO- 10.230.18.134

Results in the following gorouter logs

cf-env.domain.com - [20/08/2015:13:32:57 +0000] "GET / HTTP/1.1" 200 0 5576 "-" "curl/7.43.0" 172.21.27.221:41193 x_forwarded_for:"172.21.27.221" ...

cf-env.domain.com - [20/08/2015:13:35:47 +0000] "GET / HTTP/1.1" 200 0 5655 "-" "Wget/1.16.3 (linux-gnu)" 172.21.27.221:41218 x_forwarded_for:"-" ....

I.e when using curl the x_forwarded_for header is set, but when using wget its not.

Doing a tcpdump on the gorouter we get these headers for the incoming request.

GET / HTTP/1.1
host:cf-env.domain.com
User-Agent: curl/7.43.0
Accept: */*

GET / HTTP/1.1
User-Agent: Wget/1.16.3 (linux-gnu)
Accept: */*
Accept-Encoding: identity
host: cf-env.domain.com
Connection: Keep-Alive

As we can see there is two(well, three if you cound User-Agent) differing headers, the wget request have the Accept-Encoding and Connection headers whereas the curl request does not.

So, when going directly to a gorouter with curl(that previously set the x_forwarded_for header) and setting the Connection header to Keep-Alive we get this

$ curl -H "host:cf-env.domain.com" -H "Connection: Keep-Alive" 10.230.18.134

GET / HTTP/1.1
host:cf-env.domain.com
User-Agent: curl/7.43.0
Accept: */*
Connection: Keep-Alive

cf-env.domain.com - [20/08/2015:13:46:25 +0000] "GET /curl-keep-alive HTTP/1.1" 404 0 18 "-" "curl/7.43.0" 172.21.27.221:41313 x_forwarded_for:"-" ...

So, Keep-Alive seems to be the difference between getting x_forwarded_for set or not in this case.

Interestingly, when hitting an app with either wget or curl via HAProxy(that both logs x_forwarded_for: sourceIP, HAProxyIP) we can see the headers for the requests on the Gorouter have Connection unset.

$ # On my machine
$ curl -H "host:cf-env.domain.com" 10.230.18.68/curl
$ curl -H "host:cf-env.domain.com" -H "Connection: Keep-Alive" 10.230.18.68/curl-keep-alive-set
$ wget --header=host:cf-env.domain.com -qO- 10.230.18.68/wget

$ # Tcpdump on HAProxy
GET /curl HTTP/1.1
host: cf-env.domain.com
User-Agent: curl/7.43.0
Accept: */*

GET /curl-keep-alive-set HTTP/1.1
host: cf-env.domain.com
User-Agent: curl/7.43.0
Accept: */*
Connection: Keep-Alive

GET /wget HTTP/1.1
User-Agent: Wget/1.16.3 (linux-gnu)
Accept: */*
Accept-Encoding: identity
host: cf-env.domain.com
Connection: Keep-Alive


$ # Tcpdump on Gorouter
GET /curl HTTP/1.1
User-Agent: curl/7.43.0
Accept: */*
host: cf-env.domain.com
X-Forwarded-Proto: http
X-Forwarded-For: 172.21.27.221

GET /curl-keep-alive-set HTTP/1.1
User-Agent: curl/7.43.0
Accept: */*
host: cf-env.domain.com
X-Forwarded-Proto: http
X-Forwarded-For: 172.21.27.221

GET /wget HTTP/1.1
User-Agent: Wget/1.16.3 (linux-gnu)
Accept: */*
Accept-Encoding: identity
host: cf-env.domain.com
X-Forwarded-Proto: http
X-Forwarded-For: 172.21.27.221

$ cf logs cf-env
cf-env.domain.com - [20/08/2015:14:10:12 +0000] "GET /curl HTTP/1.1" 404 0 18 "-" "curl/7.43.0" 10.230.18.68:60573 x_forwarded_for:"172.21.27.221, 10.230.18.68" ...
cf-env.domain.com - [20/08/2015:14:10:17 +0000] "GET /curl-keep-alive-set HTTP/1.1" 404 0 18 "-" "curl/7.43.0" 10.230.18.68:60594 x_forwarded_for:"172.21.27.221, 10.230.18.68" ...
cf-env.domain.com - [20/08/2015:14:10:22 +0000] "GET /wget HTTP/1.1" 404 0 18 "-" "Wget/1.16.3 (linux-gnu)" 10.230.18.68:60615 x_forwarded_for:"172.21.27.221, 10.230.18.68" ...

So it seems HAProxy is stripping the Connection header before forwarding it to a Gorouter and that is why we dont get different x_forwarded_for ips depending on client whereas we get different when using F5 as it doesnt strip the Connection header.

8081 - 8100 of 9417