Re: I'm getting different x_forwarded_for in my Gorouter access logs depending on what browser/cli-tool I use.


Simon Johansson <simon@...>
 

Disregarding F5/HAProxy comletely and going directly(10.230.18.134) to a Gorouter also yields different results depending on the client.

$ curl -H "host:cf-env.domain.com" 10.230.18.134
$ wget --header=host:cf-env.domain.com -qO- 10.230.18.134

Results in the following gorouter logs

cf-env.domain.com - [20/08/2015:13:32:57 +0000] "GET / HTTP/1.1" 200 0 5576 "-" "curl/7.43.0" 172.21.27.221:41193 x_forwarded_for:"172.21.27.221" ...

cf-env.domain.com - [20/08/2015:13:35:47 +0000] "GET / HTTP/1.1" 200 0 5655 "-" "Wget/1.16.3 (linux-gnu)" 172.21.27.221:41218 x_forwarded_for:"-" ....

I.e when using curl the x_forwarded_for header is set, but when using wget its not.

Doing a tcpdump on the gorouter we get these headers for the incoming request.

GET / HTTP/1.1
host:cf-env.domain.com
User-Agent: curl/7.43.0
Accept: */*

GET / HTTP/1.1
User-Agent: Wget/1.16.3 (linux-gnu)
Accept: */*
Accept-Encoding: identity
host: cf-env.domain.com
Connection: Keep-Alive

As we can see there is two(well, three if you cound User-Agent) differing headers, the wget request have the Accept-Encoding and Connection headers whereas the curl request does not.

So, when going directly to a gorouter with curl(that previously set the x_forwarded_for header) and setting the Connection header to Keep-Alive we get this

$ curl -H "host:cf-env.domain.com" -H "Connection: Keep-Alive" 10.230.18.134

GET / HTTP/1.1
host:cf-env.domain.com
User-Agent: curl/7.43.0
Accept: */*
Connection: Keep-Alive

cf-env.domain.com - [20/08/2015:13:46:25 +0000] "GET /curl-keep-alive HTTP/1.1" 404 0 18 "-" "curl/7.43.0" 172.21.27.221:41313 x_forwarded_for:"-" ...

So, Keep-Alive seems to be the difference between getting x_forwarded_for set or not in this case.

Interestingly, when hitting an app with either wget or curl via HAProxy(that both logs x_forwarded_for: sourceIP, HAProxyIP) we can see the headers for the requests on the Gorouter have Connection unset.

$ # On my machine
$ curl -H "host:cf-env.domain.com" 10.230.18.68/curl
$ curl -H "host:cf-env.domain.com" -H "Connection: Keep-Alive" 10.230.18.68/curl-keep-alive-set
$ wget --header=host:cf-env.domain.com -qO- 10.230.18.68/wget

$ # Tcpdump on HAProxy
GET /curl HTTP/1.1
host: cf-env.domain.com
User-Agent: curl/7.43.0
Accept: */*

GET /curl-keep-alive-set HTTP/1.1
host: cf-env.domain.com
User-Agent: curl/7.43.0
Accept: */*
Connection: Keep-Alive

GET /wget HTTP/1.1
User-Agent: Wget/1.16.3 (linux-gnu)
Accept: */*
Accept-Encoding: identity
host: cf-env.domain.com
Connection: Keep-Alive


$ # Tcpdump on Gorouter
GET /curl HTTP/1.1
User-Agent: curl/7.43.0
Accept: */*
host: cf-env.domain.com
X-Forwarded-Proto: http
X-Forwarded-For: 172.21.27.221

GET /curl-keep-alive-set HTTP/1.1
User-Agent: curl/7.43.0
Accept: */*
host: cf-env.domain.com
X-Forwarded-Proto: http
X-Forwarded-For: 172.21.27.221

GET /wget HTTP/1.1
User-Agent: Wget/1.16.3 (linux-gnu)
Accept: */*
Accept-Encoding: identity
host: cf-env.domain.com
X-Forwarded-Proto: http
X-Forwarded-For: 172.21.27.221

$ cf logs cf-env
cf-env.domain.com - [20/08/2015:14:10:12 +0000] "GET /curl HTTP/1.1" 404 0 18 "-" "curl/7.43.0" 10.230.18.68:60573 x_forwarded_for:"172.21.27.221, 10.230.18.68" ...
cf-env.domain.com - [20/08/2015:14:10:17 +0000] "GET /curl-keep-alive-set HTTP/1.1" 404 0 18 "-" "curl/7.43.0" 10.230.18.68:60594 x_forwarded_for:"172.21.27.221, 10.230.18.68" ...
cf-env.domain.com - [20/08/2015:14:10:22 +0000] "GET /wget HTTP/1.1" 404 0 18 "-" "Wget/1.16.3 (linux-gnu)" 10.230.18.68:60615 x_forwarded_for:"172.21.27.221, 10.230.18.68" ...

So it seems HAProxy is stripping the Connection header before forwarding it to a Gorouter and that is why we dont get different x_forwarded_for ips depending on client whereas we get different when using F5 as it doesnt strip the Connection header.

Join cf-dev@lists.cloudfoundry.org to automatically receive all group messages.