Disregarding F5/HAProxy comletely and going directly(10.230.18.134) to a Gorouter also yields different results depending on the client.
$ curl -H "host:cf-env.domain.com" 10.230.18.134
$ wget --header=host:cf-env.domain.com -qO- 10.230.18.134
Results in the following gorouter logs
cf-env.domain.com - [20/08/2015:13:32:57 +0000] "GET / HTTP/1.1" 200 0 5576 "-" "curl/7.43.0" 172.21.27.221:41193 x_forwarded_for:"172.21.27.221" ...
cf-env.domain.com - [20/08/2015:13:35:47 +0000] "GET / HTTP/1.1" 200 0 5655 "-" "Wget/1.16.3 (linux-gnu)" 172.21.27.221:41218 x_forwarded_for:"-" ....
I.e when using curl the x_forwarded_for header is set, but when using wget its not.
Doing a tcpdump on the gorouter we get these headers for the incoming request.
GET / HTTP/1.1
host:cf-env.domain.com
User-Agent: curl/7.43.0
Accept: */*
GET / HTTP/1.1
User-Agent: Wget/1.16.3 (linux-gnu)
Accept: */*
Accept-Encoding: identity
host: cf-env.domain.com
Connection: Keep-Alive
As we can see there is two(well, three if you cound User-Agent) differing headers, the wget request have the Accept-Encoding and Connection headers whereas the curl request does not.
So, when going directly to a gorouter with curl(that previously set the x_forwarded_for header) and setting the Connection header to Keep-Alive we get this
$ curl -H "host:cf-env.domain.com" -H "Connection: Keep-Alive" 10.230.18.134
GET / HTTP/1.1
host:cf-env.domain.com
User-Agent: curl/7.43.0
Accept: */*
Connection: Keep-Alive
cf-env.domain.com - [20/08/2015:13:46:25 +0000] "GET /curl-keep-alive HTTP/1.1" 404 0 18 "-" "curl/7.43.0" 172.21.27.221:41313 x_forwarded_for:"-" ...
So, Keep-Alive seems to be the difference between getting x_forwarded_for set or not in this case.
Interestingly, when hitting an app with either wget or curl via HAProxy(that both logs x_forwarded_for: sourceIP, HAProxyIP) we can see the headers for the requests on the Gorouter have Connection unset.
$ # On my machine
$ curl -H "host:cf-env.domain.com" 10.230.18.68/curl
$ curl -H "host:cf-env.domain.com" -H "Connection: Keep-Alive" 10.230.18.68/curl-keep-alive-set
$ wget --header=host:cf-env.domain.com -qO- 10.230.18.68/wget
$ # Tcpdump on HAProxy
GET /curl HTTP/1.1
host: cf-env.domain.com
User-Agent: curl/7.43.0
Accept: */*
GET /curl-keep-alive-set HTTP/1.1
host: cf-env.domain.com
User-Agent: curl/7.43.0
Accept: */*
Connection: Keep-Alive
GET /wget HTTP/1.1
User-Agent: Wget/1.16.3 (linux-gnu)
Accept: */*
Accept-Encoding: identity
host: cf-env.domain.com
Connection: Keep-Alive
$ # Tcpdump on Gorouter
GET /curl HTTP/1.1
User-Agent: curl/7.43.0
Accept: */*
host: cf-env.domain.com
X-Forwarded-Proto: http
X-Forwarded-For: 172.21.27.221
GET /curl-keep-alive-set HTTP/1.1
User-Agent: curl/7.43.0
Accept: */*
host: cf-env.domain.com
X-Forwarded-Proto: http
X-Forwarded-For: 172.21.27.221
GET /wget HTTP/1.1
User-Agent: Wget/1.16.3 (linux-gnu)
Accept: */*
Accept-Encoding: identity
host: cf-env.domain.com
X-Forwarded-Proto: http
X-Forwarded-For: 172.21.27.221
$ cf logs cf-env
cf-env.domain.com - [20/08/2015:14:10:12 +0000] "GET /curl HTTP/1.1" 404 0 18 "-" "curl/7.43.0" 10.230.18.68:60573 x_forwarded_for:"172.21.27.221, 10.230.18.68" ...
cf-env.domain.com - [20/08/2015:14:10:17 +0000] "GET /curl-keep-alive-set HTTP/1.1" 404 0 18 "-" "curl/7.43.0" 10.230.18.68:60594 x_forwarded_for:"172.21.27.221, 10.230.18.68" ...
cf-env.domain.com - [20/08/2015:14:10:22 +0000] "GET /wget HTTP/1.1" 404 0 18 "-" "Wget/1.16.3 (linux-gnu)" 10.230.18.68:60615 x_forwarded_for:"172.21.27.221, 10.230.18.68" ...
So it seems HAProxy is stripping the Connection header before forwarding it to a Gorouter and that is why we dont get different x_forwarded_for ips depending on client whereas we get different when using F5 as it doesnt strip the Connection header.
