Howdie!

We have some devs who want to be able to trace a request troughout their
applications.

user -> a -> b -> c
|
|-> d -> e

When a user makes a request to "a" uuid is generated inside the app, and
the request to "b" from "a" will set a header(call it WAKAWAKA to uuid),
"b"'s call will passthrough WAKAWAKA to "c" and "d", "d" will passthrough
WAKAWAKA to "e". Etc.

We aggregate all RTR logs into ELK so it would be super helpful to them to
be able to filter on WAKAWAKA and get all the access logs(and app logs
aswell, they mostly use GELF so its easy for them to add whatewher field
they want) from the services involved.

I had a quick peek at the gorouter(
https://github.com/cloudfoundry/gorouter/blob/76668f5818ea8c089ff52a14fcdfbf703c8e8767/access_log/access_log_record.go#L40)
and it seems like this should be a simple PR.

1. To gorouter.yml add
passthrough_headers:
- WAKAWAKA
- X-Random-Header

2. In makeRecord at the bottom add something like(in psuedo)

data = {}
for header in passthrough_headers:
header_val = r.FormatRequestHeader("X-Forwarded-For")
if header_val:
passthrough_headers[header] = header_val

if data:
fmt.Fprintf(b, data.to_stringified_json())

That would yield a log line like
blurgh.dev.cf.private.domain.com - [21/07/2015:10:17:05 +0000] "GET
/statements?ascending=true&since=2015-06-30T14%3A10%3A03.078Z&skipStatementId=30a88204-0779-4385-9859-e4aabd30baf0
HTTP/1.1" 200 0 17 "-" "NING/1.0" 10.230.31.2:46204 x_forwarded_for:"-"
vcap_request_id:1e58195a-cde6-4afd-7f03-43061c9ea91c
response_time:0.004927106 app_id:9784cd03-050d-4b74-9e90-5f17134a3f08
{"WAKAWAKA": "Space is the place", "X-Random-Header": "Once upon a midnight
dreary, while I pondered weak and weary"}

The reason for a stringified JSON is to make it easy to parse with logstash
or other loganalysis tools.

Before I spend time implementing, is this something you would merge?

Hello Simon,

The X-Cf-Requestid header already provides a uuid. Couldn't app "a" add the
value of X-Cf-Requestid to a header of your choosing? Call it WAKAWAKA or
X-Random-Header, but it doesn't need to be a platform standard. Wouldn't a
search for the value of X-Cf-Requestid then provide the desired results?

Thank you,
Shannon

Shannon Coen
Product Manager, Cloud Foundry
Pivotal, Inc.

On Tue, Jul 21, 2015 at 6:06 AM, Simon Johansson <simon(a)simonjohansson.com>
wrote:

Howdie!

We have some devs who want to be able to trace a request troughout their
applications.

user -> a -> b -> c
|
|-> d -> e

When a user makes a request to "a" uuid is generated inside the app, and
the request to "b" from "a" will set a header(call it WAKAWAKA to uuid),
"b"'s call will passthrough WAKAWAKA to "c" and "d", "d" will passthrough
WAKAWAKA to "e". Etc.

We aggregate all RTR logs into ELK so it would be super helpful to them to
be able to filter on WAKAWAKA and get all the access logs(and app logs
aswell, they mostly use GELF so its easy for them to add whatewher field
they want) from the services involved.

I had a quick peek at the gorouter(
https://github.com/cloudfoundry/gorouter/blob/76668f5818ea8c089ff52a14fcdfbf703c8e8767/access_log/access_log_record.go#L40)
and it seems like this should be a simple PR.

1. To gorouter.yml add
passthrough_headers:
- WAKAWAKA
- X-Random-Header

2. In makeRecord at the bottom add something like(in psuedo)

data = {}
for header in passthrough_headers:
header_val = r.FormatRequestHeader("X-Forwarded-For")
if header_val:
passthrough_headers[header] = header_val

if data:
fmt.Fprintf(b, data.to_stringified_json())

That would yield a log line like
blurgh.dev.cf.private.domain.com - [21/07/2015:10:17:05 +0000] "GET
/statements?ascending=true&since=2015-06-30T14%3A10%3A03.078Z&skipStatementId=30a88204-0779-4385-9859-e4aabd30baf0
HTTP/1.1" 200 0 17 "-" "NING/1.0" 10.230.31.2:46204 x_forwarded_for:"-"
vcap_request_id:1e58195a-cde6-4afd-7f03-43061c9ea91c
response_time:0.004927106 app_id:9784cd03-050d-4b74-9e90-5f17134a3f08
{"WAKAWAKA": "Space is the place", "X-Random-Header": "Once upon a midnight
dreary, while I pondered weak and weary"}

The reason for a stringified JSON is to make it easy to parse with
logstash or other loganalysis tools.

Before I spend time implementing, is this something you would merge?

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

Sure that header can be used. But then we are adding CF specific stuff into
the implementation of our services, which is something we want to avoid at
all costs.
Also Im not entirely sure if all the libraries we use for Zipkin supports
using custom headers.

All our public apps are fronted by different CDNs, which sets headers that
we might want to store for debugging, so we still need a way to pass those
trough into the log.

I don't see the difference between WAKAWAKA and X-Cf-Requestid. Gorouter
would have to add some header with a uuid for the request. Your apps have
to have logic to pass this header on, so that a log search returns the
original request as well as subsequent requests between apps. Could you
please clarify?

Thank you,

Shannon Coen
Product Manager, Cloud Foundry
Pivotal, Inc.

On Tue, Jul 21, 2015 at 11:09 AM, Simon Johansson <simon(a)simonjohansson.com>
wrote:

Sure that header can be used. But then we are adding CF specific stuff
into the implementation of our services, which is something we want to
avoid at all costs.
Also Im not entirely sure if all the libraries we use for Zipkin supports
using custom headers.

All our public apps are fronted by different CDNs, which sets headers that
we might want to store for debugging, so we still need a way to pass those
trough into the log.

On Tuesday, 21 July 2015, Shannon Coen <scoen(a)pivotal.io> wrote:

Hello Simon,

The X-Cf-Requestid header already provides a uuid. Couldn't app "a" add
the value of X-Cf-Requestid to a header of your choosing? Call it WAKAWAKA
or X-Random-Header, but it doesn't need to be a platform standard. Wouldn't
a search for the value of X-Cf-Requestid then provide the desired results?

Thank you,
Shannon

Shannon Coen
Product Manager, Cloud Foundry
Pivotal, Inc.

On Tue, Jul 21, 2015 at 6:06 AM, Simon Johansson <
simon(a)simonjohansson.com> wrote:

Howdie!

We have some devs who want to be able to trace a request troughout their
applications.

user -> a -> b -> c
|
|-> d -> e

When a user makes a request to "a" uuid is generated inside the app, and
the request to "b" from "a" will set a header(call it WAKAWAKA to uuid),
"b"'s call will passthrough WAKAWAKA to "c" and "d", "d" will passthrough
WAKAWAKA to "e". Etc.

We aggregate all RTR logs into ELK so it would be super helpful to them
to be able to filter on WAKAWAKA and get all the access logs(and app logs
aswell, they mostly use GELF so its easy for them to add whatewher field
they want) from the services involved.

I had a quick peek at the gorouter(
https://github.com/cloudfoundry/gorouter/blob/76668f5818ea8c089ff52a14fcdfbf703c8e8767/access_log/access_log_record.go#L40)
and it seems like this should be a simple PR.

1. To gorouter.yml add
passthrough_headers:
- WAKAWAKA
- X-Random-Header

2. In makeRecord at the bottom add something like(in psuedo)

data = {}
for header in passthrough_headers:
header_val = r.FormatRequestHeader("X-Forwarded-For")
if header_val:
passthrough_headers[header] = header_val

if data:
fmt.Fprintf(b, data.to_stringified_json())

That would yield a log line like
blurgh.dev.cf.private.domain.com - [21/07/2015:10:17:05 +0000] "GET
/statements?ascending=true&since=2015-06-30T14%3A10%3A03.078Z&skipStatementId=30a88204-0779-4385-9859-e4aabd30baf0
HTTP/1.1" 200 0 17 "-" "NING/1.0" 10.230.31.2:46204 x_forwarded_for:"-"
vcap_request_id:1e58195a-cde6-4afd-7f03-43061c9ea91c
response_time:0.004927106 app_id:9784cd03-050d-4b74-9e90-5f17134a3f08
{"WAKAWAKA": "Space is the place", "X-Random-Header": "Once upon a midnight
dreary, while I pondered weak and weary"}

The reason for a stringified JSON is to make it easy to parse with
logstash or other loganalysis tools.

Before I spend time implementing, is this something you would merge?

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

WAKAWAKA is just an example
But the difference is that WAKAWAKA is not platform specific whereas
X-Cf-Requestid is. If we want to move our app to Heroku, or a VM, or
whatewher we have platform specific implementation details in our
app(namely we rely on a header that is not there anymore). But that is not
the point of this thread.

The point is,

say we have an app that is fronted by a CDN, and the CDN sets the
X-Im-a-shark header with some value that we are interested to see in our
logs. The easiest way to achivie this without having to implement it into
our own apps is just to tell the Gorouter that it should append the value
of that header into the string that it logs so the event that flow via
Doppler and ultimately into cf logs/ELK will contain that value.

The reason why this would be such a valuable feature for us is that we dont
have to do anything. CF already provide the out of the box facility to give
us routing logs, so if we can piggy back on that for what we are interested
in we dont have to add libraries to our apps to log interesting headers on
the side.

So if you move to Heroku you will be able to change the platform ??? I
don't think so..
So you can put the new header from your app or at least the runtime
buildpack or docker.

Your application should be platform dependent that's why you need to
implement this header into your application to avoid any lock-in or forking
issue.

On Wed, Jul 22, 2015 at 6:27 PM, Simon Johansson <simon(a)simonjohansson.com>
wrote:

WAKAWAKA is just an example
But the difference is that WAKAWAKA is not platform specific whereas
X-Cf-Requestid is. If we want to move our app to Heroku, or a VM, or
whatewher we have platform specific implementation details in our
app(namely we rely on a header that is not there anymore). But that is not
the point of this thread.

The point is,

say we have an app that is fronted by a CDN, and the CDN sets the
X-Im-a-shark header with some value that we are interested to see in our
logs. The easiest way to achivie this without having to implement it into
our own apps is just to tell the Gorouter that it should append the value
of that header into the string that it logs so the event that flow via
Doppler and ultimately into cf logs/ELK will contain that value.

The reason why this would be such a valuable feature for us is that we
dont have to do anything. CF already provide the out of the box facility to
give us routing logs, so if we can piggy back on that for what we are
interested in we dont have to add libraries to our apps to log interesting
headers on the side.



On Wed, Jul 22, 2015 at 12:45 AM, Shannon Coen <scoen(a)pivotal.io> wrote:

I don't see the difference between WAKAWAKA and X-Cf-Requestid. Gorouter
would have to add some header with a uuid for the request. Your apps have
to have logic to pass this header on, so that a log search returns the
original request as well as subsequent requests between apps. Could you
please clarify?

Thank you,

Shannon Coen
Product Manager, Cloud Foundry
Pivotal, Inc.

On Tue, Jul 21, 2015 at 11:09 AM, Simon Johansson <
simon(a)simonjohansson.com> wrote:

Sure that header can be used. But then we are adding CF specific stuff
into the implementation of our services, which is something we want to
avoid at all costs.
Also Im not entirely sure if all the libraries we use for Zipkin
supports using custom headers.

All our public apps are fronted by different CDNs, which sets headers
that we might want to store for debugging, so we still need a way to pass
those trough into the log.

On Tuesday, 21 July 2015, Shannon Coen <scoen(a)pivotal.io> wrote:

Hello Simon,

The X-Cf-Requestid header already provides a uuid. Couldn't app "a" add
the value of X-Cf-Requestid to a header of your choosing? Call it WAKAWAKA
or X-Random-Header, but it doesn't need to be a platform standard. Wouldn't
a search for the value of X-Cf-Requestid then provide the desired results?

Thank you,
Shannon

Shannon Coen
Product Manager, Cloud Foundry
Pivotal, Inc.

On Tue, Jul 21, 2015 at 6:06 AM, Simon Johansson <
simon(a)simonjohansson.com> wrote:

Howdie!

We have some devs who want to be able to trace a request troughout
their applications.

user -> a -> b -> c
|
|-> d -> e

When a user makes a request to "a" uuid is generated inside the app,
and the request to "b" from "a" will set a header(call it WAKAWAKA to
uuid), "b"'s call will passthrough WAKAWAKA to "c" and "d", "d" will
passthrough WAKAWAKA to "e". Etc.

We aggregate all RTR logs into ELK so it would be super helpful to
them to be able to filter on WAKAWAKA and get all the access logs(and app
logs aswell, they mostly use GELF so its easy for them to add whatewher
field they want) from the services involved.

I had a quick peek at the gorouter(
https://github.com/cloudfoundry/gorouter/blob/76668f5818ea8c089ff52a14fcdfbf703c8e8767/access_log/access_log_record.go#L40)
and it seems like this should be a simple PR.

1. To gorouter.yml add
passthrough_headers:
- WAKAWAKA
- X-Random-Header

2. In makeRecord at the bottom add something like(in psuedo)

data = {}
for header in passthrough_headers:
header_val = r.FormatRequestHeader("X-Forwarded-For")
if header_val:
passthrough_headers[header] = header_val

if data:
fmt.Fprintf(b, data.to_stringified_json())

That would yield a log line like
blurgh.dev.cf.private.domain.com - [21/07/2015:10:17:05 +0000] "GET
/statements?ascending=true&since=2015-06-30T14%3A10%3A03.078Z&skipStatementId=30a88204-0779-4385-9859-e4aabd30baf0
HTTP/1.1" 200 0 17 "-" "NING/1.0" 10.230.31.2:46204
x_forwarded_for:"-" vcap_request_id:1e58195a-cde6-4afd-7f03-43061c9ea91c
response_time:0.004927106 app_id:9784cd03-050d-4b74-9e90-5f17134a3f08
{"WAKAWAKA": "Space is the place", "X-Random-Header": "Once upon a midnight
dreary, while I pondered weak and weary"}

The reason for a stringified JSON is to make it easy to parse with
logstash or other loganalysis tools.

Before I spend time implementing, is this something you would merge?

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

I feel that we are stearing this conversation into the wrong direction.

Gist it, I would like to with cf logs app / firehose-to-syslog see my
RTR-logs as

blurgh.dev.cf.private.domain.com - blablablab
vcap_request_id:1e58195a-cde6-4afd-7f03-43061c9ea91c
response_time:0.004927106 app_id:9784cd03-050d-4b74-9e90-5f17134a3f08
*{"X-Random-Header": "Once upon a midnight dreary, while I pondered weak
and weary", "Another-Header": "wryyyy"}*

By being able to tell Gorouter what headers Im interested in via the
manifest
logging:
add_extra_headers_if_available:
- X-Random-Header
- Another-Header
- This-header-will-only-be-in-1%-of-the-requests

Is this something you would be interested in mergin, I dont want to
implement it if there is no chance of it being merged.


On Wed, Jul 22, 2015 at 11:30 AM, Gwenn Etourneau <getourneau(a)pivotal.io>
wrote:

So if you move to Heroku you will be able to change the platform ??? I
don't think so..
So you can put the new header from your app or at least the runtime
buildpack or docker.

Your application should be platform dependent that's why you need to
implement this header into your application to avoid any lock-in or forking
issue.

On Wed, Jul 22, 2015 at 6:27 PM, Simon Johansson <simon(a)simonjohansson.com

wrote:

WAKAWAKA is just an example
But the difference is that WAKAWAKA is not platform specific whereas
X-Cf-Requestid is. If we want to move our app to Heroku, or a VM, or
whatewher we have platform specific implementation details in our
app(namely we rely on a header that is not there anymore). But that is not
the point of this thread.

The point is,

say we have an app that is fronted by a CDN, and the CDN sets the
X-Im-a-shark header with some value that we are interested to see in our
logs. The easiest way to achivie this without having to implement it into
our own apps is just to tell the Gorouter that it should append the value
of that header into the string that it logs so the event that flow via
Doppler and ultimately into cf logs/ELK will contain that value.

The reason why this would be such a valuable feature for us is that we
dont have to do anything. CF already provide the out of the box facility to
give us routing logs, so if we can piggy back on that for what we are
interested in we dont have to add libraries to our apps to log interesting
headers on the side.



On Wed, Jul 22, 2015 at 12:45 AM, Shannon Coen <scoen(a)pivotal.io> wrote:

I don't see the difference between WAKAWAKA and X-Cf-Requestid. Gorouter
would have to add some header with a uuid for the request. Your apps have
to have logic to pass this header on, so that a log search returns the
original request as well as subsequent requests between apps. Could you
please clarify?

Thank you,

Shannon Coen
Product Manager, Cloud Foundry
Pivotal, Inc.

On Tue, Jul 21, 2015 at 11:09 AM, Simon Johansson <
simon(a)simonjohansson.com> wrote:

Sure that header can be used. But then we are adding CF specific stuff
into the implementation of our services, which is something we want to
avoid at all costs.
Also Im not entirely sure if all the libraries we use for Zipkin
supports using custom headers.

All our public apps are fronted by different CDNs, which sets headers
that we might want to store for debugging, so we still need a way to pass
those trough into the log.

On Tuesday, 21 July 2015, Shannon Coen <scoen(a)pivotal.io> wrote:

Hello Simon,

The X-Cf-Requestid header already provides a uuid. Couldn't app "a"
add the value of X-Cf-Requestid to a header of your choosing? Call it
WAKAWAKA or X-Random-Header, but it doesn't need to be a platform standard.
Wouldn't a search for the value of X-Cf-Requestid then provide the desired
results?

Thank you,
Shannon

Shannon Coen
Product Manager, Cloud Foundry
Pivotal, Inc.

On Tue, Jul 21, 2015 at 6:06 AM, Simon Johansson <
simon(a)simonjohansson.com> wrote:

Howdie!

We have some devs who want to be able to trace a request troughout
their applications.

user -> a -> b -> c
|
|-> d -> e

When a user makes a request to "a" uuid is generated inside the app,
and the request to "b" from "a" will set a header(call it WAKAWAKA to
uuid), "b"'s call will passthrough WAKAWAKA to "c" and "d", "d" will
passthrough WAKAWAKA to "e". Etc.

We aggregate all RTR logs into ELK so it would be super helpful to
them to be able to filter on WAKAWAKA and get all the access logs(and app
logs aswell, they mostly use GELF so its easy for them to add whatewher
field they want) from the services involved.

I had a quick peek at the gorouter(
https://github.com/cloudfoundry/gorouter/blob/76668f5818ea8c089ff52a14fcdfbf703c8e8767/access_log/access_log_record.go#L40)
and it seems like this should be a simple PR.

1. To gorouter.yml add
passthrough_headers:
- WAKAWAKA
- X-Random-Header

2. In makeRecord at the bottom add something like(in psuedo)

data = {}
for header in passthrough_headers:
header_val = r.FormatRequestHeader("X-Forwarded-For")
if header_val:
passthrough_headers[header] = header_val

if data:
fmt.Fprintf(b, data.to_stringified_json())

That would yield a log line like
blurgh.dev.cf.private.domain.com - [21/07/2015:10:17:05 +0000] "GET
/statements?ascending=true&since=2015-06-30T14%3A10%3A03.078Z&skipStatementId=30a88204-0779-4385-9859-e4aabd30baf0
HTTP/1.1" 200 0 17 "-" "NING/1.0" 10.230.31.2:46204
x_forwarded_for:"-" vcap_request_id:1e58195a-cde6-4afd-7f03-43061c9ea91c
response_time:0.004927106 app_id:9784cd03-050d-4b74-9e90-5f17134a3f08
{"WAKAWAKA": "Space is the place", "X-Random-Header": "Once upon a midnight
dreary, while I pondered weak and weary"}

The reason for a stringified JSON is to make it easy to parse with
logstash or other loganalysis tools.

Before I spend time implementing, is this something you would merge?

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

Simon I know what you want todo :) I am shinji62 lol
I am jsut afraid that you want to customize the platform and you will be
stuck with because other don't support custom header.

On Wed, Jul 22, 2015 at 6:36 PM, Simon Johansson <simon(a)simonjohansson.com>
wrote:

I feel that we are stearing this conversation into the wrong direction.

Gist it, I would like to with cf logs app / firehose-to-syslog see my
RTR-logs as

blurgh.dev.cf.private.domain.com - blablablab
vcap_request_id:1e58195a-cde6-4afd-7f03-43061c9ea91c
response_time:0.004927106 app_id:9784cd03-050d-4b74-9e90-5f17134a3f08
*{"X-Random-Header": "Once upon a midnight dreary, while I pondered weak
and weary", "Another-Header": "wryyyy"}*

By being able to tell Gorouter what headers Im interested in via the
manifest
logging:
add_extra_headers_if_available:
- X-Random-Header
- Another-Header
- This-header-will-only-be-in-1%-of-the-requests

Is this something you would be interested in mergin, I dont want to
implement it if there is no chance of it being merged.


On Wed, Jul 22, 2015 at 11:30 AM, Gwenn Etourneau <getourneau(a)pivotal.io>
wrote:

So if you move to Heroku you will be able to change the platform ??? I
don't think so..
So you can put the new header from your app or at least the runtime
buildpack or docker.

Your application should be platform dependent that's why you need to
implement this header into your application to avoid any lock-in or forking
issue.

On Wed, Jul 22, 2015 at 6:27 PM, Simon Johansson <
simon(a)simonjohansson.com> wrote:

WAKAWAKA is just an example
But the difference is that WAKAWAKA is not platform specific whereas
X-Cf-Requestid is. If we want to move our app to Heroku, or a VM, or
whatewher we have platform specific implementation details in our
app(namely we rely on a header that is not there anymore). But that is not
the point of this thread.

The point is,

say we have an app that is fronted by a CDN, and the CDN sets the
X-Im-a-shark header with some value that we are interested to see in our
logs. The easiest way to achivie this without having to implement it into
our own apps is just to tell the Gorouter that it should append the value
of that header into the string that it logs so the event that flow via
Doppler and ultimately into cf logs/ELK will contain that value.

The reason why this would be such a valuable feature for us is that we
dont have to do anything. CF already provide the out of the box facility to
give us routing logs, so if we can piggy back on that for what we are
interested in we dont have to add libraries to our apps to log interesting
headers on the side.



On Wed, Jul 22, 2015 at 12:45 AM, Shannon Coen <scoen(a)pivotal.io> wrote:

I don't see the difference between WAKAWAKA and X-Cf-Requestid.
Gorouter would have to add some header with a uuid for the request. Your
apps have to have logic to pass this header on, so that a log search
returns the original request as well as subsequent requests between apps.
Could you please clarify?

Thank you,

Shannon Coen
Product Manager, Cloud Foundry
Pivotal, Inc.

On Tue, Jul 21, 2015 at 11:09 AM, Simon Johansson <
simon(a)simonjohansson.com> wrote:

Sure that header can be used. But then we are adding CF specific stuff
into the implementation of our services, which is something we want to
avoid at all costs.
Also Im not entirely sure if all the libraries we use for Zipkin
supports using custom headers.

All our public apps are fronted by different CDNs, which sets headers
that we might want to store for debugging, so we still need a way to pass
those trough into the log.

On Tuesday, 21 July 2015, Shannon Coen <scoen(a)pivotal.io> wrote:

Hello Simon,

The X-Cf-Requestid header already provides a uuid. Couldn't app "a"
add the value of X-Cf-Requestid to a header of your choosing? Call it
WAKAWAKA or X-Random-Header, but it doesn't need to be a platform standard.
Wouldn't a search for the value of X-Cf-Requestid then provide the desired
results?

Thank you,
Shannon

Shannon Coen
Product Manager, Cloud Foundry
Pivotal, Inc.

On Tue, Jul 21, 2015 at 6:06 AM, Simon Johansson <
simon(a)simonjohansson.com> wrote:

Howdie!

We have some devs who want to be able to trace a request troughout
their applications.

user -> a -> b -> c
|
|-> d -> e

When a user makes a request to "a" uuid is generated inside the app,
and the request to "b" from "a" will set a header(call it WAKAWAKA to
uuid), "b"'s call will passthrough WAKAWAKA to "c" and "d", "d" will
passthrough WAKAWAKA to "e". Etc.

We aggregate all RTR logs into ELK so it would be super helpful to
them to be able to filter on WAKAWAKA and get all the access logs(and app
logs aswell, they mostly use GELF so its easy for them to add whatewher
field they want) from the services involved.

I had a quick peek at the gorouter(
https://github.com/cloudfoundry/gorouter/blob/76668f5818ea8c089ff52a14fcdfbf703c8e8767/access_log/access_log_record.go#L40)
and it seems like this should be a simple PR.

1. To gorouter.yml add
passthrough_headers:
- WAKAWAKA
- X-Random-Header

2. In makeRecord at the bottom add something like(in psuedo)

data = {}
for header in passthrough_headers:
header_val = r.FormatRequestHeader("X-Forwarded-For")
if header_val:
passthrough_headers[header] = header_val

if data:
fmt.Fprintf(b, data.to_stringified_json())

That would yield a log line like
blurgh.dev.cf.private.domain.com - [21/07/2015:10:17:05 +0000] "GET
/statements?ascending=true&since=2015-06-30T14%3A10%3A03.078Z&skipStatementId=30a88204-0779-4385-9859-e4aabd30baf0
HTTP/1.1" 200 0 17 "-" "NING/1.0" 10.230.31.2:46204
x_forwarded_for:"-" vcap_request_id:1e58195a-cde6-4afd-7f03-43061c9ea91c
response_time:0.004927106 app_id:9784cd03-050d-4b74-9e90-5f17134a3f08
{"WAKAWAKA": "Space is the place", "X-Random-Header": "Once upon a midnight
dreary, while I pondered weak and weary"}

The reason for a stringified JSON is to make it easy to parse with
logstash or other loganalysis tools.

Before I spend time implementing, is this something you would merge?

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

Well, from my point of view the customization is just adding the
flexibility of being able to annotate our logs with what we consider to be
important for our debugging purposes, a feature that is surely interesting
for other parties aswell. We are not interested in adding extra headers
into CF, just pulling headers from incoming requests(where the headers have
ben set elsewhere) to Gorouter into a doppler event. :)

On Wed, Jul 22, 2015 at 11:43 AM, Gwenn Etourneau <getourneau(a)pivotal.io>
wrote:

Simon I know what you want todo :) I am shinji62 lol
I am jsut afraid that you want to customize the platform and you will be
stuck with because other don't support custom header.

On Wed, Jul 22, 2015 at 6:36 PM, Simon Johansson <simon(a)simonjohansson.com

wrote:

I feel that we are stearing this conversation into the wrong direction.

Gist it, I would like to with cf logs app / firehose-to-syslog see my
RTR-logs as

blurgh.dev.cf.private.domain.com - blablablab
vcap_request_id:1e58195a-cde6-4afd-7f03-43061c9ea91c
response_time:0.004927106 app_id:9784cd03-050d-4b74-9e90-5f17134a3f08
*{"X-Random-Header": "Once upon a midnight dreary, while I pondered weak
and weary", "Another-Header": "wryyyy"}*

By being able to tell Gorouter what headers Im interested in via the
manifest
logging:
add_extra_headers_if_available:
- X-Random-Header
- Another-Header
- This-header-will-only-be-in-1%-of-the-requests

Is this something you would be interested in mergin, I dont want to
implement it if there is no chance of it being merged.


On Wed, Jul 22, 2015 at 11:30 AM, Gwenn Etourneau <getourneau(a)pivotal.io>
wrote:

So if you move to Heroku you will be able to change the platform ??? I
don't think so..
So you can put the new header from your app or at least the runtime
buildpack or docker.

Your application should be platform dependent that's why you need to
implement this header into your application to avoid any lock-in or forking
issue.

On Wed, Jul 22, 2015 at 6:27 PM, Simon Johansson <
simon(a)simonjohansson.com> wrote:

WAKAWAKA is just an example
But the difference is that WAKAWAKA is not platform specific whereas
X-Cf-Requestid is. If we want to move our app to Heroku, or a VM, or
whatewher we have platform specific implementation details in our
app(namely we rely on a header that is not there anymore). But that is not
the point of this thread.

The point is,

say we have an app that is fronted by a CDN, and the CDN sets the
X-Im-a-shark header with some value that we are interested to see in our
logs. The easiest way to achivie this without having to implement it into
our own apps is just to tell the Gorouter that it should append the value
of that header into the string that it logs so the event that flow via
Doppler and ultimately into cf logs/ELK will contain that value.

The reason why this would be such a valuable feature for us is that we
dont have to do anything. CF already provide the out of the box facility to
give us routing logs, so if we can piggy back on that for what we are
interested in we dont have to add libraries to our apps to log interesting
headers on the side.



On Wed, Jul 22, 2015 at 12:45 AM, Shannon Coen <scoen(a)pivotal.io>
wrote:

I don't see the difference between WAKAWAKA and X-Cf-Requestid.
Gorouter would have to add some header with a uuid for the request. Your
apps have to have logic to pass this header on, so that a log search
returns the original request as well as subsequent requests between apps.
Could you please clarify?

Thank you,

Shannon Coen
Product Manager, Cloud Foundry
Pivotal, Inc.

On Tue, Jul 21, 2015 at 11:09 AM, Simon Johansson <
simon(a)simonjohansson.com> wrote:

Sure that header can be used. But then we are adding CF specific
stuff into the implementation of our services, which is something we want
to avoid at all costs.
Also Im not entirely sure if all the libraries we use for Zipkin
supports using custom headers.

All our public apps are fronted by different CDNs, which sets headers
that we might want to store for debugging, so we still need a way to pass
those trough into the log.

On Tuesday, 21 July 2015, Shannon Coen <scoen(a)pivotal.io> wrote:

Hello Simon,

The X-Cf-Requestid header already provides a uuid. Couldn't app "a"
add the value of X-Cf-Requestid to a header of your choosing? Call it
WAKAWAKA or X-Random-Header, but it doesn't need to be a platform standard.
Wouldn't a search for the value of X-Cf-Requestid then provide the desired
results?

Thank you,
Shannon

Shannon Coen
Product Manager, Cloud Foundry
Pivotal, Inc.

On Tue, Jul 21, 2015 at 6:06 AM, Simon Johansson <
simon(a)simonjohansson.com> wrote:

Howdie!

We have some devs who want to be able to trace a request troughout
their applications.

user -> a -> b -> c
|
|-> d -> e

When a user makes a request to "a" uuid is generated inside the
app, and the request to "b" from "a" will set a header(call it WAKAWAKA to
uuid), "b"'s call will passthrough WAKAWAKA to "c" and "d", "d" will
passthrough WAKAWAKA to "e". Etc.

We aggregate all RTR logs into ELK so it would be super helpful to
them to be able to filter on WAKAWAKA and get all the access logs(and app
logs aswell, they mostly use GELF so its easy for them to add whatewher
field they want) from the services involved.

I had a quick peek at the gorouter(
https://github.com/cloudfoundry/gorouter/blob/76668f5818ea8c089ff52a14fcdfbf703c8e8767/access_log/access_log_record.go#L40)
and it seems like this should be a simple PR.

1. To gorouter.yml add
passthrough_headers:
- WAKAWAKA
- X-Random-Header

2. In makeRecord at the bottom add something like(in psuedo)

data = {}
for header in passthrough_headers:
header_val = r.FormatRequestHeader("X-Forwarded-For")
if header_val:
passthrough_headers[header] = header_val

if data:
fmt.Fprintf(b, data.to_stringified_json())

That would yield a log line like
blurgh.dev.cf.private.domain.com - [21/07/2015:10:17:05 +0000]
"GET
/statements?ascending=true&since=2015-06-30T14%3A10%3A03.078Z&skipStatementId=30a88204-0779-4385-9859-e4aabd30baf0
HTTP/1.1" 200 0 17 "-" "NING/1.0" 10.230.31.2:46204
x_forwarded_for:"-" vcap_request_id:1e58195a-cde6-4afd-7f03-43061c9ea91c
response_time:0.004927106 app_id:9784cd03-050d-4b74-9e90-5f17134a3f08
{"WAKAWAKA": "Space is the place", "X-Random-Header": "Once upon a midnight
dreary, while I pondered weak and weary"}

The reason for a stringified JSON is to make it easy to parse with
logstash or other loganalysis tools.

Before I spend time implementing, is this something you would merge?

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

Some time ago routing services were discussed on a CAB [1]. Here is a description of this proposal.

Do you think that using such service will allow your developers to cover this requirements?

[1] http://www.activestate.com/blog/2015/02/cloud-foundry-advisory-board-meeting-2015-february <http://www.activestate.com/blog/2015/02/cloud-foundry-advisory-board-meeting-2015-february>
[2] https://docs.google.com/document/d/1bGOQxiKkmaw6uaRWGd-sXpxL0Y28d3QihcluI15FiIA/edit#heading=h.8djffzes9pnb <https://docs.google.com/document/d/1bGOQxiKkmaw6uaRWGd-sXpxL0Y28d3QihcluI15FiIA/edit#heading=h.8djffzes9pnb>

Simon,

Some buildpacks such as java-buildpack and php, offer the ability to log
incoming http request and control format of the logs inc custom headers.
Would that help in your case ?

https://github.com/cloudfoundry/java-buildpack/issues/57
https://github.com/cloudfoundry/java-buildpack/blob/master/docs/container-tomcat.md#configuration

Guillaume.
Le 22 juil. 2015 11:50, "Simon Johansson" <simon(a)simonjohansson.com> a
écrit :

This is not something that would be merged, as originally proposed, without
additional investigation and discussion.

If I understand correctly, you're proposing that an operator of
CloudFoundry configure GoRouter, which is a multi-tenant, shared service,
with knowledge specific to one or a few applications. This should not be an
operator responsibility, nor should the solution be specific to one or a
few applications.

The goal is "the flexibility of being able to annotate our logs with what
we consider to be important for our debugging purposes." More specifically
you're requesting logging of headers. Do you have a preference?

If GoRouter logged whatever headers were included in the request, wouldn't
this satisfy your requirements? Doesn't GoRouter do this already?

I'm interested in solving your requirement generically for all
applications, and focussing the user experience on the correct persona.
Based on what you've described, the persona is the app developer, so
control of what is logged should be in their hands. I'm also not convinced
GoRouter should have any knowledge of headers specific to one application
or another.





Shannon Coen
Product Manager, Cloud Foundry
Pivotal, Inc.

On Wed, Jul 22, 2015 at 3:05 AM, Alex Lomov <alexander.lomov(a)altoros.com>
wrote:

Some time ago routing services were discussed on a CAB [1]. Here is a
description of this proposal.

Do you think that using such service will allow your developers to cover
this requirements?

[1]
http://www.activestate.com/blog/2015/02/cloud-foundry-advisory-board-meeting-2015-february
[2]
https://docs.google.com/document/d/1bGOQxiKkmaw6uaRWGd-sXpxL0Y28d3QihcluI15FiIA/edit#heading=h.8djffzes9pnb

On Jul 21, 2015, at 4:06 PM, Simon Johansson <simon(a)simonjohansson.com>
wrote:

Howdie!

We have some devs who want to be able to trace a request troughout their
applications.

user -> a -> b -> c
|
|-> d -> e

When a user makes a request to "a" uuid is generated inside the app, and
the request to "b" from "a" will set a header(call it WAKAWAKA to uuid),
"b"'s call will passthrough WAKAWAKA to "c" and "d", "d" will passthrough
WAKAWAKA to "e". Etc.

We aggregate all RTR logs into ELK so it would be super helpful to them to
be able to filter on WAKAWAKA and get all the access logs(and app logs
aswell, they mostly use GELF so its easy for them to add whatewher field
they want) from the services involved.

I had a quick peek at the gorouter(
https://github.com/cloudfoundry/gorouter/blob/76668f5818ea8c089ff52a14fcdfbf703c8e8767/access_log/access_log_record.go#L40)
and it seems like this should be a simple PR.

1. To gorouter.yml add
passthrough_headers:
- WAKAWAKA
- X-Random-Header

2. In makeRecord at the bottom add something like(in psuedo)

data = {}
for header in passthrough_headers:
header_val = r.FormatRequestHeader("X-Forwarded-For")
if header_val:
passthrough_headers[header] = header_val

if data:
fmt.Fprintf(b, data.to_stringified_json())

That would yield a log line like
blurgh.dev.cf.private.domain.com - [21/07/2015:10:17:05 +0000] "GET
/statements?ascending=true&since=2015-06-30T14%3A10%3A03.078Z&skipStatementId=30a88204-0779-4385-9859-e4aabd30baf0
HTTP/1.1" 200 0 17 "-" "NING/1.0" 10.230.31.2:46204 x_forwarded_for:"-"
vcap_request_id:1e58195a-cde6-4afd-7f03-43061c9ea91c
response_time:0.004927106 app_id:9784cd03-050d-4b74-9e90-5f17134a3f08
{"WAKAWAKA": "Space is the place", "X-Random-Header": "Once upon a midnight
dreary, while I pondered weak and weary"}

The reason for a stringified JSON is to make it easy to parse with
logstash or other loganalysis tools.

Before I spend time implementing, is this something you would merge?
_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev



_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

Shannon, Simon,

As the lead of the logsearch.io (ELK) project; I'm also interested in having
GoRouter log additional headers. Specifically Trace-Id and Span-Id
generated by the spring-cloud-sleuth project
(https://github.com/spring-cloud-incubator/spring-cloud-sleuth ).

If GoRouter logged whatever headers were included in the request, wouldn't
this satisfy your requirements?

This would certainly satisfy my requirements, and I think Simon's too.

However, not having a whitelist of headers to log opens a possible DDOS
attack vector on the GoRouter, so I think having a operator configureable
whitelist (with some sensible defaults like Trace-Id and Span-Id) is the
right approach.

Doesn't GoRouter do this already?

I don't think so.

Specifically, sending the following curl to an app hosted on PWS:

curl --header "Trace-Id: 1c884728-466c-4ba3-8caa-a02a780c6d56"
http://www.birdsangola.org/

Gives the following [RTR] log from loggregator:

Fri Jul 24 2015 13:10:52 GMT+0100 (BST) [RTR] OUT www.birdsangola.org -
[24/07/2015:12:10:52 +0000] "GET / HTTP/1.1" 200 0 7772 "-" "curl/7.30.0"
10.10.2.185:46765 x_forwarded_for:"92.40.249.226"
vcap_request_id:3a33d5f6-dc11-42c4-61c7-32a1a2557200
response_time:0.001380276 app_id:0c34cc9f-45a8-440e-b876-b0cde564fbe3

It doesn't look like the extra Trace-Id header has been passed through to
the loggregator [RTR] log.

I'd be happy to work with Simon to contribute to a PR that implements the
"above whitelist of headers to log" feature.

Thoughts?

:D



--
View this message in context: http://cf-dev.70369.x6.nabble.com/cf-dev-Allow-gorouter-to-log-random-headers-tp800p877.html
Sent from the CF Dev mailing list archive at Nabble.com.

If I understand correctly, you're proposing that an operator of

CloudFoundry configure GoRouter, which is a multi-tenant, shared service,
with knowledge specific to one or a few applications

That is indeed the proposal. In my org we work closely with the different
development streams to provide a good out of the box experience by creating
certain opt-in features that makes their life easier. This would be one of
those features, if you pass certain headers in your requests thats
standardised across the entire online organisation you will be able to
query for them in Kibana.
I understand that this is not the case in all environments.

If GoRouter logged whatever headers were included in the request,

wouldn't this satisfy your requirements?

This would indeed satisfy my requests, but as David points out("However,
not having a whitelist of headers to log opens a possible DDOS attack
vector on the GoRouter") it might now be appropriate.

Based on what you've described, the persona is the app developer.

Not necessarily, we operators are also very interested in certain headers.

so control of what is logged should be in their hands.

If one via the API could set headers that should be logged for an org,
space or application that would be magical. The need for a list of
"must-log-all-these-headers" would still be there I think as not to have to
maintain a list of these standardised headers across different objects.


On Thu, Jul 23, 2015 at 10:51 PM, Shannon Coen <scoen(a)pivotal.io> wrote:

This is not something that would be merged, as originally proposed,
without additional investigation and discussion.

If I understand correctly, you're proposing that an operator of
CloudFoundry configure GoRouter, which is a multi-tenant, shared service,
with knowledge specific to one or a few applications. This should not be an
operator responsibility, nor should the solution be specific to one or a
few applications.

The goal is "the flexibility of being able to annotate our logs with what
we consider to be important for our debugging purposes." More specifically
you're requesting logging of headers. Do you have a preference?

If GoRouter logged whatever headers were included in the request, wouldn't
this satisfy your requirements? Doesn't GoRouter do this already?

I'm interested in solving your requirement generically for all
applications, and focussing the user experience on the correct persona.
Based on what you've described, the persona is the app developer, so
control of what is logged should be in their hands. I'm also not convinced
GoRouter should have any knowledge of headers specific to one application
or another.





Shannon Coen
Product Manager, Cloud Foundry
Pivotal, Inc.

On Wed, Jul 22, 2015 at 3:05 AM, Alex Lomov <alexander.lomov(a)altoros.com>
wrote:

Some time ago routing services were discussed on a CAB [1]. Here is a
description of this proposal.

Do you think that using such service will allow your developers to cover
this requirements?

[1]
http://www.activestate.com/blog/2015/02/cloud-foundry-advisory-board-meeting-2015-february
[2]
https://docs.google.com/document/d/1bGOQxiKkmaw6uaRWGd-sXpxL0Y28d3QihcluI15FiIA/edit#heading=h.8djffzes9pnb

On Jul 21, 2015, at 4:06 PM, Simon Johansson <simon(a)simonjohansson.com>
wrote:

Howdie!

We have some devs who want to be able to trace a request troughout their
applications.

user -> a -> b -> c
|
|-> d -> e

When a user makes a request to "a" uuid is generated inside the app, and
the request to "b" from "a" will set a header(call it WAKAWAKA to uuid),
"b"'s call will passthrough WAKAWAKA to "c" and "d", "d" will passthrough
WAKAWAKA to "e". Etc.

We aggregate all RTR logs into ELK so it would be super helpful to them
to be able to filter on WAKAWAKA and get all the access logs(and app logs
aswell, they mostly use GELF so its easy for them to add whatewher field
they want) from the services involved.

I had a quick peek at the gorouter(
https://github.com/cloudfoundry/gorouter/blob/76668f5818ea8c089ff52a14fcdfbf703c8e8767/access_log/access_log_record.go#L40)
and it seems like this should be a simple PR.

1. To gorouter.yml add
passthrough_headers:
- WAKAWAKA
- X-Random-Header

2. In makeRecord at the bottom add something like(in psuedo)

data = {}
for header in passthrough_headers:
header_val = r.FormatRequestHeader("X-Forwarded-For")
if header_val:
passthrough_headers[header] = header_val

if data:
fmt.Fprintf(b, data.to_stringified_json())

That would yield a log line like
blurgh.dev.cf.private.domain.com - [21/07/2015:10:17:05 +0000] "GET
/statements?ascending=true&since=2015-06-30T14%3A10%3A03.078Z&skipStatementId=30a88204-0779-4385-9859-e4aabd30baf0
HTTP/1.1" 200 0 17 "-" "NING/1.0" 10.230.31.2:46204 x_forwarded_for:"-"
vcap_request_id:1e58195a-cde6-4afd-7f03-43061c9ea91c
response_time:0.004927106 app_id:9784cd03-050d-4b74-9e90-5f17134a3f08
{"WAKAWAKA": "Space is the place", "X-Random-Header": "Once upon a midnight
dreary, while I pondered weak and weary"}

The reason for a stringified JSON is to make it easy to parse with
logstash or other loganalysis tools.

Before I spend time implementing, is this something you would merge?
_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev



_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

shannon,

from what i'm reading here about the use case, the main interest is that a
CF operator knows that their cf installation is more deeply integrated with
a specific log parsing solution for all/many apps on that platform that
choose to use it (whether than is ELK, Zipkin, etc). it does not sound like
it is a special case with lots of variation by many different app teams on
the cf installation. rather, it sounds like this is most likely to be used
as an installation-wide option to enhance the app developer / app
operations experience.

it seems like an operator configured whitelist set of headers that get
logged with the RTR message satisfies the current needs well and is
reasonable.

if we were to find that in the future lots of variation and different app
teams on the same CF wanted to have the RTR tier log many different headers
in the access log, then we could enhance the "log the whitelist of headers
to the access log" capability to be exposed to a limited number of headers
that each route could be configured by developers to specify that would
apply in addition to the operator configured logged headers. but it sounds
like that isn't needed right now.

simon/david, did i summarize this correctly?

On Fri, Jul 24, 2015 at 7:23 AM, Simon Johansson <simon(a)simonjohansson.com>
wrote:

If I understand correctly, you're proposing that an operator of

CloudFoundry configure GoRouter, which is a multi-tenant, shared service,
with knowledge specific to one or a few applications

That is indeed the proposal. In my org we work closely with the different
development streams to provide a good out of the box experience by creating
certain opt-in features that makes their life easier. This would be one of
those features, if you pass certain headers in your requests thats
standardised across the entire online organisation you will be able to
query for them in Kibana.
I understand that this is not the case in all environments.

If GoRouter logged whatever headers were included in the request,

wouldn't this satisfy your requirements?

This would indeed satisfy my requests, but as David points out("However,
not having a whitelist of headers to log opens a possible DDOS attack
vector on the GoRouter") it might now be appropriate.

Based on what you've described, the persona is the app developer.

Not necessarily, we operators are also very interested in certain headers.

so control of what is logged should be in their hands.

If one via the API could set headers that should be logged for an org,
space or application that would be magical. The need for a list of
"must-log-all-these-headers" would still be there I think as not to have to
maintain a list of these standardised headers across different objects.


On Thu, Jul 23, 2015 at 10:51 PM, Shannon Coen <scoen(a)pivotal.io> wrote:

This is not something that would be merged, as originally proposed,
without additional investigation and discussion.

If I understand correctly, you're proposing that an operator of
CloudFoundry configure GoRouter, which is a multi-tenant, shared service,
with knowledge specific to one or a few applications. This should not be an
operator responsibility, nor should the solution be specific to one or a
few applications.

The goal is "the flexibility of being able to annotate our logs with
what we consider to be important for our debugging purposes." More
specifically you're requesting logging of headers. Do you have a preference?

If GoRouter logged whatever headers were included in the request,
wouldn't this satisfy your requirements? Doesn't GoRouter do this already?

I'm interested in solving your requirement generically for all
applications, and focussing the user experience on the correct persona.
Based on what you've described, the persona is the app developer, so
control of what is logged should be in their hands. I'm also not convinced
GoRouter should have any knowledge of headers specific to one application
or another.





Shannon Coen
Product Manager, Cloud Foundry
Pivotal, Inc.

On Wed, Jul 22, 2015 at 3:05 AM, Alex Lomov <alexander.lomov(a)altoros.com>
wrote:

Some time ago routing services were discussed on a CAB [1]. Here is a
description of this proposal.

Do you think that using such service will allow your developers to cover
this requirements?

[1]
http://www.activestate.com/blog/2015/02/cloud-foundry-advisory-board-meeting-2015-february
[2]
https://docs.google.com/document/d/1bGOQxiKkmaw6uaRWGd-sXpxL0Y28d3QihcluI15FiIA/edit#heading=h.8djffzes9pnb

On Jul 21, 2015, at 4:06 PM, Simon Johansson <simon(a)simonjohansson.com>
wrote:

Howdie!

We have some devs who want to be able to trace a request troughout their
applications.

user -> a -> b -> c
|
|-> d -> e

When a user makes a request to "a" uuid is generated inside the app, and
the request to "b" from "a" will set a header(call it WAKAWAKA to uuid),
"b"'s call will passthrough WAKAWAKA to "c" and "d", "d" will passthrough
WAKAWAKA to "e". Etc.

We aggregate all RTR logs into ELK so it would be super helpful to them
to be able to filter on WAKAWAKA and get all the access logs(and app logs
aswell, they mostly use GELF so its easy for them to add whatewher field
they want) from the services involved.

I had a quick peek at the gorouter(
https://github.com/cloudfoundry/gorouter/blob/76668f5818ea8c089ff52a14fcdfbf703c8e8767/access_log/access_log_record.go#L40)
and it seems like this should be a simple PR.

1. To gorouter.yml add
passthrough_headers:
- WAKAWAKA
- X-Random-Header

2. In makeRecord at the bottom add something like(in psuedo)

data = {}
for header in passthrough_headers:
header_val = r.FormatRequestHeader("X-Forwarded-For")
if header_val:
passthrough_headers[header] = header_val

if data:
fmt.Fprintf(b, data.to_stringified_json())

That would yield a log line like
blurgh.dev.cf.private.domain.com - [21/07/2015:10:17:05 +0000] "GET
/statements?ascending=true&since=2015-06-30T14%3A10%3A03.078Z&skipStatementId=30a88204-0779-4385-9859-e4aabd30baf0
HTTP/1.1" 200 0 17 "-" "NING/1.0" 10.230.31.2:46204 x_forwarded_for:"-"
vcap_request_id:1e58195a-cde6-4afd-7f03-43061c9ea91c
response_time:0.004927106 app_id:9784cd03-050d-4b74-9e90-5f17134a3f08
{"WAKAWAKA": "Space is the place", "X-Random-Header": "Once upon a midnight
dreary, while I pondered weak and weary"}

The reason for a stringified JSON is to make it easy to parse with
logstash or other loganalysis tools.

Before I spend time implementing, is this something you would merge?
_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev



_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

--
Thank you,

James Bayer

simon/david, did i summarize this correctly?

Spot on summarisation. :)

On Fri, Jul 24, 2015 at 5:26 PM, James Bayer <jbayer(a)pivotal.io> wrote:

shannon,

from what i'm reading here about the use case, the main interest is that a
CF operator knows that their cf installation is more deeply integrated with
a specific log parsing solution for all/many apps on that platform that
choose to use it (whether than is ELK, Zipkin, etc). it does not sound like
it is a special case with lots of variation by many different app teams on
the cf installation. rather, it sounds like this is most likely to be used
as an installation-wide option to enhance the app developer / app
operations experience.

it seems like an operator configured whitelist set of headers that get
logged with the RTR message satisfies the current needs well and is
reasonable.

if we were to find that in the future lots of variation and different app
teams on the same CF wanted to have the RTR tier log many different headers
in the access log, then we could enhance the "log the whitelist of headers
to the access log" capability to be exposed to a limited number of headers
that each route could be configured by developers to specify that would
apply in addition to the operator configured logged headers. but it sounds
like that isn't needed right now.

simon/david, did i summarize this correctly?

On Fri, Jul 24, 2015 at 7:23 AM, Simon Johansson <simon(a)simonjohansson.com

wrote:

If I understand correctly, you're proposing that an operator of

CloudFoundry configure GoRouter, which is a multi-tenant, shared service,
with knowledge specific to one or a few applications

That is indeed the proposal. In my org we work closely with the different
development streams to provide a good out of the box experience by creating
certain opt-in features that makes their life easier. This would be one of
those features, if you pass certain headers in your requests thats
standardised across the entire online organisation you will be able to
query for them in Kibana.
I understand that this is not the case in all environments.

If GoRouter logged whatever headers were included in the request,

wouldn't this satisfy your requirements?

This would indeed satisfy my requests, but as David points out("However,
not having a whitelist of headers to log opens a possible DDOS attack
vector on the GoRouter") it might now be appropriate.

Based on what you've described, the persona is the app developer.

Not necessarily, we operators are also very interested in certain headers.

so control of what is logged should be in their hands.

If one via the API could set headers that should be logged for an org,
space or application that would be magical. The need for a list of
"must-log-all-these-headers" would still be there I think as not to have to
maintain a list of these standardised headers across different objects.


On Thu, Jul 23, 2015 at 10:51 PM, Shannon Coen <scoen(a)pivotal.io> wrote:

This is not something that would be merged, as originally proposed,
without additional investigation and discussion.

If I understand correctly, you're proposing that an operator of
CloudFoundry configure GoRouter, which is a multi-tenant, shared service,
with knowledge specific to one or a few applications. This should not be an
operator responsibility, nor should the solution be specific to one or a
few applications.

The goal is "the flexibility of being able to annotate our logs with
what we consider to be important for our debugging purposes." More
specifically you're requesting logging of headers. Do you have a preference?

If GoRouter logged whatever headers were included in the request,
wouldn't this satisfy your requirements? Doesn't GoRouter do this already?

I'm interested in solving your requirement generically for all
applications, and focussing the user experience on the correct persona.
Based on what you've described, the persona is the app developer, so
control of what is logged should be in their hands. I'm also not convinced
GoRouter should have any knowledge of headers specific to one application
or another.





Shannon Coen
Product Manager, Cloud Foundry
Pivotal, Inc.

On Wed, Jul 22, 2015 at 3:05 AM, Alex Lomov <alexander.lomov(a)altoros.com

wrote:

Some time ago routing services were discussed on a CAB [1]. Here is a
description of this proposal.

Do you think that using such service will allow your developers to
cover this requirements?

[1]
http://www.activestate.com/blog/2015/02/cloud-foundry-advisory-board-meeting-2015-february
[2]
https://docs.google.com/document/d/1bGOQxiKkmaw6uaRWGd-sXpxL0Y28d3QihcluI15FiIA/edit#heading=h.8djffzes9pnb

On Jul 21, 2015, at 4:06 PM, Simon Johansson <simon(a)simonjohansson.com>
wrote:

Howdie!

We have some devs who want to be able to trace a request troughout
their applications.

user -> a -> b -> c
|
|-> d -> e

When a user makes a request to "a" uuid is generated inside the app,
and the request to "b" from "a" will set a header(call it WAKAWAKA to
uuid), "b"'s call will passthrough WAKAWAKA to "c" and "d", "d" will
passthrough WAKAWAKA to "e". Etc.

We aggregate all RTR logs into ELK so it would be super helpful to them
to be able to filter on WAKAWAKA and get all the access logs(and app logs
aswell, they mostly use GELF so its easy for them to add whatewher field
they want) from the services involved.

I had a quick peek at the gorouter(
https://github.com/cloudfoundry/gorouter/blob/76668f5818ea8c089ff52a14fcdfbf703c8e8767/access_log/access_log_record.go#L40)
and it seems like this should be a simple PR.

1. To gorouter.yml add
passthrough_headers:
- WAKAWAKA
- X-Random-Header

2. In makeRecord at the bottom add something like(in psuedo)

data = {}
for header in passthrough_headers:
header_val = r.FormatRequestHeader("X-Forwarded-For")
if header_val:
passthrough_headers[header] = header_val

if data:
fmt.Fprintf(b, data.to_stringified_json())

That would yield a log line like
blurgh.dev.cf.private.domain.com - [21/07/2015:10:17:05 +0000] "GET
/statements?ascending=true&since=2015-06-30T14%3A10%3A03.078Z&skipStatementId=30a88204-0779-4385-9859-e4aabd30baf0
HTTP/1.1" 200 0 17 "-" "NING/1.0" 10.230.31.2:46204
x_forwarded_for:"-" vcap_request_id:1e58195a-cde6-4afd-7f03-43061c9ea91c
response_time:0.004927106 app_id:9784cd03-050d-4b74-9e90-5f17134a3f08
{"WAKAWAKA": "Space is the place", "X-Random-Header": "Once upon a midnight
dreary, while I pondered weak and weary"}

The reason for a stringified JSON is to make it easy to parse with
logstash or other loganalysis tools.

Before I spend time implementing, is this something you would merge?
_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev



_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

--
Thank you,

James Bayer

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

James,

Yep - we're pretty much on the same page.

Only addition I would ask for is that the whitelist contain some "sensible"
defaults (eg, Trace-Id, Span-Id) that are switched on by default; since
then tight integration with tools like spring-cloud / buildpacks would work
out the box.

D

--
David Laing
logsearch.io - build your own open source cloud logging cluster
http://davidlaing.com

Thank you, James. That helps.

What bothered me about this thread was that the use case was presented as,
"as an unspecified persona, I want to customize the multi-tenant router to
log specified headers for a few apps". This is troubling.

Instead it sounds like the use case is "as an operator of Cloud Foundry, I
want to enable logging of specified headers for all applications." This is
more compelling.

I'm skeptical that this manifest property should have a default. Who's to
say what the minimum list of "sensible" defaults is? Does WAKAWAKA make the
cut? j/k.

We'll take a PR.

Shannon Coen
Product Manager, Cloud Foundry
Pivotal, Inc.

Shannon,

Understood. PR for "as an operator of Cloud Foundry, I want to enable
logging of specified headers for all applications." coming up.

Experience will tell whether there are any sensible defaults; but we can
leave that for a future enhancement of this functionality.

Regards

David

--
David Laing
logsearch.io - build your own open source cloud logging cluster
http://davidlaing.com

Here is a PR David and I worked on this morning to implement the above

https://github.com/cloudfoundry/gorouter/pull/92

Comments are appreciated.