shannon,

from what i'm reading here about the use case, the main interest is that a
CF operator knows that their cf installation is more deeply integrated with
a specific log parsing solution for all/many apps on that platform that
choose to use it (whether than is ELK, Zipkin, etc). it does not sound like
it is a special case with lots of variation by many different app teams on
the cf installation. rather, it sounds like this is most likely to be used
as an installation-wide option to enhance the app developer / app
operations experience.

it seems like an operator configured whitelist set of headers that get
logged with the RTR message satisfies the current needs well and is
reasonable.

if we were to find that in the future lots of variation and different app
teams on the same CF wanted to have the RTR tier log many different headers
in the access log, then we could enhance the "log the whitelist of headers
to the access log" capability to be exposed to a limited number of headers
that each route could be configured by developers to specify that would
apply in addition to the operator configured logged headers. but it sounds
like that isn't needed right now.

simon/david, did i summarize this correctly?

On Fri, Jul 24, 2015 at 7:23 AM, Simon Johansson <simon(a)simonjohansson.com

wrote:

If I understand correctly, you're proposing that an operator of

CloudFoundry configure GoRouter, which is a multi-tenant, shared service,
with knowledge specific to one or a few applications

That is indeed the proposal. In my org we work closely with the different
development streams to provide a good out of the box experience by creating
certain opt-in features that makes their life easier. This would be one of
those features, if you pass certain headers in your requests thats
standardised across the entire online organisation you will be able to
query for them in Kibana.
I understand that this is not the case in all environments.

If GoRouter logged whatever headers were included in the request,

wouldn't this satisfy your requirements?

This would indeed satisfy my requests, but as David points out("However,
not having a whitelist of headers to log opens a possible DDOS attack
vector on the GoRouter") it might now be appropriate.

Based on what you've described, the persona is the app developer.

Not necessarily, we operators are also very interested in certain headers.

so control of what is logged should be in their hands.

If one via the API could set headers that should be logged for an org,
space or application that would be magical. The need for a list of
"must-log-all-these-headers" would still be there I think as not to have to
maintain a list of these standardised headers across different objects.


On Thu, Jul 23, 2015 at 10:51 PM, Shannon Coen <scoen(a)pivotal.io> wrote:

This is not something that would be merged, as originally proposed,
without additional investigation and discussion.

If I understand correctly, you're proposing that an operator of
CloudFoundry configure GoRouter, which is a multi-tenant, shared service,
with knowledge specific to one or a few applications. This should not be an
operator responsibility, nor should the solution be specific to one or a
few applications.

The goal is "the flexibility of being able to annotate our logs with
what we consider to be important for our debugging purposes." More
specifically you're requesting logging of headers. Do you have a preference?

If GoRouter logged whatever headers were included in the request,
wouldn't this satisfy your requirements? Doesn't GoRouter do this already?

I'm interested in solving your requirement generically for all
applications, and focussing the user experience on the correct persona.
Based on what you've described, the persona is the app developer, so
control of what is logged should be in their hands. I'm also not convinced
GoRouter should have any knowledge of headers specific to one application
or another.





Shannon Coen
Product Manager, Cloud Foundry
Pivotal, Inc.

On Wed, Jul 22, 2015 at 3:05 AM, Alex Lomov <alexander.lomov(a)altoros.com

wrote:

Some time ago routing services were discussed on a CAB [1]. Here is a
description of this proposal.

Do you think that using such service will allow your developers to
cover this requirements?

[1]
http://www.activestate.com/blog/2015/02/cloud-foundry-advisory-board-meeting-2015-february
[2]
https://docs.google.com/document/d/1bGOQxiKkmaw6uaRWGd-sXpxL0Y28d3QihcluI15FiIA/edit#heading=h.8djffzes9pnb

On Jul 21, 2015, at 4:06 PM, Simon Johansson <simon(a)simonjohansson.com>
wrote:

Howdie!

We have some devs who want to be able to trace a request troughout
their applications.

user -> a -> b -> c
|
|-> d -> e

When a user makes a request to "a" uuid is generated inside the app,
and the request to "b" from "a" will set a header(call it WAKAWAKA to
uuid), "b"'s call will passthrough WAKAWAKA to "c" and "d", "d" will
passthrough WAKAWAKA to "e". Etc.

We aggregate all RTR logs into ELK so it would be super helpful to them
to be able to filter on WAKAWAKA and get all the access logs(and app logs
aswell, they mostly use GELF so its easy for them to add whatewher field
they want) from the services involved.

I had a quick peek at the gorouter(
https://github.com/cloudfoundry/gorouter/blob/76668f5818ea8c089ff52a14fcdfbf703c8e8767/access_log/access_log_record.go#L40)
and it seems like this should be a simple PR.

1. To gorouter.yml add
passthrough_headers:
- WAKAWAKA
- X-Random-Header

2. In makeRecord at the bottom add something like(in psuedo)

data = {}
for header in passthrough_headers:
header_val = r.FormatRequestHeader("X-Forwarded-For")
if header_val:
passthrough_headers[header] = header_val

if data:
fmt.Fprintf(b, data.to_stringified_json())

That would yield a log line like
blurgh.dev.cf.private.domain.com - [21/07/2015:10:17:05 +0000] "GET
/statements?ascending=true&since=2015-06-30T14%3A10%3A03.078Z&skipStatementId=30a88204-0779-4385-9859-e4aabd30baf0
HTTP/1.1" 200 0 17 "-" "NING/1.0" 10.230.31.2:46204
x_forwarded_for:"-" vcap_request_id:1e58195a-cde6-4afd-7f03-43061c9ea91c
response_time:0.004927106 app_id:9784cd03-050d-4b74-9e90-5f17134a3f08
{"WAKAWAKA": "Space is the place", "X-Random-Header": "Once upon a midnight
dreary, while I pondered weak and weary"}

The reason for a stringified JSON is to make it easy to parse with
logstash or other loganalysis tools.

Before I spend time implementing, is this something you would merge?
_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev



_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

--
Thank you,

James Bayer

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev