Re: cflinuxfs2 stack not fount under v210.2 release
What cf stacks command give you ?
toggle quoted message
Show quoted text
On Tue, Jul 28, 2015 at 11:55 AM, Yitao Jiang <jiangyt.cn(a)gmail.com> wrote: We built our own release using community source code, and he just told me that using cf-210002. I know that the stack are install during the deployment and i just verify that all deas have installed cflinusfs2 stack.
On Tue, Jul 28, 2015 at 10:07 AM, Gwenn Etourneau <getourneau(a)pivotal.io> wrote:
I mean which version are the DEA ? the stack do not live in the stemcell ... The stack are install during the deployment
On Tue, Jul 28, 2015 at 11:04 AM, Yitao Jiang <jiangyt.cn(a)gmail.com> wrote:
Yes,all deas are built from same stemcell, all with fresh installation
+-------------------------------------------+---------+--------------------------------------+ | Name | Version | CID |
+-------------------------------------------+---------+--------------------------------------+ | bosh-openstack-kvm-ubuntu-trusty-go_agent | 2978* | bc3592a8-64fd-44b1-a9c0-797953f86e13 |
+-------------------------------------------+---------+--------------------------------------+
(*) Currently in-use
On Tue, Jul 28, 2015 at 9:42 AM, Gwenn Etourneau <getourneau(a)pivotal.io> wrote:
Are you sure all you DEA are at the same version ? I think so but I don't know maybe rare case.
On Tue, Jul 28, 2015 at 8:35 AM, Yitao Jiang <jiangyt.cn(a)gmail.com> wrote:
Hi,
Does any body ever have the problem of deploying application failed in *cflinuxfs2 stack not found?* we deployed v210.2 release and occasionally it failed in* cflinuxfs2 not found* when deploying. It's weird that the nats with 2 nodes all works fine, i can manually received the msgs distributed among the cluster.
Regards,
Yitao jiangyt.github.io
_______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
_______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
--
Regards,
Yitao jiangyt.github.io
_______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
_______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
--
Regards,
Yitao jiangyt.github.io
_______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
|
|
Re: cflinuxfs2 stack not fount under v210.2 release
We built our own release using community source code, and he just told me that using cf-210002. I know that the stack are install during the deployment and i just verify that all deas have installed cflinusfs2 stack. On Tue, Jul 28, 2015 at 10:07 AM, Gwenn Etourneau <getourneau(a)pivotal.io> wrote: I mean which version are the DEA ? the stack do not live in the stemcell ... The stack are install during the deployment
On Tue, Jul 28, 2015 at 11:04 AM, Yitao Jiang <jiangyt.cn(a)gmail.com> wrote:
Yes,all deas are built from same stemcell, all with fresh installation
+-------------------------------------------+---------+--------------------------------------+ | Name | Version | CID |
+-------------------------------------------+---------+--------------------------------------+ | bosh-openstack-kvm-ubuntu-trusty-go_agent | 2978* | bc3592a8-64fd-44b1-a9c0-797953f86e13 |
+-------------------------------------------+---------+--------------------------------------+
(*) Currently in-use
On Tue, Jul 28, 2015 at 9:42 AM, Gwenn Etourneau <getourneau(a)pivotal.io> wrote:
Are you sure all you DEA are at the same version ? I think so but I don't know maybe rare case.
On Tue, Jul 28, 2015 at 8:35 AM, Yitao Jiang <jiangyt.cn(a)gmail.com> wrote:
Hi,
Does any body ever have the problem of deploying application failed in *cflinuxfs2 stack not found?* we deployed v210.2 release and occasionally it failed in* cflinuxfs2 not found* when deploying. It's weird that the nats with 2 nodes all works fine, i can manually received the msgs distributed among the cluster.
Regards,
Yitao jiangyt.github.io
_______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
_______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
--
Regards,
Yitao jiangyt.github.io
_______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
_______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
-- Regards, Yitao jiangyt.github.io
|
|
Re: Allow gorouter to log random headers.
Thank you, James. That helps.
What bothered me about this thread was that the use case was presented as, "as an unspecified persona, I want to customize the multi-tenant router to log specified headers for a few apps". This is troubling.
Instead it sounds like the use case is "as an operator of Cloud Foundry, I want to enable logging of specified headers for all applications." This is more compelling.
I'm skeptical that this manifest property should have a default. Who's to say what the minimum list of "sensible" defaults is? Does WAKAWAKA make the cut? j/k.
We'll take a PR.
Shannon Coen Product Manager, Cloud Foundry Pivotal, Inc.
toggle quoted message
Show quoted text
On Fri, Jul 24, 2015 at 8:46 AM, David Laing <david(a)davidlaing.com> wrote: James,
Yep - we're pretty much on the same page.
Only addition I would ask for is that the whitelist contain some "sensible" defaults (eg, Trace-Id, Span-Id) that are switched on by default; since then tight integration with tools like spring-cloud / buildpacks would work out the box.
D
On 24 July 2015 at 16:26, James Bayer <jbayer(a)pivotal.io> wrote:
shannon,
from what i'm reading here about the use case, the main interest is that a CF operator knows that their cf installation is more deeply integrated with a specific log parsing solution for all/many apps on that platform that choose to use it (whether than is ELK, Zipkin, etc). it does not sound like it is a special case with lots of variation by many different app teams on the cf installation. rather, it sounds like this is most likely to be used as an installation-wide option to enhance the app developer / app operations experience.
it seems like an operator configured whitelist set of headers that get logged with the RTR message satisfies the current needs well and is reasonable.
if we were to find that in the future lots of variation and different app teams on the same CF wanted to have the RTR tier log many different headers in the access log, then we could enhance the "log the whitelist of headers to the access log" capability to be exposed to a limited number of headers that each route could be configured by developers to specify that would apply in addition to the operator configured logged headers. but it sounds like that isn't needed right now.
simon/david, did i summarize this correctly?
On Fri, Jul 24, 2015 at 7:23 AM, Simon Johansson < simon(a)simonjohansson.com> wrote:
If I understand correctly, you're proposing that an operator of CloudFoundry configure GoRouter, which is a multi-tenant, shared service, with knowledge specific to one or a few applications
That is indeed the proposal. In my org we work closely with the different development streams to provide a good out of the box experience by creating certain opt-in features that makes their life easier. This would be one of those features, if you pass certain headers in your requests thats standardised across the entire online organisation you will be able to query for them in Kibana. I understand that this is not the case in all environments.
If GoRouter logged whatever headers were included in the request, wouldn't this satisfy your requirements?
This would indeed satisfy my requests, but as David points out("However, not having a whitelist of headers to log opens a possible DDOS attack vector on the GoRouter") it might now be appropriate.
Based on what you've described, the persona is the app developer. Not necessarily, we operators are also very interested in certain headers.
so control of what is logged should be in their hands. If one via the API could set headers that should be logged for an org, space or application that would be magical. The need for a list of "must-log-all-these-headers" would still be there I think as not to have to maintain a list of these standardised headers across different objects.
On Thu, Jul 23, 2015 at 10:51 PM, Shannon Coen <scoen(a)pivotal.io> wrote:
This is not something that would be merged, as originally proposed, without additional investigation and discussion.
If I understand correctly, you're proposing that an operator of CloudFoundry configure GoRouter, which is a multi-tenant, shared service, with knowledge specific to one or a few applications. This should not be an operator responsibility, nor should the solution be specific to one or a few applications.
The goal is "the flexibility of being able to annotate our logs with what we consider to be important for our debugging purposes." More specifically you're requesting logging of headers. Do you have a preference?
If GoRouter logged whatever headers were included in the request, wouldn't this satisfy your requirements? Doesn't GoRouter do this already?
I'm interested in solving your requirement generically for all applications, and focussing the user experience on the correct persona. Based on what you've described, the persona is the app developer, so control of what is logged should be in their hands. I'm also not convinced GoRouter should have any knowledge of headers specific to one application or another.
Shannon Coen Product Manager, Cloud Foundry Pivotal, Inc.
On Wed, Jul 22, 2015 at 3:05 AM, Alex Lomov < alexander.lomov(a)altoros.com> wrote:
Some time ago routing services were discussed on a CAB [1]. Here is a description of this proposal.
Do you think that using such service will allow your developers to cover this requirements?
[1] http://www.activestate.com/blog/2015/02/cloud-foundry-advisory-board-meeting-2015-february [2] https://docs.google.com/document/d/1bGOQxiKkmaw6uaRWGd-sXpxL0Y28d3QihcluI15FiIA/edit#heading=h.8djffzes9pnb
On Jul 21, 2015, at 4:06 PM, Simon Johansson <simon(a)simonjohansson.com> wrote:
Howdie!
We have some devs who want to be able to trace a request troughout their applications.
user -> a -> b -> c | |-> d -> e
When a user makes a request to "a" uuid is generated inside the app, and the request to "b" from "a" will set a header(call it WAKAWAKA to uuid), "b"'s call will passthrough WAKAWAKA to "c" and "d", "d" will passthrough WAKAWAKA to "e". Etc.
We aggregate all RTR logs into ELK so it would be super helpful to them to be able to filter on WAKAWAKA and get all the access logs(and app logs aswell, they mostly use GELF so its easy for them to add whatewher field they want) from the services involved.
I had a quick peek at the gorouter( https://github.com/cloudfoundry/gorouter/blob/76668f5818ea8c089ff52a14fcdfbf703c8e8767/access_log/access_log_record.go#L40) and it seems like this should be a simple PR.
1. To gorouter.yml add passthrough_headers: - WAKAWAKA - X-Random-Header
2. In makeRecord at the bottom add something like(in psuedo)
data = {} for header in passthrough_headers: header_val = r.FormatRequestHeader("X-Forwarded-For") if header_val: passthrough_headers[header] = header_val
if data: fmt.Fprintf(b, data.to_stringified_json())
That would yield a log line like blurgh.dev.cf.private.domain.com - [21/07/2015:10:17:05 +0000] "GET /statements?ascending=true&since=2015-06-30T14%3A10%3A03.078Z&skipStatementId=30a88204-0779-4385-9859-e4aabd30baf0 HTTP/1.1" 200 0 17 "-" "NING/1.0" 10.230.31.2:46204 x_forwarded_for:"-" vcap_request_id:1e58195a-cde6-4afd-7f03-43061c9ea91c response_time:0.004927106 app_id:9784cd03-050d-4b74-9e90-5f17134a3f08 {"WAKAWAKA": "Space is the place", "X-Random-Header": "Once upon a midnight dreary, while I pondered weak and weary"}
The reason for a stringified JSON is to make it easy to parse with logstash or other loganalysis tools.
Before I spend time implementing, is this something you would merge? _______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
_______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
_______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
_______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
-- Thank you,
James Bayer
_______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
-- David Laing logsearch.io - build your own open source cloud logging cluster http://davidlaing.com
_______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
|
|
Re: cflinuxfs2 stack not fount under v210.2 release
I mean which version are the DEA ? the stack do not live in the stemcell ... The stack are install during the deployment
toggle quoted message
Show quoted text
On Tue, Jul 28, 2015 at 11:04 AM, Yitao Jiang <jiangyt.cn(a)gmail.com> wrote: Yes,all deas are built from same stemcell, all with fresh installation
+-------------------------------------------+---------+--------------------------------------+ | Name | Version | CID |
+-------------------------------------------+---------+--------------------------------------+ | bosh-openstack-kvm-ubuntu-trusty-go_agent | 2978* | bc3592a8-64fd-44b1-a9c0-797953f86e13 |
+-------------------------------------------+---------+--------------------------------------+
(*) Currently in-use
On Tue, Jul 28, 2015 at 9:42 AM, Gwenn Etourneau <getourneau(a)pivotal.io> wrote:
Are you sure all you DEA are at the same version ? I think so but I don't know maybe rare case.
On Tue, Jul 28, 2015 at 8:35 AM, Yitao Jiang <jiangyt.cn(a)gmail.com> wrote:
Hi,
Does any body ever have the problem of deploying application failed in *cflinuxfs2 stack not found?* we deployed v210.2 release and occasionally it failed in* cflinuxfs2 not found* when deploying. It's weird that the nats with 2 nodes all works fine, i can manually received the msgs distributed among the cluster.
Regards,
Yitao jiangyt.github.io
_______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
_______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
--
Regards,
Yitao jiangyt.github.io
_______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
|
|
Re: cflinuxfs2 stack not fount under v210.2 release
Yes,all deas are built from same stemcell, all with fresh installation +-------------------------------------------+---------+--------------------------------------+ | Name | Version | CID | +-------------------------------------------+---------+--------------------------------------+ | bosh-openstack-kvm-ubuntu-trusty-go_agent | 2978* | bc3592a8-64fd-44b1-a9c0-797953f86e13 | +-------------------------------------------+---------+--------------------------------------+ (*) Currently in-use On Tue, Jul 28, 2015 at 9:42 AM, Gwenn Etourneau <getourneau(a)pivotal.io> wrote: Are you sure all you DEA are at the same version ? I think so but I don't know maybe rare case.
On Tue, Jul 28, 2015 at 8:35 AM, Yitao Jiang <jiangyt.cn(a)gmail.com> wrote:
Hi,
Does any body ever have the problem of deploying application failed in *cflinuxfs2 stack not found?* we deployed v210.2 release and occasionally it failed in* cflinuxfs2 not found* when deploying. It's weird that the nats with 2 nodes all works fine, i can manually received the msgs distributed among the cluster.
Regards,
Yitao jiangyt.github.io
_______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
_______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
-- Regards, Yitao jiangyt.github.io
|
|
Re: cflinuxfs2 stack not fount under v210.2 release
Are you sure all you DEA are at the same version ? I think so but I don't know maybe rare case.
toggle quoted message
Show quoted text
On Tue, Jul 28, 2015 at 8:35 AM, Yitao Jiang <jiangyt.cn(a)gmail.com> wrote: Hi,
Does any body ever have the problem of deploying application failed in *cflinuxfs2 stack not found?* we deployed v210.2 release and occasionally it failed in* cflinuxfs2 not found* when deploying. It's weird that the nats with 2 nodes all works fine, i can manually received the msgs distributed among the cluster.
Regards,
Yitao jiangyt.github.io
_______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
|
|
Re: 3 etcd nodes don't work well in single zone
|
|
cflinuxfs2 stack not fount under v210.2 release
Hi,
Does any body ever have the problem of deploying application failed in *cflinuxfs2 stack not found?* we deployed v210.2 release and occasionally it failed in* cflinuxfs2 not found* when deploying. It's weird that the nats with 2 nodes all works fine, i can manually received the msgs distributed among the cluster.
Regards,
Yitao jiangyt.github.io
|
|
Re: 3 etcd nodes don't work well in single zone
|
|
[buildpacks] PMC - 2015-07-27 notes
Happy Monday, We had a short meeting of the Buildpacks PMC today, permanent notes are at: https://github.com/cloudfoundry/pmc-notes/blob/master/Buildpacks/2015-07-27-buildpacks.mdwhich I've snapshotted below. Please note that we're going to move all Buildpacks PMC discussion to the cf-dev@ mailing list, as discussed in today's PMC call. That is, we won't be convening for the next month or so, and will instead communicate transparently on the mailing list. If you have objections or thoughts on this, please don't hesitate to reach out to either Chip or me. Cheers, -mike ---- *# Buildpacks PMC Meeting 2015-06-15* *## Agenda* 1. Update on Java Buildpack (Ryan Morgan) 2. Update on core Buildpacks and rootfs (Mike Dalessio) 3. Proposals * `null-buildpack` deprecation 4. Open Discussion * Moving Buildpacks PMC discussion to the mailing list *## Attendees* * Chip Childers, Cloud Foundry Foundation * Stormy Peters, Cloud Foundry Foundation * Mike Dalessio, Pivotal (PMC lead) * Ryan Morgan, Pivotal *## Update on Java Buildpack (Ryan Morgan)* * Java Buildpack 3.1 released on June 25th. [Release notes]( https://github.com/cloudfoundry/java-buildpack/releases/tag/v3.1) * Work continues on support for Hardware security devices. (Safenet Luna security provider) * Not much work remains in the backlog, next up is migration to Concourse *## Update on core Buildpacks and `rootfs` (Mike Dalessio)* *__Recent work__* Released ruby-buildpack v1.6.0, php-buildpack v4.0.0, python-buildpack v1.5.0, nodejs-buildpack v1.5.0 and staticfile-buildpack v1.2.1, which all contain CF-specific precompiled binaries. (NB the go-buildpack uses Google-built binaries and we're OK with this.) Released binary-buildpack v1.0.1 which contains Michael Fraenkel's (IBM) PR for Diego compatibility. Released go-buildpack 1.5.0 with a Godeps upgrade and removal of the custom python interpreter (now uses rootfs python). Updates to all buildpacks to remove lucid64 binaries from manifests. Operators can use the buildpack tooling to generate these binaries and package their own buildpacks as they deem necessary. Note that as part of this effort, the [binary builder][] tooling was open-sourced. Open-sourced [concourse CI pipelines][]. [Rootfs 1.1.0][] released. Note that rootfs releases are now given version numbers and have release notes, and the corresponding docker images are tagged with that version number. The team has been working on a series of spikes intended to drive towards a technical solution for implementing shared behaviors (across buildpacks) into shared code. This work can be followed in the Tracker backlog in the [architecture epic][]. We moved [buildpack-packager][], [machete][], and [machete-firewall-test][] into the `cloudfoundry` org, as they're part of our core tooling. These were previously in the deprecated `cf-buildpacks` org or were overlooked during the initial Foundation repository cleanup. [binary builder]: https://github.com/cloudfoundry/binary-builder [concourse CI pipelines]: https://github.com/cloudfoundry/buildpacks-ci [Rootfs 1.1.0]: https://github.com/cloudfoundry/stacks/releases/tag/1.1.0 [architecture epic]: https://www.pivotaltracker.com/epic/show/1898760 [buildpack-packager]: https://github.com/cloudfoundry/buildpack-packager [machete]: https://github.com/cloudfoundry/machete [machete-firewall-test]: https://github.com/cloudfoundry/machete-firewall-test*__Upcoming work__* The team will be experimenting with extension points to the buildpack life cycle, for both operators and developers. Among the goals: * allow extension of buildpack behavior by application developers without having to fork the buildpack * allow extension of buildpack behavior by operators which is currently not possible We're also considering focused sprints for individual buildpacks. *## Proposals* *### `null-buildpack` deprecation* The repository `cloudfoundry-incubator/null-buildpack` has been deprecated by the adoption of `cloudfoundry/binary-buildpack`, and so it's being proposed to move this to the attic or to delete it. After a brief discussion, it was decided to place this repository in `cloudfoundry-attic`. *## Open Discussion* *### Moving Buildpacks PMC discussion to the mailing list* Chip and Mike noted that attendance has not been regular at the Buildpacks PMC. Chip suggested that perhaps the Buidpacks PMC discussion could move to the mailing list, rather than continue to have the call. No objections from those present. We'll try using email, rather than waiting for a PMC call, for the next month; and we'll revisit this decision based on feedback.
|
|
Re: Logstash and Multiline Log Entry
Mike Youngstrom <youngm@...>
Yet another request for improved multi line log message handling. Is there any update from the LAMB team on plans to improve this problem? There have been several proposed solutions but I'm not aware of anything actually making it into the LAMB tracker. It would be great if we could hear from Erik on this issue. Does the LAMB team believe it is not an issue? Are there plans to improve this situation? Whatever the perspective lets discuss it as a community and see if there are any options better than the current. I'd really like to see something turned into a tracker issue if there are better options. Mike [0] http://lists.cloudfoundry.org/pipermail/cf-dev/2015-June/000423.html[1] http://lists.cloudfoundry.org/pipermail/cf-dev/2015-May/000083.html[2] https://groups.google.com/a/cloudfoundry.org/forum/?utm_medium=email&utm_source=footer#!msg/vcap-dev/B1W6_vO0oyo/84X1eAtFsKoJ On Mon, Jul 27, 2015 at 9:47 AM, Simon Johansson <simon(a)simonjohansson.com> wrote: This is a tricky one. Especially if you have more than one logstash machine doing filtering as they will do filtering independently of each other as the events come in.
The reason why CF adds a timestamp to each line is because how syslog works, where each line is its own even.
What we tend to do in my company is to log this kind of stuff via GELF or with Sentry.
On Mon, Jul 27, 2015 at 5:41 PM, Steve Wall <stevewallone(a)gmail.com> wrote:
Hello, We are sending CF logs message to an ELK stack. Multiline logs message are broken out into several log messages in Logstash. One end per line of the multiline log message. This is problematic when stack traces dumped to the log. Each line of the stack trace is translated into a log message. Trying to view this through Kibana is nearly impossible. Logstash provides a Grok feature allowing for the manipulation of the log messages. One common solution is to create a Grok filter that using a timestamp to indicate when a log entry starts and to combine all lines until the next timestamp into one log message. The problem is that CF adds a timestamp to every line. Has anyone come up with a good Grok expression to handle multiline log message coming out of CF? Thanks! Steve
_______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
_______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
|
|
Re: Logstash and Multiline Log Entry
Simon Johansson <simon@...>
This is a tricky one. Especially if you have more than one logstash machine doing filtering as they will do filtering independently of each other as the events come in.
The reason why CF adds a timestamp to each line is because how syslog works, where each line is its own even.
What we tend to do in my company is to log this kind of stuff via GELF or with Sentry.
toggle quoted message
Show quoted text
On Mon, Jul 27, 2015 at 5:41 PM, Steve Wall <stevewallone(a)gmail.com> wrote: Hello, We are sending CF logs message to an ELK stack. Multiline logs message are broken out into several log messages in Logstash. One end per line of the multiline log message. This is problematic when stack traces dumped to the log. Each line of the stack trace is translated into a log message. Trying to view this through Kibana is nearly impossible. Logstash provides a Grok feature allowing for the manipulation of the log messages. One common solution is to create a Grok filter that using a timestamp to indicate when a log entry starts and to combine all lines until the next timestamp into one log message. The problem is that CF adds a timestamp to every line. Has anyone come up with a good Grok expression to handle multiline log message coming out of CF? Thanks! Steve
_______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
|
|
Logstash and Multiline Log Entry
Hello, We are sending CF logs message to an ELK stack. Multiline logs message are broken out into several log messages in Logstash. One end per line of the multiline log message. This is problematic when stack traces dumped to the log. Each line of the stack trace is translated into a log message. Trying to view this through Kibana is nearly impossible. Logstash provides a Grok feature allowing for the manipulation of the log messages. One common solution is to create a Grok filter that using a timestamp to indicate when a log entry starts and to combine all lines until the next timestamp into one log message. The problem is that CF adds a timestamp to every line. Has anyone come up with a good Grok expression to handle multiline log message coming out of CF? Thanks! Steve
|
|
Re: CF // Login using REST API
On Mon, Jul 27, 2015 at 4:58 AM, Brena, Juan Antonio (ext) < juan.brena.external(a)atos.net> wrote: Good morning,
I have installed CF using the following installer: *https://github.com/yudai/cf_nise_installer* <https://github.com/yudai/cf_nise_installer>
Using the following request:
*https://api.CF_IP.xip.io/v2/info* <https://api.CF_IP.xip.io/v2/info>
Y receive the following information:
{ "name": "vcap", "build": "2222", "support": "http://support.cloudfoundry.com", "version": 2, "description": "Cloud Foundry sponsored by Pivotal", "authorization_endpoint": "https://uaa. CF_IP.xip.io", "token_endpoint": "https://uaa. CF_IP.xip.io", "min_cli_version": null, "min_recommended_cli_version": null, "api_version": "2.25.0", "logging_endpoint": "wss://loggregator. CF_IP.xip.io:4443" }
I would like to know, how to get a token (Login process) to get apps:
*https://api.CF_IP.xip.io/v2/apps* <https://api.CF_IP.xip.io/v2/apps>
Many thanks in advance.
If you just want to login and use your system, install the cf cli. https://github.com/cloudfoundry/cli/releasesThen run `cf api api.CF_IP.xip.io` followed by `cf login`. This will prompt you for your credentials. I can't recall what that installer set's up, but it'll be something like admin / admin or admin / admin-secret. Check the docs, it'll probably say. For additional help with cf, run `cf -h` to see the available commands or `cf command -h` to get details on a particular command. If you want to login programmatically (like if you're going to write an app to interact with CF) then you need to go through one of the Oauth2 work flows. If you're unfamiliar with Oauth2, google has lots of resources on the flows and how it works. As far as CF, a couple things to get you started... 1.) Run `CF_TRACE=true cf login` (or `set CF_TRACE=true` followed by `cf login` on Windows) and watch the output. It'll dump the HTTP request and response data, so you can see how the cli does it. 2.) Beyond that, more detail can be found here. https://github.com/cloudfoundry/uaa/tree/master/docsI have found this particular document to be useful. https://github.com/cloudfoundry/uaa/blob/master/docs/UAA-APIs.rstand of course, post any follow up questions to the list. Dan ________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
|
|
CF // Login using REST API
Brena, Juan Antonio (ext) <juan.brena.external@...>
Good morning, I have installed CF using the following installer: https://github.com/yudai/cf_nise_installerUsing the following request: https://api.CF_IP.xip.io/v2/infoY receive the following information: { "name": "vcap", "build": "2222", "support": " http://support.cloudfoundry.com", "version": 2, "description": "Cloud Foundry sponsored by Pivotal", "authorization_endpoint": " https://uaa. CF_IP.xip.io", "token_endpoint": " https://uaa. CF_IP.xip.io", "min_cli_version": null, "min_recommended_cli_version": null, "api_version": "2.25.0", "logging_endpoint": "wss://loggregator. CF_IP.xip.io:4443" } I would like to know, how to get a token (Login process) to get apps: https://api.CF_IP.xip.io/v2/appsMany thanks in advance. Juan Antonio This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, the Atos group liability cannot be triggered for the message content. Although the sender endeavors to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted. Este mensaje y los ficheros adjuntos pueden contener información confidencial destinada solamente a la(s) persona(s) mencionadas anteriormente y pueden estar protegidos por secreto profesional. Si usted recibe este correo electrónico por error, gracias por informar inmediatamente al remitente y destruir el mensaje. Al no estar asegurada la integridad de este mensaje sobre la red, Atos no se hace responsable por su contenido. Su contenido no constituye ningún compromiso para el grupo Atos, salvo ratificación escrita por ambas partes. Aunque se esfuerza al máximo por mantener su red libre de virus, el emisor no puede garantizar nada al respecto y no será responsable de cualesquiera daños que puedan resultar de una transmisión de virus.
|
|
Announcing cf-mysql-release v22
Happy Friday! On behalf of the CF Core Services team I am pleased to announce v22 of cf- mysql-release < https://github.com/cloudfoundry/cf-mysql-release/releases/tag/v22>. -- cf-mysql-release < https://github.com/cloudfoundry/cf-mysql-release> is a BOSH release that delivers a MySQL-compatible DataBase-as-a-Service for Cloud Foundry users. Through Cloud Foundry, users can provision databases and deliver unique credentials to bound applications. v22 is a minor update; we release regularly to make sure you have access to all the latest features and fixes. The major changes in v22 are updated MariaDB packages and several configuration and security changes. We're continuing to make the included stubs easier to use, in particular there's a new stub with this release to make it easy to customize the service plans. We've also expanded our documentation. Please send us feedback on how we're doing! For all the details, please see the release notes for v22 < https://github.com/cloudfoundry/cf-mysql-release/releases/tag/v22>. Special note for Amazon AWS users: We weren't satisfied with the difficulty of configuring cf-mysql-release across multiple AWS AZ's, so we've put some effort into making the stub much easier to follow. For v22, please feel free to use this pre-release version of the stub that will appear in v23. It's been tested and validated to work with v22: http://bit.ly/v23-sample-aws-stubIf you'd like to know more, detailed release notes can be found on GitHub < https://github.com/cloudfoundry/cf-mysql-release/releases/tag/v20> and bosh.io < http://bosh.io/releases/github.com/cloudfoundry/cf-mysql-release?version=20>. If you'd like to know even more than that, each line contains a link to the original story under which a feature was commissioned. As always, we'd love to hear if you're having any problems with this version of the software. Please open a GitHub issue < https://github.com/cloudfoundry/cf-mysql-release/issues>. If you're willing, we're always very happy to receive a Pull Request < https://github.com/cloudfoundry/cf-mysql-release/pulls>. PS - For those of who you might be wondering why we've skipped v21: A particularly large robot in the Release Engineering Department experienced a momentarily lapse of judgement. It did report that v21 was quite tasty, but unfortunately no longer available for general release. We've responded by adjusting the robot's diet and medications. Future releases should navigate the process unharmed. We apologize for the inconvenience, and send our condolences to v21's family and friends. -- Marco Nicosia Product Manager Pivotal Software, Inc. mnicosia(a)pivotal.io c: 650-796-2948
|
|
Re: Notifications on ORG, SPACE and USER modifications
I think ETags is reasonable thought as well.
toggle quoted message
Show quoted text
On Thu, Jul 23, 2015 at 4:39 PM, Benjamin Black <bblack(a)pivotal.io> wrote: ETags and a 304 response are specifically intended for that purpose. I'd recommend that over relying on Last-Modified.
b
On Thu, Jul 23, 2015 at 12:34 AM, Koper, Dies <diesk(a)fast.au.fujitsu.com> wrote:
Or setting the Last-Modified HTTP response header accordingly, and allow clients to use HTTP caching mechanisms (Last-Modified, etc.) to get quick empty responses with the current APIs if no changes have been made? (Or maybe this is already working so – haven’t checked).
Regards,
Dies Koper
*From:* cf-dev-bounces(a)lists.cloudfoundry.org [mailto: cf-dev-bounces(a)lists.cloudfoundry.org] *On Behalf Of *Matt Cowger *Sent:* Thursday, July 23, 2015 4:45 PM *To:* Discussions about Cloud Foundry projects and the system overall. *Subject:* Re: [cf-dev] Notifications on ORG, SPACE and USER modifications
I've wanted something similar as well.
On a related note, having a CC API 'serial' number (for each object in CC - apps, spaces, etc) that increments on every change relevant to that object would be of value for detecting if something has changed.
On Thu, Jul 23, 2015 at 3:27 PM, Dieu Cao <dcao(a)pivotal.io> wrote:
There are a few different approaches to this and different concerns that are possible.
The requests I've seen have been around wanting to be able to subscribe to and filter the various events that cc currently generates so that other behavior could be triggered.
We currently have events, app usage events, and service usage events.
Is it acceptable for the notifications to be lossy? Depends on the use case but If so, then the firehose may be an acceptable approach.
The CAPI team is currently focusing on other work in the near term, such as the v3 API and private brokers, but would be happy to collaborate on a proposal.
On Wed, Jul 22, 2015 at 2:05 PM, Juan Pablo Genovese < juanpgenovese(a)gmail.com> wrote:
My take:
CC should have callbacks on for each model create, update and delete methods. Those callbacks will send a message to an MQ, which you can subscribe to consume those messages.
This can be expanded to pretty much every event we need to track.
What do you think?
JP
2015-07-22 17:30 GMT-03:00 Matthias X Hub <matthias.hub(a)de.ibm.com>:
Hi,
we (=IBM) are also having the need and are currently investigating how to solve this. We plan to work on a proposal to discuss this further with the cf community. I'll keep you updated on that.
Regards, Matthias
From: Mike Youngstrom <youngm(a)gmail.com> To: "Discussions about Cloud Foundry projects and the system overall." <cf-dev(a)lists.cloudfoundry.org> Date: 22.07.2015 20:57 Subject: Re: [cf-dev] Notifications on ORG, SPACE and USER modifications Sent by: cf-dev-bounces(a)lists.cloudfoundry.org ------------------------------
We have the same need. Today we are polling the CC.
It would be nice for us also if we could get CC event notifications via something like the firehose.
Mike
On Wed, Jul 22, 2015 at 10:23 AM, Juan Pablo Genovese < juanpgenovese(a)gmail.com> wrote: I mean, I know you can list those events thru the API, but I want something that will react on an event instead of having to be constantly polling for them.
2015-07-22 13:18 GMT-03:00 Juan Pablo Genovese <juanpgenovese(a)gmail.com>:
Sree,
thanks! Any pointers on how can I hook up to these audit events?
Thank you!
2015-07-22 13:12 GMT-03:00 Sree Tummidi <stummidi(a)pivotal.io>: I believe there are audit events generated for all these actions which can be captured and forwarded to an SIEM solution like splunk
Thanks, Sree
Sent from my iPhone
On Jul 22, 2015, at 8:54 AM, Juan Pablo Genovese <juanpgenovese(a)gmail.com> wrote:
Guys,
I need to somehow hook up into the Cloud Controller (CC) to capture ORG, SPACE and USER deletion, insertion and update.
So far, I considered some approaches, such as forking the CC (the least favorite) and modifying the code with some hooks, tapping into Nginx to capture the requests, and using triggers in the database to capture each event and send the necessary info to a service.
What do you think? Any other idea you might have?
Thanks!
-- Mis mejores deseos, Best wishes, Meilleurs vœux,
Juan Pablo ------------------------------------------------------
http://www.jpgenovese.com _______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
_______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
-- Mis mejores deseos, Best wishes, Meilleurs vœux,
Juan Pablo ------------------------------------------------------ http://www.jpgenovese.com
-- Mis mejores deseos, Best wishes, Meilleurs vœux,
Juan Pablo ------------------------------------------------------ http://www.jpgenovese.com
_______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
_______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
_______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
--
Mis mejores deseos, Best wishes, Meilleurs vœux,
Juan Pablo ------------------------------------------------------
http://www.jpgenovese.com
_______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
_______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
--
-- Matt
_______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
_______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
-- -- Matt
|
|
Hello, just a test ,pls ignore
|
|
Re: Allow gorouter to log random headers.
James,
Yep - we're pretty much on the same page.
Only addition I would ask for is that the whitelist contain some "sensible" defaults (eg, Trace-Id, Span-Id) that are switched on by default; since then tight integration with tools like spring-cloud / buildpacks would work out the box.
D
toggle quoted message
Show quoted text
On 24 July 2015 at 16:26, James Bayer <jbayer(a)pivotal.io> wrote: shannon,
from what i'm reading here about the use case, the main interest is that a CF operator knows that their cf installation is more deeply integrated with a specific log parsing solution for all/many apps on that platform that choose to use it (whether than is ELK, Zipkin, etc). it does not sound like it is a special case with lots of variation by many different app teams on the cf installation. rather, it sounds like this is most likely to be used as an installation-wide option to enhance the app developer / app operations experience.
it seems like an operator configured whitelist set of headers that get logged with the RTR message satisfies the current needs well and is reasonable.
if we were to find that in the future lots of variation and different app teams on the same CF wanted to have the RTR tier log many different headers in the access log, then we could enhance the "log the whitelist of headers to the access log" capability to be exposed to a limited number of headers that each route could be configured by developers to specify that would apply in addition to the operator configured logged headers. but it sounds like that isn't needed right now.
simon/david, did i summarize this correctly?
On Fri, Jul 24, 2015 at 7:23 AM, Simon Johansson <simon(a)simonjohansson.com
wrote: If I understand correctly, you're proposing that an operator of CloudFoundry configure GoRouter, which is a multi-tenant, shared service, with knowledge specific to one or a few applications
That is indeed the proposal. In my org we work closely with the different development streams to provide a good out of the box experience by creating certain opt-in features that makes their life easier. This would be one of those features, if you pass certain headers in your requests thats standardised across the entire online organisation you will be able to query for them in Kibana. I understand that this is not the case in all environments.
If GoRouter logged whatever headers were included in the request, wouldn't this satisfy your requirements?
This would indeed satisfy my requests, but as David points out("However, not having a whitelist of headers to log opens a possible DDOS attack vector on the GoRouter") it might now be appropriate.
Based on what you've described, the persona is the app developer. Not necessarily, we operators are also very interested in certain headers.
so control of what is logged should be in their hands. If one via the API could set headers that should be logged for an org, space or application that would be magical. The need for a list of "must-log-all-these-headers" would still be there I think as not to have to maintain a list of these standardised headers across different objects.
On Thu, Jul 23, 2015 at 10:51 PM, Shannon Coen <scoen(a)pivotal.io> wrote:
This is not something that would be merged, as originally proposed, without additional investigation and discussion.
If I understand correctly, you're proposing that an operator of CloudFoundry configure GoRouter, which is a multi-tenant, shared service, with knowledge specific to one or a few applications. This should not be an operator responsibility, nor should the solution be specific to one or a few applications.
The goal is "the flexibility of being able to annotate our logs with what we consider to be important for our debugging purposes." More specifically you're requesting logging of headers. Do you have a preference?
If GoRouter logged whatever headers were included in the request, wouldn't this satisfy your requirements? Doesn't GoRouter do this already?
I'm interested in solving your requirement generically for all applications, and focussing the user experience on the correct persona. Based on what you've described, the persona is the app developer, so control of what is logged should be in their hands. I'm also not convinced GoRouter should have any knowledge of headers specific to one application or another.
Shannon Coen Product Manager, Cloud Foundry Pivotal, Inc.
On Wed, Jul 22, 2015 at 3:05 AM, Alex Lomov <alexander.lomov(a)altoros.com
wrote: Some time ago routing services were discussed on a CAB [1]. Here is a description of this proposal.
Do you think that using such service will allow your developers to cover this requirements?
[1] http://www.activestate.com/blog/2015/02/cloud-foundry-advisory-board-meeting-2015-february [2] https://docs.google.com/document/d/1bGOQxiKkmaw6uaRWGd-sXpxL0Y28d3QihcluI15FiIA/edit#heading=h.8djffzes9pnb
On Jul 21, 2015, at 4:06 PM, Simon Johansson <simon(a)simonjohansson.com> wrote:
Howdie!
We have some devs who want to be able to trace a request troughout their applications.
user -> a -> b -> c | |-> d -> e
When a user makes a request to "a" uuid is generated inside the app, and the request to "b" from "a" will set a header(call it WAKAWAKA to uuid), "b"'s call will passthrough WAKAWAKA to "c" and "d", "d" will passthrough WAKAWAKA to "e". Etc.
We aggregate all RTR logs into ELK so it would be super helpful to them to be able to filter on WAKAWAKA and get all the access logs(and app logs aswell, they mostly use GELF so its easy for them to add whatewher field they want) from the services involved.
I had a quick peek at the gorouter( https://github.com/cloudfoundry/gorouter/blob/76668f5818ea8c089ff52a14fcdfbf703c8e8767/access_log/access_log_record.go#L40) and it seems like this should be a simple PR.
1. To gorouter.yml add passthrough_headers: - WAKAWAKA - X-Random-Header
2. In makeRecord at the bottom add something like(in psuedo)
data = {} for header in passthrough_headers: header_val = r.FormatRequestHeader("X-Forwarded-For") if header_val: passthrough_headers[header] = header_val
if data: fmt.Fprintf(b, data.to_stringified_json())
That would yield a log line like blurgh.dev.cf.private.domain.com - [21/07/2015:10:17:05 +0000] "GET /statements?ascending=true&since=2015-06-30T14%3A10%3A03.078Z&skipStatementId=30a88204-0779-4385-9859-e4aabd30baf0 HTTP/1.1" 200 0 17 "-" "NING/1.0" 10.230.31.2:46204 x_forwarded_for:"-" vcap_request_id:1e58195a-cde6-4afd-7f03-43061c9ea91c response_time:0.004927106 app_id:9784cd03-050d-4b74-9e90-5f17134a3f08 {"WAKAWAKA": "Space is the place", "X-Random-Header": "Once upon a midnight dreary, while I pondered weak and weary"}
The reason for a stringified JSON is to make it easy to parse with logstash or other loganalysis tools.
Before I spend time implementing, is this something you would merge? _______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
_______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
_______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
_______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
-- Thank you,
James Bayer
_______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
|
|
Re: Allow gorouter to log random headers.
Simon Johansson <simon@...>
simon/david, did i summarize this correctly? Spot on summarisation. :) On Fri, Jul 24, 2015 at 5:26 PM, James Bayer <jbayer(a)pivotal.io> wrote: shannon,
from what i'm reading here about the use case, the main interest is that a CF operator knows that their cf installation is more deeply integrated with a specific log parsing solution for all/many apps on that platform that choose to use it (whether than is ELK, Zipkin, etc). it does not sound like it is a special case with lots of variation by many different app teams on the cf installation. rather, it sounds like this is most likely to be used as an installation-wide option to enhance the app developer / app operations experience.
it seems like an operator configured whitelist set of headers that get logged with the RTR message satisfies the current needs well and is reasonable.
if we were to find that in the future lots of variation and different app teams on the same CF wanted to have the RTR tier log many different headers in the access log, then we could enhance the "log the whitelist of headers to the access log" capability to be exposed to a limited number of headers that each route could be configured by developers to specify that would apply in addition to the operator configured logged headers. but it sounds like that isn't needed right now.
simon/david, did i summarize this correctly?
On Fri, Jul 24, 2015 at 7:23 AM, Simon Johansson <simon(a)simonjohansson.com
wrote: If I understand correctly, you're proposing that an operator of CloudFoundry configure GoRouter, which is a multi-tenant, shared service, with knowledge specific to one or a few applications
That is indeed the proposal. In my org we work closely with the different development streams to provide a good out of the box experience by creating certain opt-in features that makes their life easier. This would be one of those features, if you pass certain headers in your requests thats standardised across the entire online organisation you will be able to query for them in Kibana. I understand that this is not the case in all environments.
If GoRouter logged whatever headers were included in the request, wouldn't this satisfy your requirements?
This would indeed satisfy my requests, but as David points out("However, not having a whitelist of headers to log opens a possible DDOS attack vector on the GoRouter") it might now be appropriate.
Based on what you've described, the persona is the app developer. Not necessarily, we operators are also very interested in certain headers.
so control of what is logged should be in their hands. If one via the API could set headers that should be logged for an org, space or application that would be magical. The need for a list of "must-log-all-these-headers" would still be there I think as not to have to maintain a list of these standardised headers across different objects.
On Thu, Jul 23, 2015 at 10:51 PM, Shannon Coen <scoen(a)pivotal.io> wrote:
This is not something that would be merged, as originally proposed, without additional investigation and discussion.
If I understand correctly, you're proposing that an operator of CloudFoundry configure GoRouter, which is a multi-tenant, shared service, with knowledge specific to one or a few applications. This should not be an operator responsibility, nor should the solution be specific to one or a few applications.
The goal is "the flexibility of being able to annotate our logs with what we consider to be important for our debugging purposes." More specifically you're requesting logging of headers. Do you have a preference?
If GoRouter logged whatever headers were included in the request, wouldn't this satisfy your requirements? Doesn't GoRouter do this already?
I'm interested in solving your requirement generically for all applications, and focussing the user experience on the correct persona. Based on what you've described, the persona is the app developer, so control of what is logged should be in their hands. I'm also not convinced GoRouter should have any knowledge of headers specific to one application or another.
Shannon Coen Product Manager, Cloud Foundry Pivotal, Inc.
On Wed, Jul 22, 2015 at 3:05 AM, Alex Lomov <alexander.lomov(a)altoros.com
wrote: Some time ago routing services were discussed on a CAB [1]. Here is a description of this proposal.
Do you think that using such service will allow your developers to cover this requirements?
[1] http://www.activestate.com/blog/2015/02/cloud-foundry-advisory-board-meeting-2015-february [2] https://docs.google.com/document/d/1bGOQxiKkmaw6uaRWGd-sXpxL0Y28d3QihcluI15FiIA/edit#heading=h.8djffzes9pnb
On Jul 21, 2015, at 4:06 PM, Simon Johansson <simon(a)simonjohansson.com> wrote:
Howdie!
We have some devs who want to be able to trace a request troughout their applications.
user -> a -> b -> c | |-> d -> e
When a user makes a request to "a" uuid is generated inside the app, and the request to "b" from "a" will set a header(call it WAKAWAKA to uuid), "b"'s call will passthrough WAKAWAKA to "c" and "d", "d" will passthrough WAKAWAKA to "e". Etc.
We aggregate all RTR logs into ELK so it would be super helpful to them to be able to filter on WAKAWAKA and get all the access logs(and app logs aswell, they mostly use GELF so its easy for them to add whatewher field they want) from the services involved.
I had a quick peek at the gorouter( https://github.com/cloudfoundry/gorouter/blob/76668f5818ea8c089ff52a14fcdfbf703c8e8767/access_log/access_log_record.go#L40) and it seems like this should be a simple PR.
1. To gorouter.yml add passthrough_headers: - WAKAWAKA - X-Random-Header
2. In makeRecord at the bottom add something like(in psuedo)
data = {} for header in passthrough_headers: header_val = r.FormatRequestHeader("X-Forwarded-For") if header_val: passthrough_headers[header] = header_val
if data: fmt.Fprintf(b, data.to_stringified_json())
That would yield a log line like blurgh.dev.cf.private.domain.com - [21/07/2015:10:17:05 +0000] "GET /statements?ascending=true&since=2015-06-30T14%3A10%3A03.078Z&skipStatementId=30a88204-0779-4385-9859-e4aabd30baf0 HTTP/1.1" 200 0 17 "-" "NING/1.0" 10.230.31.2:46204 x_forwarded_for:"-" vcap_request_id:1e58195a-cde6-4afd-7f03-43061c9ea91c response_time:0.004927106 app_id:9784cd03-050d-4b74-9e90-5f17134a3f08 {"WAKAWAKA": "Space is the place", "X-Random-Header": "Once upon a midnight dreary, while I pondered weak and weary"}
The reason for a stringified JSON is to make it easy to parse with logstash or other loganalysis tools.
Before I spend time implementing, is this something you would merge? _______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
_______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
_______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
_______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
-- Thank you,
James Bayer
_______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
|
|