Logstash and Multiline Log Entry
Steve Wall
Hello,
We are sending CF logs message to an ELK stack. Multiline logs message are broken out into several log messages in Logstash. One end per line of the multiline log message. This is problematic when stack traces dumped to the log. Each line of the stack trace is translated into a log message. Trying to view this through Kibana is nearly impossible. Logstash provides a Grok feature allowing for the manipulation of the log messages. One common solution is to create a Grok filter that using a timestamp to indicate when a log entry starts and to combine all lines until the next timestamp into one log message. The problem is that CF adds a timestamp to every line. Has anyone come up with a good Grok expression to handle multiline log message coming out of CF? Thanks! Steve
|
|
Re: CF // Login using REST API
Daniel Mikusa
On Mon, Jul 27, 2015 at 4:58 AM, Brena, Juan Antonio (ext) <
juan.brena.external(a)atos.net> wrote: Good morning, If you just want to login and use your system, install the cf cli. https://github.com/cloudfoundry/cli/releases Then run `cf api api.CF_IP.xip.io` followed by `cf login`. This will prompt you for your credentials. I can't recall what that installer set's up, but it'll be something like admin / admin or admin / admin-secret. Check the docs, it'll probably say. For additional help with cf, run `cf -h` to see the available commands or `cf command -h` to get details on a particular command. If you want to login programmatically (like if you're going to write an app to interact with CF) then you need to go through one of the Oauth2 work flows. If you're unfamiliar with Oauth2, google has lots of resources on the flows and how it works. As far as CF, a couple things to get you started... 1.) Run `CF_TRACE=true cf login` (or `set CF_TRACE=true` followed by `cf login` on Windows) and watch the output. It'll dump the HTTP request and response data, so you can see how the cli does it. 2.) Beyond that, more detail can be found here. https://github.com/cloudfoundry/uaa/tree/master/docs I have found this particular document to be useful. https://github.com/cloudfoundry/uaa/blob/master/docs/UAA-APIs.rst and of course, post any follow up questions to the list. Dan ________________________________ cf-dev mailing list
|
|
CF // Login using REST API
Brena, Juan Antonio (ext) <juan.brena.external@...>
Good morning,
I have installed CF using the following installer: https://github.com/yudai/cf_nise_installer Using the following request: https://api.CF_IP.xip.io/v2/info Y receive the following information: { "name": "vcap", "build": "2222", "support": "http://support.cloudfoundry.com", "version": 2, "description": "Cloud Foundry sponsored by Pivotal", "authorization_endpoint": "https://uaa. CF_IP.xip.io", "token_endpoint": "https://uaa. CF_IP.xip.io", "min_cli_version": null, "min_recommended_cli_version": null, "api_version": "2.25.0", "logging_endpoint": "wss://loggregator. CF_IP.xip.io:4443" } I would like to know, how to get a token (Login process) to get apps: https://api.CF_IP.xip.io/v2/apps Many thanks in advance. Juan Antonio This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, the Atos group liability cannot be triggered for the message content. Although the sender endeavors to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted. Este mensaje y los ficheros adjuntos pueden contener información confidencial destinada solamente a la(s) persona(s) mencionadas anteriormente y pueden estar protegidos por secreto profesional. Si usted recibe este correo electrónico por error, gracias por informar inmediatamente al remitente y destruir el mensaje. Al no estar asegurada la integridad de este mensaje sobre la red, Atos no se hace responsable por su contenido. Su contenido no constituye ningún compromiso para el grupo Atos, salvo ratificación escrita por ambas partes. Aunque se esfuerza al máximo por mantener su red libre de virus, el emisor no puede garantizar nada al respecto y no será responsable de cualesquiera daños que puedan resultar de una transmisión de virus.
|
|
Announcing cf-mysql-release v22
Marco Nicosia
Happy Friday!
On behalf of the CF Core Services team I am pleased to announce v22 of cf- mysql-release <https://github.com/cloudfoundry/cf-mysql-release/releases/tag/v22>. -- cf-mysql-release <https://github.com/cloudfoundry/cf-mysql-release> is a BOSH release that delivers a MySQL-compatible DataBase-as-a-Service for Cloud Foundry users. Through Cloud Foundry, users can provision databases and deliver unique credentials to bound applications. v22 is a minor update; we release regularly to make sure you have access to all the latest features and fixes. The major changes in v22 are updated MariaDB packages and several configuration and security changes. We're continuing to make the included stubs easier to use, in particular there's a new stub with this release to make it easy to customize the service plans. We've also expanded our documentation. Please send us feedback on how we're doing! For all the details, please see the release notes for v22 <https://github.com/cloudfoundry/cf-mysql-release/releases/tag/v22>. Special note for Amazon AWS users: We weren't satisfied with the difficulty of configuring cf-mysql-release across multiple AWS AZ's, so we've put some effort into making the stub much easier to follow. For v22, please feel free to use this pre-release version of the stub that will appear in v23. It's been tested and validated to work with v22: http://bit.ly/v23-sample-aws-stub If you'd like to know more, detailed release notes can be found on GitHub <https://github.com/cloudfoundry/cf-mysql-release/releases/tag/v20> and bosh.io <http://bosh.io/releases/github.com/cloudfoundry/cf-mysql-release?version=20>. If you'd like to know even more than that, each line contains a link to the original story under which a feature was commissioned. As always, we'd love to hear if you're having any problems with this version of the software. Please open a GitHub issue <https://github.com/cloudfoundry/cf-mysql-release/issues>. If you're willing, we're always very happy to receive a Pull Request <https://github.com/cloudfoundry/cf-mysql-release/pulls>. PS - For those of who you might be wondering why we've skipped v21: A particularly large robot in the Release Engineering Department experienced a momentarily lapse of judgement. It did report that v21 was quite tasty, but unfortunately no longer available for general release. We've responded by adjusting the robot's diet and medications. Future releases should navigate the process unharmed. We apologize for the inconvenience, and send our condolences to v21's family and friends. -- Marco Nicosia Product Manager Pivotal Software, Inc. mnicosia(a)pivotal.io c: 650-796-2948
|
|
Re: Notifications on ORG, SPACE and USER modifications
Matt Cowger
I think ETags is reasonable thought as well.
toggle quoted messageShow quoted text
On Thu, Jul 23, 2015 at 4:39 PM, Benjamin Black <bblack(a)pivotal.io> wrote:
ETags and a 304 response are specifically intended for that purpose. I'd --
-- Matt
|
|
Hello, just a test ,pls ignore
Xiao
|
|
Re: Allow gorouter to log random headers.
David Laing
James,
toggle quoted messageShow quoted text
Yep - we're pretty much on the same page. Only addition I would ask for is that the whitelist contain some "sensible" defaults (eg, Trace-Id, Span-Id) that are switched on by default; since then tight integration with tools like spring-cloud / buildpacks would work out the box. D
On 24 July 2015 at 16:26, James Bayer <jbayer(a)pivotal.io> wrote:
shannon,
|
|
Re: Allow gorouter to log random headers.
Simon Johansson <simon@...>
simon/david, did i summarize this correctly?Spot on summarisation. :) On Fri, Jul 24, 2015 at 5:26 PM, James Bayer <jbayer(a)pivotal.io> wrote: shannon,
|
|
Re: Allow gorouter to log random headers.
James Bayer
shannon,
from what i'm reading here about the use case, the main interest is that a CF operator knows that their cf installation is more deeply integrated with a specific log parsing solution for all/many apps on that platform that choose to use it (whether than is ELK, Zipkin, etc). it does not sound like it is a special case with lots of variation by many different app teams on the cf installation. rather, it sounds like this is most likely to be used as an installation-wide option to enhance the app developer / app operations experience. it seems like an operator configured whitelist set of headers that get logged with the RTR message satisfies the current needs well and is reasonable. if we were to find that in the future lots of variation and different app teams on the same CF wanted to have the RTR tier log many different headers in the access log, then we could enhance the "log the whitelist of headers to the access log" capability to be exposed to a limited number of headers that each route could be configured by developers to specify that would apply in addition to the operator configured logged headers. but it sounds like that isn't needed right now. simon/david, did i summarize this correctly? On Fri, Jul 24, 2015 at 7:23 AM, Simon Johansson <simon(a)simonjohansson.com> wrote: If I understand correctly, you're proposing that an operator ofCloudFoundry configure GoRouter, which is a multi-tenant, shared service, -- Thank you, James Bayer
|
|
Re: How cloud foundry deployed by bosh-lite manage DEA
Daniel Mikusa
On Fri, Jul 24, 2015 at 11:17 AM, GuopingZhang <goodier_cn(a)hotmail.com>
wrote: Dan, thanks a lot for your explanation!Yes. That's my understanding of it. Dan
|
|
Re: How cloud foundry deployed by bosh-lite manage DEA
GuopingZhang
Dan, thanks a lot for your explanation!So you mean, warden container can have its own sub-warden container ?
toggle quoted messageShow quoted text
Guoping Date: Fri, 24 Jul 2015 09:13:13 -0400 From: dmikusa(a)pivotal.io To: cf-dev(a)lists.cloudfoundry.org Subject: Re: [cf-dev] How cloud foundry deployed by bosh-lite manage DEA
On Fri, Jul 24, 2015 at 8:35 AM, GuopingZhang <goodier_cn(a)hotmail.com> wrote:
I read that bosh-lite use warden as container for cloud foundry components.I deployed two java spring apps into the local cloud foundry. Below is my VMs (warden container), which (two) warden container is running my 2 apps? Or the apps are actually running as java process in my virtualbox of unbuntu (which contains all below containers), without running inside a warden container? It's a little different because you're using bosh-lite. In that case, you have one VM (probably running in Virtualbox) and that VM has some containers on it. Those containers run the various components of CF. Inside the DEA container (labeled "runner_z1" below), you'll find your apps running each in it's own sub-container. Dan +------------------------------------+---------+---------------+--------------+| Job/index | State | Resource Pool | IPs |+------------------------------------+---------+---------------+--------------+| api_z1/0 | running | large_z1 | 10.244.0.134 || doppler_z1/0 | running | medium_z1 | 10.244.0.142 || etcd_z1/0 | running | medium_z1 | 10.244.0.42 || ha_proxy_z1/0 | running | router_z1 | 10.244.0.34 || hm9000_z1/0 | running | medium_z1 | 10.244.0.138 || loggregator_trafficcontroller_z1/0 | running | small_z1 | 10.244.0.146 || nats_z1/0 | running | medium_z1 | 10.244.0.6 || postgres_z1/0 | running | medium_z1 | 10.244.0.30 || router_z1/0 | running | router_z1 | 10.244.0.22 || runner_z1/0 | running | runner_z1 | 10.244.0.26 | | uaa_z1/0 | running | medium_z1 | 10.244.0.130 |+------------------------------------+---------+---------------+--------------+ _______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org https://lists.cloudfoundry.org/mailman/listinfo/cf-dev _______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
|
|
Re: UAA: How to set client_credentials token grant type to not expire
Filip Hanik
public static final intMAX_VALUE
toggle quoted messageShow quoted text
<http://docs.oracle.com/javase/7/docs/api/java/lang/Integer.html#MAX_VALUE> 2147483647 that's the largest value you can set. However, I just realized, that if you set this value, the UAA will get an overflow and the int will become negative. so set the value to 315 569 260 that's a 10 year expiration. the computer that retrieved that token won't be around in 10 years so it's a safe number.
On Friday, July 24, 2015, Kayode Odeyemi <dreyemi(a)gmail.com> wrote:
Thanks Filip.
|
|
Re: Allow gorouter to log random headers.
Simon Johansson <simon@...>
If I understand correctly, you're proposing that an operator ofCloudFoundry configure GoRouter, which is a multi-tenant, shared service, with knowledge specific to one or a few applications That is indeed the proposal. In my org we work closely with the different development streams to provide a good out of the box experience by creating certain opt-in features that makes their life easier. This would be one of those features, if you pass certain headers in your requests thats standardised across the entire online organisation you will be able to query for them in Kibana. I understand that this is not the case in all environments. If GoRouter logged whatever headers were included in the request,wouldn't this satisfy your requirements? This would indeed satisfy my requests, but as David points out("However, not having a whitelist of headers to log opens a possible DDOS attack vector on the GoRouter") it might now be appropriate. Based on what you've described, the persona is the app developer.Not necessarily, we operators are also very interested in certain headers. so control of what is logged should be in their hands.If one via the API could set headers that should be logged for an org, space or application that would be magical. The need for a list of "must-log-all-these-headers" would still be there I think as not to have to maintain a list of these standardised headers across different objects. On Thu, Jul 23, 2015 at 10:51 PM, Shannon Coen <scoen(a)pivotal.io> wrote: This is not something that would be merged, as originally proposed,
|
|
Re: Did anybody deploy a wiki as app to CF?
James Bayer
what an exciting effort!
lots of things work with CF, it's just a matter of curating some good information. maybe i can get Yudai to pair with me on some translations! On Thu, Jul 23, 2015 at 10:41 PM, Noburou TANIGUCHI <dev(a)nota.m001.jp> wrote: Thank you, all. -- Thank you, James Bayer
|
|
Re: UAA: How to set client_credentials token grant type to not expire
Paul Bakare
Thanks Filip.
toggle quoted messageShow quoted text
However, everytime I set that (for example, I set it to 2542838400), I get this: "The request sent by the client was syntactically incorrect." That's December 31st 2050 when decoded. Is it possible that, that number is too large?
On Thu, Jul 23, 2015 at 7:09 PM, Filip Hanik <fhanik(a)pivotal.io> wrote:
|
|
Re: How cloud foundry deployed by bosh-lite manage DEA
Daniel Mikusa
On Fri, Jul 24, 2015 at 8:35 AM, GuopingZhang <goodier_cn(a)hotmail.com>
wrote: It's a little different because you're using bosh-lite. In that case, you have one VM (probably running in Virtualbox) and that VM has some containers on it. Those containers run the various components of CF. Inside the DEA container (labeled "runner_z1" below), you'll find your apps running each in it's own sub-container. Dan | uaa_z1/0 | running | medium_z1 | 10.244.0.130 |
|
|
Re: How to update blobs in blob.cfblob.com ?
Alexander Lomov <alexander.lomov@...>
Thank you for the answer.
toggle quoted messageShow quoted text
I like the idea with pathing existing sources. Going to make another PR soon. Have a good day, Alex L.
On Jul 24, 2015, at 3:36 PM, Matthew Sykes <matthew.sykes(a)gmail.com<mailto:matthew.sykes(a)gmail.com>> wrote:
Hi. This isn't what I expected. You've basically changed the bucket configuration to point to your own and uploaded the blobs to that bucket. That's why bosh is telling you there's nothing to upload. There's also a general expectation that blobs are pristine artifacts from a well known distribution point. In this case, that's not true. You've used a script to modify some content and there's no way for someone to know that without looking in detail so it's going to be tough to trust what's there. Regardless, given what your scripts are doing, you should be able to generate patches for each of the packages, add them as source artifacts, and update the packaging scripts to apply the patches before the build. That's probably how this should be done to keep things sane. On Fri, Jul 24, 2015 at 4:37 AM, Lomov Alexander <alexander.lomov(a)altoros.com<mailto:alexander.lomov(a)altoros.com>> wrote: Hi, Matthew. After I made `bosh upload blobs`, `bosh blobs` command tells me `No blobs to upload`. Still I created a pull request where you can figure out what blobs are changed from `config/blobs.yml` file [1]. Could you tell if it is okey? [1] https://github.com/cloudfoundry/bosh/pull/892 On Jul 1, 2015, at 5:38 PM, Matthew Sykes <matthew.sykes(a)gmail.com<mailto:matthew.sykes(a)gmail.com>> wrote: Since you won't be able to upload the blobs to the cf-release bucket, I'd suggest you capture the output of `bosh blobs` in your pull request. That command should enumerate all of the new blobs and their sizes. For each entry that's there, point to a publicly available URL and a hash that can be used to verify it. When the PR is reviewed, if things look good, the pair will likely pull the blobs down to evaluate them and test the overall function. On Wed, Jul 1, 2015 at 6:57 AM, Alexander Lomov <alexander.lomov(a)altoros.com<mailto:alexander.lomov(a)altoros.com>> wrote: Hi, all. I work on adding support of Power architecture to CF. During the work I needed to update not only cf-release, but existing blobs (postgresql, mysql client and etc.). I wonder how I can make PR to cf-release project with updated blobs. I couldn't find any clue in the contributing guild [1] , so I've decided to write here. [1] https://github.com/cloudfoundry/cf-release/blob/master/CONTRIBUTING.md Thank you, Alex L. ------------------------ Alex Lomov Altoros — Cloud Foundry deployment, training and integration Twitter: @code1n<https://twitter.com/code1n> GitHub: @allomov<https://gist.github.com/allomov> _______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org> https://lists.cloudfoundry.org/mailman/listinfo/cf-dev -- Matthew Sykes matthew.sykes(a)gmail.com<mailto:matthew.sykes(a)gmail.com> _______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org> https://lists.cloudfoundry.org/mailman/listinfo/cf-dev _______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org> https://lists.cloudfoundry.org/mailman/listinfo/cf-dev -- Matthew Sykes matthew.sykes(a)gmail.com<mailto:matthew.sykes(a)gmail.com> _______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org> https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
|
|
Re: How to update blobs in blob.cfblob.com ?
Matthew Sykes <matthew.sykes@...>
Hi. This isn't what I expected. You've basically changed the bucket
configuration to point to your own and uploaded the blobs to that bucket. That's why bosh is telling you there's nothing to upload. There's also a general expectation that blobs are pristine artifacts from a well known distribution point. In this case, that's not true. You've used a script to modify some content and there's no way for someone to know that without looking in detail so it's going to be tough to trust what's there. Regardless, given what your scripts are doing, you should be able to generate patches for each of the packages, add them as source artifacts, and update the packaging scripts to apply the patches before the build. That's probably how this should be done to keep things sane. On Fri, Jul 24, 2015 at 4:37 AM, Lomov Alexander < alexander.lomov(a)altoros.com> wrote: Hi, Matthew. -- Matthew Sykes matthew.sykes(a)gmail.com
|
|
How cloud foundry deployed by bosh-lite manage DEA
GuopingZhang
I read that bosh-lite use warden as container for cloud foundry components.I deployed two java spring apps into the local cloud foundry.
Below is my VMs (warden container), which (two) warden container is running my 2 apps? Or the apps are actually running as java process in my virtualbox of unbuntu (which contains all below containers), without running inside a warden container? +------------------------------------+---------+---------------+--------------+| Job/index | State | Resource Pool | IPs |+------------------------------------+---------+---------------+--------------+| api_z1/0 | running | large_z1 | 10.244.0.134 || doppler_z1/0 | running | medium_z1 | 10.244.0.142 || etcd_z1/0 | running | medium_z1 | 10.244.0.42 || ha_proxy_z1/0 | running | router_z1 | 10.244.0.34 || hm9000_z1/0 | running | medium_z1 | 10.244.0.138 || loggregator_trafficcontroller_z1/0 | running | small_z1 | 10.244.0.146 || nats_z1/0 | running | medium_z1 | 10.244.0.6 || postgres_z1/0 | running | medium_z1 | 10.244.0.30 || router_z1/0 | running | router_z1 | 10.244.0.22 || runner_z1/0 | running | runner_z1 | 10.244.0.26 || uaa_z1/0 | running | medium_z1 | 10.244.0.130 |+------------------------------------+---------+---------------+--------------+
|
|
Re: Allow gorouter to log random headers.
David Laing
Shannon, Simon,
As the lead of the logsearch.io (ELK) project; I'm also interested in having GoRouter log additional headers. Specifically Trace-Id and Span-Id generated by the spring-cloud-sleuth project (https://github.com/spring-cloud-incubator/spring-cloud-sleuth ). If GoRouter logged whatever headers were included in the request, wouldn'tThis would certainly satisfy my requirements, and I think Simon's too. However, not having a whitelist of headers to log opens a possible DDOS attack vector on the GoRouter, so I think having a operator configureable whitelist (with some sensible defaults like Trace-Id and Span-Id) is the right approach. Doesn't GoRouter do this already?I don't think so. Specifically, sending the following curl to an app hosted on PWS: curl --header "Trace-Id: 1c884728-466c-4ba3-8caa-a02a780c6d56" http://www.birdsangola.org/ Gives the following [RTR] log from loggregator: Fri Jul 24 2015 13:10:52 GMT+0100 (BST) [RTR] OUT www.birdsangola.org - [24/07/2015:12:10:52 +0000] "GET / HTTP/1.1" 200 0 7772 "-" "curl/7.30.0" 10.10.2.185:46765 x_forwarded_for:"92.40.249.226" vcap_request_id:3a33d5f6-dc11-42c4-61c7-32a1a2557200 response_time:0.001380276 app_id:0c34cc9f-45a8-440e-b876-b0cde564fbe3 It doesn't look like the extra Trace-Id header has been passed through to the loggregator [RTR] log. I'd be happy to work with Simon to contribute to a PR that implements the "above whitelist of headers to log" feature. Thoughts? :D -- View this message in context: http://cf-dev.70369.x6.nabble.com/cf-dev-Allow-gorouter-to-log-random-headers-tp800p877.html Sent from the CF Dev mailing list archive at Nabble.com.
|
|