Date
1 - 19 of 19
CF integration with logger and monitoring tools
|
Johannes Tuchscherer
Hi Swatz,
toggle quoted message
Show quoted text
In your config file, you will have to use 'datadog-firehose-nozzle' as the username and the password you specified when you created the uaa client. And no, there is no need to restart UAA. On Wed, Aug 26, 2015 at 2:59 AM, Swatz bosh <swatzron(a)gmail.com> wrote:
Can someone please help me with above data-dog error? |
|
|
|
Swatz bosh
Can someone please help me with above data-dog error?
|
|
|
|
Swatz bosh
I am getting error while running data-dog-nozzle -
https://github.com/cloudfoundry-incubator/datadog-firehose-nozzle go run main.go -config config/datadog-firehose-nozzle.json -----------ERROR------------- 2015/08/25 12:09:50 Starting DataDog Firehose Nozzle... 2015/08/25 12:09:50 Error while reading from the firehose: Unauthorized error: You are not authorized. Error: Invalid authorization 2015/08/25 12:09:50 Closing connection with traffic controller due to Unauthorized error: You are not authorized. Error: Invalid authorization 2015/08/25 12:09:50 Posting 3 metrics 2015/08/25 12:09:50 DataDog Firehose Nozzle shutting down... ---------------------------------------- below command of uaac, gives scope etc. uaac client get datadog-firehose-nozzle scope: doppler.firehose oauth.approvals openid client_id: datadog-firehose-nozzle resource_ids: none authorized_grant_types: authorization_code client_credentials refresh_token autoapprove: authorities: doppler.firehose oauth.login lastmodified: 1440494682000 I have added this property using - uaac client add datadog-firehose-nozzle --scope openid,oauth.approvals,doppler.firehose --authorized_grant_types authorization_code,client_credentials,refresh_token --authorities oauth.login,doppler.firehose I have used UAA client admin username/pswd in my nozzle config file and have also tried UAA client login username/pswd but same error. Do I need to restart uaa job after adding datadog-nozzle client? |
|
|
|
Rohit Kumar
Correct, the datadog-firehose-nozzle has its own custom client which makes
toggle quoted message
Show quoted text
REST calls to the datadog API. It doesn't use the datadog statsd agent. A similar approach should work for you. Thanks, Rohit On Mon, Aug 24, 2015 at 1:38 AM, Swatz bosh <swatzron(a)gmail.com> wrote:
Incase of datadog nozzle |
|
|
|
Swatz bosh
Incase of datadog nozzle https://github.com/cloudfoundry-incubator/datadog-firehose-nozzle
I am assuming that - datadogclient is a custom client which is a kind of aggregator which is fetching metrics from doppler. So this client is not using DataDogStasD client and its a custom client. I found that DataDog also has agent(install on every machine) which contains this stasd client, which datadog nozzle is not using, I think? So I can have custom aggregator for willy similarly, is it correct? |
|
|
|
Swatz bosh
Thanks James.
So for Wily introscope( wily-agent based monitoring tool) integration, if I go with firehose approach, I need to write my own nozzle for Introscope like for datadog ? https://github.com/cloudfoundry-incubator/datadog-firehose-nozzle I am assuming that I will be using dea-agent to fetch metrics and logs instead of wily-agent? My introscope nozzle will call doppler to fetch metrics and Introscope forwarder will forward it to Introscope Enterprise Manager (a server which has web-client to access metrics dashboard) ? |
|
|
|
James Bayer
for 1., getting custom metrics from an app container (JMX, some business
toggle quoted message
Show quoted text
foo metric happened, etc) is a goal for the logging team, but it is lower than existing work to remove the collector. [image: Inline image 1] 2. Can I start and stop log collection from firehose anytime? the nozzle clients connect over websocket, so yes i believe you can simply disconnect them and then reconnect them later. On Thu, Aug 20, 2015 at 2:32 AM, Swatz bosh <swatzron(a)gmail.com> wrote:
Hi, --
Thank you, James Bayer |
|
|
|
Swatz bosh
Hi,
So am assuming that - to integrate tool like Introscope, if I go with firehose approach then would need to write a custom nozzle/forwarder? I have couple queries - 1. Can I add custom Metric for my java application apart from the one which is mention here of firehose catalog [1] ? so that on Wily Introscope UI, I can have more sliced dashboard apart from these 8-10 metrics. 2. Can I start and stop log collection from firehose anytime? from my Introscope UI. The idea is to reduce the number of API calls from Introscope to Doppler firehose and giving flexibility to Wily-Introscope admin to collect data either periodically by running etl's or through UI. So will firehose provide REST apis to be called directly from Introscope or while writing custom nozzle for Introscope I should take care of such REST endpoints for nozzle? |
|
|
|
Gwenn Etourneau
" There is no way to to pull logs and metrics of entire CF instance into
toggle quoted message
Show quoted text
such agent based monitoring tools, it has to be binded everytime for each app ?" Fork the java-buildpack add your agent and add the service to be bind into a manifest file. Bad solution : Fork the java-buildpack add your agent and harcoded the endpoint access. On Fri, Aug 14, 2015 at 3:16 PM, Swatz bosh <swatzron(a)gmail.com> wrote:
Can you please help me on above query? |
|
|
|
Swatz bosh
Thanks for your reply Daniel.
So you are saying that - for any agent based monitoring tools like - NewRelic, AppDynamics, Wily-Introscope, Dyna-trace etc. firehose approach doesn't make any sense, and it has to be implemented only through user-defined service and which is binded to apps ?? and there is no way to to pull logs and metrics of entire CF instance into such agent based monitoring tools, it has to be binded everytime for each app ? |
|
|
|
Daniel Mikusa
I think you're talking about two separate things here:
toggle quoted message
Show quoted text
If you create a user defined service and bind that to your application [1], the Java build pack should install the agent and configure it to monitor your application. That's the key here, it will monitor that *one* application. This will work very similar to if you ran your app locally with the wily agent installed. You can then repeat this process for any number of Java apps running on CF. When you're talking about collector & firehose, you're talking about ways to pull logs and metrics from the entire CF system, not just a single app. This is unrelated to what is supported through the Java build pack. I have no idea if Interscope is capable of doing this or if it would make sense to use it for this type of data gathering. Dan [1] - https://github.com/cloudfoundry/java-buildpack/blob/master/docs/framework-introscope_agent.md On Mon, Aug 17, 2015 at 7:40 AM, Swatz bosh <swatzron(a)gmail.com> wrote:
Thanks Gwenn. |
|
|
|
Swatz bosh
Thanks Gwenn.
I found a link for integrating AppDynamics (agent based monitoring like Wily) with PCF, where its mentioned to create user-defined service and bind it to application, so that AppDynamic agent will detect the service 'VCAP_SERVICE' and start sending metrics/logs to AppDynamic collector. http://blog.pivotal.io/pivotal-cloud-foundry/products/getting-started-with-pivotal-cloud-foundry-and-appdynamics I am using here they are using 'collector' and not firehose nozzle? So does that mean I should not be using (java_buildpack agent of introscope) https://github.com/cloudfoundry/java-buildpack/blob/master/docs/framework-introscope_agent.md where introscope will need VCAP_SERVICE i.e. service binding with substring as 'introscope' to all my java apps ? I think its old 'collector' approach like AppDynamic ? And I should not be creating in user defined service for Introscope and bind to my java apps? I should rely on dea and metron to provide java application metrics and configure my firehose nozzle to point to Introscope-Enterprise Manager to collect all App metrics, is it correct? If yes, then are we saying firehouse metric will be auto understood by Introscope-Enterprise Manager? Thanks for all your response. |
|
|
|
Gwenn Etourneau
I think the easy way is to use the provided logs system by CF, so
toggle quoted message
Show quoted text
deaagent-> metron -> dopler -> Firehose consumer - forwarder -> Willy Will be strange that Willy agent / or server don't provide the specification of their format or any forwarder exist. I found some opensource things about Wily so maybe you can find some usefull work there. https://github.com/nickman/wiex On Mon, Aug 17, 2015 at 5:42 PM, Swati Goyal <swatzron(a)gmail.com> wrote:
Thanks James for your reply. |
|
|
|
Swatz bosh
Thanks James for your reply.
toggle quoted message
Show quoted text
So you are recommending firehose nozzles for sending logs and metrics to 3rd party application monitoring and logging tools. For product like Wily Introscope (now known as APM), which is agent based monitoring approach, and I think these agents right now only support Java/.Net technologies, which sends such metrics to Wily-Enterprise Manager. I have found that cf has released Java-buildpack for this Introscope agent ( https://github.com/cloudfoundry/java-buildpack/blob/master/lib/java_buildpack/framework/introscope_agent.rb ). So how would the communication would work with such Introscope agent? Will it still need nozzle? like - introscope agent (assuming Java application) will send Metrics/Logs to cf metron-agent which in-turn send it to cf doppler, then have firehose-nozzle to consume logs/metrics from doppler and send it to Enterprise Manager? Is this the flow or have to follow someother approach for such agent based monitoring tool? On Mon, Aug 17, 2015 at 1:12 PM, James Bayer <jbayer(a)pivotal.io> wrote:
i wrote back last week, but it looks like it was swallowed and never |
|
|
|
James Bayer
i wrote back last week, but it looks like it was swallowed and never posted?
toggle quoted message
Show quoted text
---------- Forwarded message ---------- From: James Bayer <jbayer(a)pivotal.io> Date: Fri, Aug 14, 2015 at 6:33 AM Subject: Re: [cf-dev] Re: CF integration with logger and monitoring tools To: "Discussions about Cloud Foundry projects and the system overall." < cf-dev(a)lists.cloudfoundry.org> cf is moving toward the loggregator firehose nozzle approach for sending stuff in cf to other systems. most logging products and services support syslog. here is a syslog nozzle for the firehose that should work well with splunk and similar: https://github.com/cloudfoundry-community/firehose-to-syslog if you're sending metrics, then you can look at something like the datadog nozzle for integration with whatever metrics service you are using: https://github.com/cloudfoundry-incubator/datadog-firehose-nozzle the current system that is being replaced is called the collector [1], which has plugins for various providers. i hope that the CF ecosystem will start having nozzles for many different metrics providers over time similar to the collector. [1] https://github.com/cloudfoundry/collector/tree/master/lib/collector/historian On Thu, Aug 13, 2015 at 11:14 PM, Swatz bosh <swatzron(a)gmail.com> wrote:
Can you please help me with above query? -- Thank you, James Bayer --
Thank you, James Bayer |
|
|
|
Swatz bosh
Hi,
Can someone help me on my query please? Thanks |
|
|
|
Swatz bosh
Can you please help me on above query?
|
|
|
|
Swatz bosh
Can you please help me with above query?
|
|
|
|
Swatz bosh
Hi,
I would like to know in CF, how to integrate with 3rd party app logging and monitoring like Graphite, Nagios, etc, what is the recommended approach. I found few articles where its mentioned firehose is the better option. Whereas I remember having collector (stats_z1/z2) job pointing to such monitoring server works well. So what steps I need to follow if I have to integrate such application monitoring tool using firehose? Do I need to write Nozzles for my monitoring tool like CloudCredo did for Graphite? https://github.com/CloudCredo/graphite-nozzle http://www.cloudcredo.com/how-to-integrate-graphite-with-cloud-foundry/ So if I have to integrate with Nagios, Wily, Splunk etc I would need nozzle for all of them? If not what changes I need to do in my configuration and in buildpack(not sure)? I also found NOAA client https://github.com/cloudfoundry/noaa which I think consumes all logs from doppler and show them on console? How this Noaa client is using firehose, is not very clear in the document? |
|
|
