Loggregator doesn't store any logs. The most it does is maintain a buffer as mentioned above which is defaulted to 100 lines of logs. If you wish to store logs you can forward them to third-party syslog drains and other consumers.

Hi Warren,

Thanks. Reg #1, Is the loggregator clears the logs after "certain period of interval"? If yes, How much is that and where do we configure that?


--
Ponraj

For #3,

The Loggregator team currently doesn't manage the cf-java-client library. There seems to be another post in the community here (https://lists.cloudfoundry.org/archives/list/cf-dev(a)lists.cloudfoundry.org/message/JWESJLSWV44KKLP7LTMSLB3L5N2I62BG/)
talking about a new v2 java client that will be more useful. If you see that there is some unexpected truncation, I'd suggest creating an issue on that repo so that they can fix it in v2.

this is the maximum number of log lines the doppler can buffer while draining messages to remote syslog.

Gianluca

No, that is 100 as in 100 log lines. Given that a log line is transported
as a UDP package and the UDP max package size is 64k, with a buffer of 100
log lines you require 6.3MB of memory per app on your Doppler server.

Hi,

Short update reg the question #2 above: I came to know from here http://docs.cloudfoundry.org/loggregator/ops.html that the number/size of log messages drained to the doppler can be controlled by a bosh deployment
manifest configuration : doppler.message_drain_buffer_size

It is specified that the doppler.message_drain_buffer_size default value is 100.

Is it 100MB?

Hi Warren,

Thanks. Have some questions regarding the loggregator.

1. I see the loggregator has all the logs drained by cf components + applications as well. Is the logs timebound? If yes, where do we configure that?

2. Are the logs space bound?- like loggregator can store this much MB of data. I guess its yes, because that's why we have third party log management services. So, If yes where do we specify or configure that?

3. The 'getRecentLogs' output dumped from cf-java-client truncates the data compared to the output of cf logs app_name --recent. Only some of the data is displayed. What could be the reason?


---
Ponraj

You have an example here:

https://github.com/prosociallearnEU/cf-nodejs-client/blob/master/lib/model/Logs.js
https://github.com/prosociallearnEU/cf-nodejs-client/blob/master/test/lib/model/LogTests.js

it is possible to not use websockets replacing some parts in the URL:
https://loggregator.YOUR_IP.xip.io + '/recent?app=' + app_guid

Juan Antonio

Hi Ponraj,

For #1
The guid that gets printed out is the boundary for the multi-part message. When we serve the recent logs we create a multipart writer here(https://github.com/cloudfoundry/loggregatorlib/blob/master/server/handlers/http_handler.go#L23) which creates a random boundary here (https://golang.org/src/mime/multipart/writer.go?s=470:505#L29). You can see this in the headers if you run the curl command with a -v flag.

For #2
I'm guessing since the content type is set to "multipart/x-protobuf", the data might not be getting decoded properly. However, if you use the noaa library we unmarshal the data correctly here (https://github.com/cloudfoundry/noaa/blob/master/consumer.go#L333)

For #3
The error you are getting is from the net/url package here (https://golang.org/src/net/url/url.go#L417) which is called when we do a ParseURIRequest here (https://github.com/cloudfoundry/noaa/blob/master/consumer.go#L212). I even tried various ways to get that error in the playground (http://play.golang.org/p/tOeuyTxnb_). The only way I could replicate your error easily was to include the quotes in the doppler address. I'm hoping this is not a Windows thing because my testing happens to be on Mac OS X.

Hope this helps.

Hi Warren,

Thanks. Reg #3:Even "wss://doppler.xx.xxx.xxxxxxxx.xxx:443" gives the same
"invalid URI for request".


--
Ponraj

Hi Ponraj,

I've replicated the behavior you are seeing.

I'll get back to you for question #1 and #2 after some investigation.

As for #3 set your DOPPLER_ADDR to wss://doppler.xx.xxx.xxxxxxxx.xxx:443. That is, use wss instead of https.

Thanks.

Hi ,

I executed the command with manually putting the outh token -pasting the o/p here:

C:\WINDOWS\system32>curl -k -H "Authorization: bearer xxxx" "https://doppler.xx.xxx.xxxxxxxx:443/apps/e0dc9133-f800-416d-9e1f-ffeb8d02e4dd/recentlogs"
--1f51e48e4a9bf243ed1e2ab55c090c4d2deb23a7ae255c5c1d8ad178acb9


◄dea_logging_agent►♣0│┌üªπ╩áç¶j2z☺1é☺♀10.78.150.44B╜☺
Ç☺Mon, 19 Oct 2015 06:23:56 GMT express deprecated res.send(status, body): Use res.status(status).se
nd(body) instead at app.js:7:7►☻↑╜╜üªπ╩áç¶"$e0dc9133-f800-416d-9e1f-ffeb8d02e4dd*♥App2☺1
--1f51e48e4a9bf243ed1e2ab55c090c4d2deb23a7ae255c5c1d8ad178acb9


router__0►♣0₧π╪á½╩áç¶j
10.78.149.103B╔♥router_z1z☺0é☺
app_url - [19/10/2015:06:23:41 +0000] "GET / HTTP/1.1" 2
00 0 40 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.
2454.101 Safari/537.36" 10.78.148.7:59978 x_forwarded_for:"ip_addr" vcap_request_id:1cbd4f2d-
84c6-4ef9-6f44-e0c84a4d5e49 response_time:0.624678694 app_id:e0dc9133-f800-416d-9e1f-ffeb8d02e4dd
►☺↑└╥╪á½╩áç¶"$e0dc9133-f800-416d-9e1f-ffeb8d02e4dd*♥RTR2☺0
--1f51e48e4a9bf243ed1e2ab55c090c4d2deb23a7ae255c5c1d8ad178acb9


router__0►♣0Ç╩ΩεΓ╩áç¶j
10.78.149.103B╔♥router_z1z☺0é☺
app_url - [19/10/2015:06:23:56 +0000] "GET / HTTP/1.1" 2
00 0 40 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.
2454.101 Safari/537.36" 10.78.148.7:59978 x_forwarded_for:"ip_addr" vcap_request_id:758c85b6-
dfef-41dc-5def-dc76ccf01498 response_time:0.999714512 app_id:e0dc9133-f800-416d-9e1f-ffeb8d02e4dd
►☺↑█╜ΩεΓ╩áç¶"$e0dc9133-f800-416d-9e1f-ffeb8d02e4dd*♥RTR2☺0
--1f51e48e4a9bf243ed1e2ab55c090c4d2deb23a7ae255c5c1d8ad178acb9--


Questions:

1. What is this guid that the command prints multiple times?

--1f51e48e4a9bf243ed1e2ab55c090c4d2deb23a7ae255c5c1d8ad178acb9--

2. Some of the output is garbled and some are not.

3. As rohit said in the previous comment, I tried installing the noaa client library and executed the sample app
to get the application logs : https://github.com/cloudfoundry/noaa

go build -o bin/sample sample/main.go

My DOPPLER_ADDR env is "https://doppler.xx.xxx.xxxxxxxx.xxx:443"

So when i run the sample.exe, I get the o/p as
===== Error getting recent messages: parse "https://doppler.xx.xxx.xxxxxxxx.xxx:443": invalid URI for request

Am not sure, how is this URI invalid?


---
Ponraj

Hi Ponraj,

It could be that your token may have expired. Also, silly question but you may want to check that you have access to the app in question. A good way to verify this is by running `cf logs <your_app_name> --recent`

Hope this helps.
Warren

You don't need to do the $(cf oauth-token | grep bearer) all within the
curl command.

Perhaps the auth issue will be more apparent if you break it down into each
individual step.

What happens if you run `cf oath-token` separately? Does the output look
OK? Do you see a line that contains bearer in it? What happens when you
manually put that line into the curl command?

-- Marco N.

--
--
Marco Nicosia
Product Manager
Pivotal Software, Inc.
mnicosia(a)pivotal.io
c: 650-796-2948

Hi,

Sorry for the spam.
Latest update : I am able to execute the command. But I dont have authorization.

I get the message.
You are not authorized. Error: Invalid authorization

--
Ponraj

Hi Rohit,

Now I am able to resolve the host, but the command doesnt seem to work. It says,

curl: option --guid)/recentlogs: is unknown

--
Ponraj

Hi Rohit,

I am using the cf-release version 211, CC API version 2.28.0 , CLI version-6.12.2.

Also I replaced "loggregator" with "doppler" . But still its not able to resolve the host. For instance, my host is : https://doppler.xx.xxx.xxxxxxxx.xxx .

Could it be proxy issue? Any other way to reach the doppler endpoint.


Regards,
Ponraj

You should use the value coming from "doppler_logging_endpoint" not the
"logging_endpoint". What version of cf-release are you using? Alternatively
if you don't have a "doppler_logging_endpoint" in the response from
/v2/info , then use the URL from "logging_endpoint" but replace
"loggregator" with "doppler".

Rohit

Hi Rohit,

Thanks for the reply. I tried this :

curl -k -H "Authorization: $(cf oauth-token | grep bearer)" https://doppler.bosh-lite.com:443/apps/$(cf app appName --guid)/recentlogs

with my logging_endpoint that i got from cf curl /v2/info (for ex: "https://xxxx:443") .But it says host could not be resolved.

Ponraj

The API endpoint to get recent logs is present on the loggregator
trafficontroller. You can get the URL for your traffic controller by
running:

cf curl /v2/info | jq .doppler_logging_endpoint

Note that, the URL which you get back will have a "wss" spec, but you will
need to use "https" when you issue a recentlogs request.

To get the recent logs for your application, you should issue a GET request
to https://<trafficontroller URL>/apps/<appid>/recentlogs . You will also
need to provide your CF oauth token as part of the "Authorization" header
for this request. For example:

curl -k -H "Authorization: $(cf oauth-token | grep bearer)"
https://doppler.bosh-lite.com:443/apps/$(cf app appName --guid)/recentlogs

The response body will contain the log messages in the dropsonde-protocol
<https://github.com/cloudfoundry/dropsonde-protocol>format, so you will
need to parse them. If you are using Go to do this an easier way would be
to use the NOAA library to get recentlogs
<https://github.com/cloudfoundry/noaa/blob/master/sample/main.go#L17-L29>.

Rohit