REST API endpoint for accessing application logs


Ponraj E
 

Hi,

I want to get the application's dumped logs from the loggregator and not the tailing logs.
CLI provides me a command to do it: cf logs APP_NAME --recent displays all the lines in the Loggregator buffer.

But how do I do it via REST API endpoint? I had set the CF_TRACE=true to see the REST calls thats been fired to get the application log, but I see only the GET call to get the application details, but after that it just dumps the log.

Thanks for the help.

Regards,
Ponraj


Rohit Kumar
 

The API endpoint to get recent logs is present on the loggregator
trafficontroller. You can get the URL for your traffic controller by
running:

cf curl /v2/info | jq .doppler_logging_endpoint

Note that, the URL which you get back will have a "wss" spec, but you will
need to use "https" when you issue a recentlogs request.

To get the recent logs for your application, you should issue a GET request
to https://<trafficontroller URL>/apps/<appid>/recentlogs . You will also
need to provide your CF oauth token as part of the "Authorization" header
for this request. For example:

curl -k -H "Authorization: $(cf oauth-token | grep bearer)"
https://doppler.bosh-lite.com:443/apps/$(cf app appName --guid)/recentlogs

The response body will contain the log messages in the dropsonde-protocol
<https://github.com/cloudfoundry/dropsonde-protocol>format, so you will
need to parse them. If you are using Go to do this an easier way would be
to use the NOAA library to get recentlogs
<https://github.com/cloudfoundry/noaa/blob/master/sample/main.go#L17-L29>.

Rohit

On Tue, Oct 13, 2015 at 6:11 AM, Ponraj E <ponraj.e(a)gmail.com> wrote:

Hi,

I want to get the application's dumped logs from the loggregator and not
the tailing logs.
CLI provides me a command to do it: cf logs APP_NAME --recent displays
all the lines in the Loggregator buffer.

But how do I do it via REST API endpoint? I had set the CF_TRACE=true to
see the REST calls thats been fired to get the application log, but I see
only the GET call to get the application details, but after that it just
dumps the log.

Thanks for the help.

Regards,
Ponraj


Ponraj E
 

Hi Rohit,

Thanks for the reply. I tried this :

curl -k -H "Authorization: $(cf oauth-token | grep bearer)" https://doppler.bosh-lite.com:443/apps/$(cf app appName --guid)/recentlogs

with my logging_endpoint that i got from cf curl /v2/info (for ex: "https://xxxx:443") .But it says host could not be resolved.

Ponraj


Rohit Kumar
 

You should use the value coming from "doppler_logging_endpoint" not the
"logging_endpoint". What version of cf-release are you using? Alternatively
if you don't have a "doppler_logging_endpoint" in the response from
/v2/info , then use the URL from "logging_endpoint" but replace
"loggregator" with "doppler".

Rohit

On Tue, Oct 13, 2015 at 7:47 AM, Ponraj E <ponraj.e(a)gmail.com> wrote:

Hi Rohit,

Thanks for the reply. I tried this :

curl -k -H "Authorization: $(cf oauth-token | grep bearer)"
https://doppler.bosh-lite.com:443/apps/$(cf app appName --guid)/recentlogs

with my logging_endpoint that i got from cf curl /v2/info (for ex: "
https://xxxx:443") .But it says host could not be resolved.

Ponraj


Ponraj E
 

Hi Rohit,

I am using the cf-release version 211, CC API version 2.28.0 , CLI version-6.12.2.

Also I replaced "loggregator" with "doppler" . But still its not able to resolve the host. For instance, my host is : https://doppler.xx.xxx.xxxxxxxx.xxx .

Could it be proxy issue? Any other way to reach the doppler endpoint.


Regards,
Ponraj


Ponraj E
 

Hi Rohit,

Now I am able to resolve the host, but the command doesnt seem to work. It says,

curl: option --guid)/recentlogs: is unknown

--
Ponraj


Ponraj E
 

Hi,

Sorry for the spam.
Latest update : I am able to execute the command. But I dont have authorization.

I get the message.
You are not authorized. Error: Invalid authorization

--
Ponraj


Marco Nicosia
 

You don't need to do the $(cf oauth-token | grep bearer) all within the
curl command.

Perhaps the auth issue will be more apparent if you break it down into each
individual step.

What happens if you run `cf oath-token` separately? Does the output look
OK? Do you see a line that contains bearer in it? What happens when you
manually put that line into the curl command?

-- Marco N.

On Thursday, October 15, 2015, Ponraj E <ponraj.e(a)gmail.com> wrote:

Hi,

Sorry for the spam.
Latest update : I am able to execute the command. But I dont have
authorization.

I get the message.
You are not authorized. Error: Invalid authorization

--
Ponraj
--
--
Marco Nicosia
Product Manager
Pivotal Software, Inc.
mnicosia(a)pivotal.io
c: 650-796-2948


Warren Fernandes
 

Hi Ponraj,

It could be that your token may have expired. Also, silly question but you may want to check that you have access to the app in question. A good way to verify this is by running `cf logs <your_app_name> --recent`

Hope this helps.
Warren


Ponraj E
 

Hi ,

I executed the command with manually putting the outh token -pasting the o/p here:

C:\WINDOWS\system32>curl -k -H "Authorization: bearer xxxx" "https://doppler.xx.xxx.xxxxxxxx:443/apps/e0dc9133-f800-416d-9e1f-ffeb8d02e4dd/recentlogs"
--1f51e48e4a9bf243ed1e2ab55c090c4d2deb23a7ae255c5c1d8ad178acb9


◄dea_logging_agent►♣0│┌üªπ╩áç¶j2z☺1é☺♀10.78.150.44B╜☺
Ç☺Mon, 19 Oct 2015 06:23:56 GMT express deprecated res.send(status, body): Use res.status(status).se
nd(body) instead at app.js:7:7►☻↑╜╜üªπ╩áç¶"$e0dc9133-f800-416d-9e1f-ffeb8d02e4dd*♥App2☺1
--1f51e48e4a9bf243ed1e2ab55c090c4d2deb23a7ae255c5c1d8ad178acb9


router__0►♣0₧π╪á½╩áç¶j
10.78.149.103B╔♥router_z1z☺0é☺
app_url - [19/10/2015:06:23:41 +0000] "GET / HTTP/1.1" 2
00 0 40 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.
2454.101 Safari/537.36" 10.78.148.7:59978 x_forwarded_for:"ip_addr" vcap_request_id:1cbd4f2d-
84c6-4ef9-6f44-e0c84a4d5e49 response_time:0.624678694 app_id:e0dc9133-f800-416d-9e1f-ffeb8d02e4dd
►☺↑└╥╪á½╩áç¶"$e0dc9133-f800-416d-9e1f-ffeb8d02e4dd*♥RTR2☺0
--1f51e48e4a9bf243ed1e2ab55c090c4d2deb23a7ae255c5c1d8ad178acb9


router__0►♣0Ç╩ΩεΓ╩áç¶j
10.78.149.103B╔♥router_z1z☺0é☺
app_url - [19/10/2015:06:23:56 +0000] "GET / HTTP/1.1" 2
00 0 40 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.
2454.101 Safari/537.36" 10.78.148.7:59978 x_forwarded_for:"ip_addr" vcap_request_id:758c85b6-
dfef-41dc-5def-dc76ccf01498 response_time:0.999714512 app_id:e0dc9133-f800-416d-9e1f-ffeb8d02e4dd
►☺↑█╜ΩεΓ╩áç¶"$e0dc9133-f800-416d-9e1f-ffeb8d02e4dd*♥RTR2☺0
--1f51e48e4a9bf243ed1e2ab55c090c4d2deb23a7ae255c5c1d8ad178acb9--


Questions:

1. What is this guid that the command prints multiple times?

--1f51e48e4a9bf243ed1e2ab55c090c4d2deb23a7ae255c5c1d8ad178acb9--

2. Some of the output is garbled and some are not.

3. As rohit said in the previous comment, I tried installing the noaa client library and executed the sample app
to get the application logs : https://github.com/cloudfoundry/noaa

go build -o bin/sample sample/main.go

My DOPPLER_ADDR env is "https://doppler.xx.xxx.xxxxxxxx.xxx:443"

So when i run the sample.exe, I get the o/p as
===== Error getting recent messages: parse "https://doppler.xx.xxx.xxxxxxxx.xxx:443": invalid URI for request

Am not sure, how is this URI invalid?


---
Ponraj


Warren Fernandes
 

Hi Ponraj,

I've replicated the behavior you are seeing.

I'll get back to you for question #1 and #2 after some investigation.

As for #3 set your DOPPLER_ADDR to wss://doppler.xx.xxx.xxxxxxxx.xxx:443. That is, use wss instead of https.

Thanks.


Ponraj E
 

Hi Warren,

Thanks. Reg #3:Even "wss://doppler.xx.xxx.xxxxxxxx.xxx:443" gives the same
"invalid URI for request".


--
Ponraj


Warren Fernandes
 

Hi Ponraj,

For #1
The guid that gets printed out is the boundary for the multi-part message. When we serve the recent logs we create a multipart writer here(https://github.com/cloudfoundry/loggregatorlib/blob/master/server/handlers/http_handler.go#L23) which creates a random boundary here (https://golang.org/src/mime/multipart/writer.go?s=470:505#L29). You can see this in the headers if you run the curl command with a -v flag.

For #2
I'm guessing since the content type is set to "multipart/x-protobuf", the data might not be getting decoded properly. However, if you use the noaa library we unmarshal the data correctly here (https://github.com/cloudfoundry/noaa/blob/master/consumer.go#L333)

For #3
The error you are getting is from the net/url package here (https://golang.org/src/net/url/url.go#L417) which is called when we do a ParseURIRequest here (https://github.com/cloudfoundry/noaa/blob/master/consumer.go#L212). I even tried various ways to get that error in the playground (http://play.golang.org/p/tOeuyTxnb_). The only way I could replicate your error easily was to include the quotes in the doppler address. I'm hoping this is not a Windows thing because my testing happens to be on Mac OS X.

Hope this helps.


Juan Antonio Breña Moral <bren at juanantonio.info...>
 


Ponraj E
 

Hi Warren,

Thanks. Have some questions regarding the loggregator.

1. I see the loggregator has all the logs drained by cf components + applications as well. Is the logs timebound? If yes, where do we configure that?

2. Are the logs space bound?- like loggregator can store this much MB of data. I guess its yes, because that's why we have third party log management services. So, If yes where do we specify or configure that?

3. The 'getRecentLogs' output dumped from cf-java-client truncates the data compared to the output of cf logs app_name --recent. Only some of the data is displayed. What could be the reason?


---
Ponraj


Ponraj E
 

Hi,

Short update reg the question #2 above: I came to know from here http://docs.cloudfoundry.org/loggregator/ops.html that the number/size of log messages drained to the doppler can be controlled by a bosh deployment
manifest configuration : doppler.message_drain_buffer_size

It is specified that the doppler.message_drain_buffer_size default value is 100.

Is it 100MB?


Johannes Tuchscherer
 

No, that is 100 as in 100 log lines. Given that a log line is transported
as a UDP package and the UDP max package size is 64k, with a buffer of 100
log lines you require 6.3MB of memory per app on your Doppler server.

On Wednesday, October 21, 2015, Ponraj E <ponraj.e(a)gmail.com> wrote:

Hi,

Short update reg the question #2 above: I came to know from here
http://docs.cloudfoundry.org/loggregator/ops.html that the number/size of
log messages drained to the doppler can be controlled by a bosh deployment
manifest configuration : doppler.message_drain_buffer_size

It is specified that the doppler.message_drain_buffer_size default value
is 100.

Is it 100MB?


Gianluca Volpe <gvolpe1968@...>
 

this is the maximum number of log lines the doppler can buffer while draining messages to remote syslog.

Gianluca

Il giorno 21/ott/2015, alle ore 09:23, Ponraj E <ponraj.e(a)gmail.com> ha scritto:

Hi,

Short update reg the question #2 above: I came to know from here http://docs.cloudfoundry.org/loggregator/ops.html that the number/size of log messages drained to the doppler can be controlled by a bosh deployment
manifest configuration : doppler.message_drain_buffer_size

It is specified that the doppler.message_drain_buffer_size default value is 100.

Is it 100MB?


Warren Fernandes
 

For #3,

The Loggregator team currently doesn't manage the cf-java-client library. There seems to be another post in the community here (https://lists.cloudfoundry.org/archives/list/cf-dev(a)lists.cloudfoundry.org/message/JWESJLSWV44KKLP7LTMSLB3L5N2I62BG/)
talking about a new v2 java client that will be more useful. If you see that there is some unexpected truncation, I'd suggest creating an issue on that repo so that they can fix it in v2.


Ponraj E
 

Hi Warren,

Thanks. Reg #1, Is the loggregator clears the logs after "certain period of interval"? If yes, How much is that and where do we configure that?


--
Ponraj