Date   

Firehose vs Pivotal Ops metrics

Qing Gong
 

I set up the open source CF and got the metrics streamed out of the firehose. I also read document about Pivotal Ops Metrics in here.

http://docs.pivotal.io/pivotalcf/customizing/use-metrics.html

What are the relationship between the two sets of metrics? The Pivotal one uses JMX and has a completely different set of metrics. Why do we have metrics in two places?

Also, are there any documents about what metrics are streamed from the firehose? I could not find any document on this other than seeing the actual data streamed from the firehose.

Thanks!


Re: CF UAA Refresh Token

Filip Hanik
 

cf client is fine. put empty string as a password

Filip


On Mon, Aug 24, 2015 at 12:54 PM, aaron_huber <aaron.m.huber(a)intel.com>
wrote:

So we're just using the cf client which doesn't have a secret defined, is
that why we need to use the admin client?



--
View this message in context:
http://cf-dev.70369.x6.nabble.com/cf-dev-CF-UAA-Refresh-Token-tp1338p1344.html
Sent from the CF Dev mailing list archive at Nabble.com.


Re: CF UAA Refresh Token

Aaron Huber
 

So we're just using the cf client which doesn't have a secret defined, is
that why we need to use the admin client?



--
View this message in context: http://cf-dev.70369.x6.nabble.com/cf-dev-CF-UAA-Refresh-Token-tp1338p1344.html
Sent from the CF Dev mailing list archive at Nabble.com.


Re: CF UAA Refresh Token

Filip Hanik
 

Your authorization header should be Basic (meaning, you need the client id
and secret to retrieve the refresh token)

On Mon, Aug 24, 2015 at 11:48 AM, Filip Hanik <fhanik(a)pivotal.io> wrote:

Simple test that does what you need

https://github.com/cloudfoundry/uaa/blob/develop/uaa/src/test/java/org/cloudfoundry/identity/uaa/integration/RefreshTokenSupportIntegrationTests.java#L147-L150

we can help investigate more if you post your token here



On Mon, Aug 24, 2015 at 11:37 AM, aaron_huber <aaron.m.huber(a)intel.com>
wrote:

Not sure I understand that. When you get a token you also automatically
get
a refresh token - are you saying the refresh token given isn't valid and
we
have to generate a new refresh token as an admin user? To clarify, all
we're trying to do is renew the token when it expires so the user doesn't
have to log in again.

Aaron



--
View this message in context:
http://cf-dev.70369.x6.nabble.com/cf-dev-CF-UAA-Refresh-Token-tp1338p1340.html
Sent from the CF Dev mailing list archive at Nabble.com.


Re: CF UAA Refresh Token

Filip Hanik
 

Simple test that does what you need
https://github.com/cloudfoundry/uaa/blob/develop/uaa/src/test/java/org/cloudfoundry/identity/uaa/integration/RefreshTokenSupportIntegrationTests.java#L147-L150

we can help investigate more if you post your token here



On Mon, Aug 24, 2015 at 11:37 AM, aaron_huber <aaron.m.huber(a)intel.com>
wrote:

Not sure I understand that. When you get a token you also automatically
get
a refresh token - are you saying the refresh token given isn't valid and we
have to generate a new refresh token as an admin user? To clarify, all
we're trying to do is renew the token when it expires so the user doesn't
have to log in again.

Aaron



--
View this message in context:
http://cf-dev.70369.x6.nabble.com/cf-dev-CF-UAA-Refresh-Token-tp1338p1340.html
Sent from the CF Dev mailing list archive at Nabble.com.


Re: CF integration with logger and monitoring tools

Rohit Kumar
 

Correct, the datadog-firehose-nozzle has its own custom client which makes
REST calls to the datadog API. It doesn't use the datadog statsd agent. A
similar approach should work for you.

Thanks,
Rohit

On Mon, Aug 24, 2015 at 1:38 AM, Swatz bosh <swatzron(a)gmail.com> wrote:

Incase of datadog nozzle
https://github.com/cloudfoundry-incubator/datadog-firehose-nozzle

I am assuming that - datadogclient is a custom client which is a kind of
aggregator which is fetching metrics from doppler. So this client is not
using DataDogStasD client and its a custom client. I found that DataDog
also has agent(install on every machine) which contains this stasd client,
which datadog nozzle is not using, I think?
So I can have custom aggregator for willy similarly, is it correct?


Re: CF UAA Refresh Token

Aaron Huber
 

Not sure I understand that. When you get a token you also automatically get
a refresh token - are you saying the refresh token given isn't valid and we
have to generate a new refresh token as an admin user? To clarify, all
we're trying to do is renew the token when it expires so the user doesn't
have to log in again.

Aaron



--
View this message in context: http://cf-dev.70369.x6.nabble.com/cf-dev-CF-UAA-Refresh-Token-tp1338p1340.html
Sent from the CF Dev mailing list archive at Nabble.com.


Re: CF UAA Refresh Token

Paul Bakare
 

This is an issue with authorization.

You have to use a user with the correct scope. Specifically, use a client
with a scope/authority of 'uaa.admin'

The easiest way is to use the 'app' client to generate a refresh token for
a user.

On Mon, Aug 24, 2015 at 7:16 PM, Keagan Mendoza <keagan.mendoza(a)intel.com>
wrote:

Hi,

I am searching for a way to get a refresh token and I haven't been able to
do so. I did not find any info on the way to refresh a token. I was looking
for some docs externally and found some examples and drafted this:

Using Node.js

refreshUserToken: function (req, res) {
var request = require('request');
var pems = require('intel-cacerts').list;
var options = {
uri: req.body.url,
ca: pems,
headers:{
'Authorization' : 'bearer ' + req.query.token
},
form:{
refresh_token: req.body.refresh_token,
client_id: 'cf',
grant_type: 'refresh_token'
}
};


This is the error I get:

{"error":"unauthorized","error_description":"An Authentication object was
not found in the SecurityContext"}


Thanks,

Keagan Mendoza


CF UAA Refresh Token

Keagan Mendoza
 

Hi,

I am searching for a way to get a refresh token and I haven't been able to do so. I did not find any info on the way to refresh a token. I was looking for some docs externally and found some examples and drafted this:

Using Node.js

refreshUserToken: function (req, res) {
var request = require('request');
var pems = require('intel-cacerts').list;
var options = {
uri: req.body.url,
ca: pems,
headers:{
'Authorization' : 'bearer ' + req.query.token
},
form:{
refresh_token: req.body.refresh_token,
client_id: 'cf',
grant_type: 'refresh_token'
}
};


This is the error I get:

{"error":"unauthorized","error_description":"An Authentication object was not found in the SecurityContext"}


Thanks,

Keagan Mendoza


Re: Update on Mailman 3 launch

Marco Nicosia
 

Hi Eric (a different Marco here),

I noticed that delivery status for cf-bosh had been set to 'disabled' last
week, and re-enabled it. It's the only of the three that I have set to
'Mime Digests' delivery mode.

I don't remember which day last week, but I haven't gotten anything since,
and I can see that there was activity on the list as recently as Saturday.


--
Marco Nicosia
Product Manager
Pivotal Software, Inc.
mnicosia(a)pivotal.io



On Mon, Aug 24, 2015 at 8:46 AM, Eric Searcy <eric(a)linuxfoundation.org>
wrote:

Hi Marco,

Just to confirm, did you leave it enabled in "mime digest" mode for longer
than a day so that there was list traffic to bundle and digest for you? I
don't see any errors in the error log related to MIME digest sends, but see
about reproducing this today and submit a bug.

The preference lookup appears to give precedence to the settings on the
subscription, then on the address, then on the user ("global"), and finally
on the system default—it stops at the first defined value it sees. I'll
file a bug to have some better clarification in the UI.

Eric


Re: Fail to stage application when scale the DEA

Layne Peng
 

i found all new apps cannot be created now. caused by this issue. Any clue for it?


Re: Update on Mailman 3 launch

Eric Searcy <eric@...>
 

errors in the error log related to MIME digest sends, but see about reproducing this today
and submit a bug.
*but I will see about reproducing this today

I'll file a bug to have some better clarification in the UI.
Requested documentation in UI as a followup to https://gitlab.com/mailman/postorius/issues/30 as well as posting another bug about unset values and precedence (https://gitlab.com/mailman/postorius/issues/54).

Thanks for your feedback, Marco!

Eric


Re: max length with Dropped log message: message too long (>64K ...)

Erik Jasiak
 

Hi James / cf-dev

There are tests, but the corner case fell through the cracks -
bug[1] and additional test request[2] added. For everyone else, most of
the discussion appeared to happen in Slack. Summary is that we're
investigating a corner case where a message passes our test for
acceptable size, but then exceeds the 64K limit once it gets wrapped in
an envelope. We then send off the message but ignore any response from
sending[3].

Special thanks to Johannes Tuchscherer for first-pass analysis of this.

Thanks,
Erik


[1] https://www.pivotaltracker.com/story/show/101894886
[2] https://www.pivotaltracker.com/story/show/101897146
[3]
https://github.com/cloudfoundry/dropsonde/blob/master/log_sender/log_sender.go#L131


James Bayer wrote:

erik,

is there a set of tests for log message length?

---------- Forwarded message ----------
From: *Koper, Dies* <diesk(a)fast.au.fujitsu.com
<mailto:diesk(a)fast.au.fujitsu.com>>
Date: Thu, Aug 20, 2015 at 1:17 PM
Subject: [cf-dev] max length with Dropped log message: message too
long (>64K ...)
To: "cf-dev(a)lists.cloudfoundry.org
<mailto:cf-dev(a)lists.cloudfoundry.org>" <cf-dev(a)lists.cloudfoundry.org
<mailto:cf-dev(a)lists.cloudfoundry.org>>


Hi,

When my app outputs more than 64K of text, I get a message “Dropped
/log/message: message too long (>/64K/without a newline).

When my app outputs much less than 64K of text, I get the app’s output.

When my app outputs just under 64K of text, nothing is output: neither
the error message, nor the app’s output.

What is exactly the limit under which output is still guaranteed, and
can we update the error message accordingly?

I tried to read the source code. Not sure if the limit is because
messages are sent over UDP (UDP max data length is 65,507), or due to
other data that is included in the transmission of messages
(timestamp, app id, etc.).

Cheers,

Dies Koper




--
Thank you,

James Bayer


Re: Update on Mailman 3 launch

Eric Searcy <eric@...>
 

Hi Marco,

Just to confirm, did you leave it enabled in "mime digest" mode for longer than a day so that there was list traffic to bundle and digest for you? I don't see any errors in the error log related to MIME digest sends, but see about reproducing this today and submit a bug.

The preference lookup appears to give precedence to the settings on the subscription, then on the address, then on the user ("global"), and finally on the system default—it stops at the first defined value it sees. I'll file a bug to have some better clarification in the UI.

Eric


Re: Update on Mailman 3 launch

Chip Childers <cchilders@...>
 

+Eric

Chip Childers | VP Technology | Cloud Foundry Foundation

On Mon, Aug 24, 2015 at 6:57 AM, Marco Voelz <marco.voelz(a)sap.com> wrote:

Dear Eric,

thanks for the update on the current state of mailman3. The digest bug you
linked seems to be related to 'summary digests' only. However, I also don't
get any mails delivered, if I choose 'mime digests' as a delivery mode.

Also, I'm not quite sure about the precedence of settings: Which settings
are taken if I have different ones in global settings, address settings,
and subscription-based settings?

Thanks and warm regards
Marco


Re: Fail to stage application when scale the DEA

James Bayer
 

layne,

does this 401 issue only affect the new DEAs (previous ones work fine) or
does it affect all DEAs (previously existing and the new scaled out ones)?

On Mon, Aug 24, 2015 at 6:42 AM, Layne Peng <layne.peng(a)emc.com> wrote:

Hi, my cf-release is v200. And I met an issue when scale out the DEA
number with BOSH. When I ssh to my DEA_NEXT node, the log shows:

{"timestamp":1440419081.440199,"message":"em-http-request.errored","log_level":"error","source":"Staging","data":{"error":"Error
downloading: Response status: unknown, Error: ","data":{"droplet_uri":"
http://staging_upload_user:PASSWORD(a)20.0.0.5:9022/v2/buildpacks/608407a4-6c63-4bf2-a64f-02ed8d0db0ce/download"}},"thread_id":17109100,"fiber_id":26594680,"process_id":30334,"file":"/var/vcap/packages/dea_next/lib/dea/utils/download.rb","lineno":43,"method":"block
in download!"}

Then I tried to download it manually, get the following error:

--2015-08-24 13:20:09-- http://staging_upload_user:*password*@
20.0.0.5:9022/v2/buildpacks/608407a4-6c63-4bf2-a64f-02ed8d0db0ce/download
Connecting to 20.0.0.5:9022... connected.
HTTP request sent, awaiting response... 401 Unauthorized
Unknown authentication scheme.

Username/Password Authentication Failed.

And I have ssh to my cloud controller to check
/var/vcap/jobs/cloud_controller_ng/config/cloud_controller_ng.yml, the
username and password seems to be correct:

# App staging parameters
staging:
timeout_in_seconds: 900
minimum_staging_memory_mb: 1024
minimum_staging_disk_mb: 4096
minimum_staging_file_descriptor_limit: 16384
auth:
user: staging_upload_user
password: "PASSWORD"

Any clue for this issue?
--
Thank you,

James Bayer


Fail to stage application when scale the DEA

Layne Peng
 

Hi, my cf-release is v200. And I met an issue when scale out the DEA number with BOSH. When I ssh to my DEA_NEXT node, the log shows:

{"timestamp":1440419081.440199,"message":"em-http-request.errored","log_level":"error","source":"Staging","data":{"error":"Error downloading: Response status: unknown, Error: ","data":{"droplet_uri":"http://staging_upload_user:PASSWORD(a)20.0.0.5:9022/v2/buildpacks/608407a4-6c63-4bf2-a64f-02ed8d0db0ce/download"}},"thread_id":17109100,"fiber_id":26594680,"process_id":30334,"file":"/var/vcap/packages/dea_next/lib/dea/utils/download.rb","lineno":43,"method":"block in download!"}

Then I tried to download it manually, get the following error:

--2015-08-24 13:20:09-- http://staging_upload_user:*password*@20.0.0.5:9022/v2/buildpacks/608407a4-6c63-4bf2-a64f-02ed8d0db0ce/download
Connecting to 20.0.0.5:9022... connected.
HTTP request sent, awaiting response... 401 Unauthorized
Unknown authentication scheme.

Username/Password Authentication Failed.

And I have ssh to my cloud controller to check /var/vcap/jobs/cloud_controller_ng/config/cloud_controller_ng.yml, the username and password seems to be correct:

# App staging parameters
staging:
timeout_in_seconds: 900
minimum_staging_memory_mb: 1024
minimum_staging_disk_mb: 4096
minimum_staging_file_descriptor_limit: 16384
auth:
user: staging_upload_user
password: "PASSWORD"

Any clue for this issue?


Re: Update on Mailman 3 launch

Marco Voelz
 

Dear Eric,

thanks for the update on the current state of mailman3. The digest bug you linked seems to be related to 'summary digests' only. However, I also don't get any mails delivered, if I choose 'mime digests' as a delivery mode.

Also, I'm not quite sure about the precedence of settings: Which settings are taken if I have different ones in global settings, address settings, and subscription-based settings?

Thanks and warm regards
Marco


Re: I'm getting different x_forwarded_for in my Gorouter access logs depending on what browser/cli-tool I use.

Simon Johansson <simon@...>
 

https://groups.google.com/forum/#!searchin/golang-nuts/Inconsistent$20X-Forwarded-For/golang-nuts/iUnZgzrOszI/ngxcLbSm1VYJ

Mail thread about the issue at golang-nuts.

This is not really a massive issue for us, the main use we have for this is to get the source IP, so whatewher comes after that is not really looked at in our env.

This issue is howewher quite confusing and its not obvious why there is inconsistency, I dont know how big of a deal that will be for other deployments. But the easy fix would be to copy setRequestXForwardedFor for proxy and just call it before we call p.accessLogger.Log in the defer in ServeHTTP.


Re: CF integration with logger and monitoring tools

Swatz bosh
 

Incase of datadog nozzle https://github.com/cloudfoundry-incubator/datadog-firehose-nozzle

I am assuming that - datadogclient is a custom client which is a kind of aggregator which is fetching metrics from doppler. So this client is not using DataDogStasD client and its a custom client. I found that DataDog also has agent(install on every machine) which contains this stasd client, which datadog nozzle is not using, I think?
So I can have custom aggregator for willy similarly, is it correct?

8061 - 8080 of 9426