The cf-release templates allow for a "system_domain" (
https://github.com/cloudfoundry/cf-release/blob/master/templates/cf-jobs.yml#L616),
this allows for a separate namespace for non-app hostnames("app_domain").
Hope this helps

Thanks,


Jim

On Thu, Oct 8, 2015 at 2:49 AM Sylvain Gibier <sylvain(a)munichconsulting.de>
wrote:

Hi,

Bug ?

Pushing an application using a manifest.yml file with an host entry set
to 'api', you are able to override the default route api.<<mycf.domain>>
with the application - preventing further cf commands to work correctly, as
it redirects traffic to new application.
I would have expect an error indicating the api.<<mycf.domain>> is
already used to prevent anyone to override api, uaa and login endpoint with
a custom app. Or am I missing something?

Is it a way - to make application/host like api, uaa, login ... not been
able to be overrided by custom application binding?

Sylvain

Hi Mike,

As Erik mentioned in the last thread, multi-line logging is something
which the loggregator team would like to solve. But there are a few
questions to answer before we can come up with a clean solution. We want a
design which solves the problem while not breaking existing apps which do
not require this functionality. Before implementing a solution we would
also want to answer if we want to do it for both runtimes or just Diego,
since the way log lines are sent to Metron differ based on the runtime.

If we were to implement the solution which you described, where newlines
are escaped with a '\', I guess the expectation is that loggregator would
internally remove the escape character. This has performance implications
because now some part of loggregator will need to inspect the log message
and coalesce the message with the succeeding ones. We will need to do this
in a way which respects multi-tenancy. That means now we are storing
additional state related to log lines per app. We will also need to decide
how long loggregator needs to wait for the next lines in a multi-line log,
before deciding to send the line which it received. To me that's not a
simple change.

I am happy to continue this discussion and hear your thoughts on the
existing proposal or any other design alternatives.

Thanks,
Rohit

On Wed, Oct 7, 2015 at 10:45 AM, Mike Youngstrom <youngm(a)gmail.com> wrote:

Splunk recently released its new "HTTP Event Collector" that greatly
simplifies how data can be streamed directly into Splunk without going to
an intermediate log file. It would be great to utilize this to efficiently
stream Loggregator information into Splunk.

For the most part loggregator appears to be very compatible with this API
with the exception of multi-line log messages.

The problem is that using this API splunk takes every request as an
independent splunk event. This completely eliminates anything splunk did
in the past to attempt to detect multi-line log messages.

Wouldn't it be great if a single loggregator event could contain multiple
log lines then these events could be easily streamed directly into Splunk
using this new api multiple lines preserved and all?

The previous attempt to bring up this topic fizzled [0]. With a new LAMB
PM coming I thought I'd ask my previous questions again.

In the previous thread [0] Erik mentioned a lot of work that he thought
would lead to multi-line log messages. But, it seems to me that the main
issue is simply how can a client actually communicate an multi-line event
to an agent? I don't think this issue is about breaking apart and then
combining log event rather how can I just I as a client hint to loggregator
that it should include multiple lines included into a single event?

Could it be as simple as escaping new lines with a '\' to notify the
agent to not end that event?

This problem cannot be solved without some help from loggregator.

Mike

[0]
https://lists.cloudfoundry.org/archives/list/cf-dev%40lists.cloudfoundry.org/thread/O6NDVGV44IBMVKZQXWOFIYOIC6CDU27G/

Sebastien

So, I'm wondering if it still makes sense to use the resource provider's

token inside our *asynchronous* usage *processing* pipeline. Shouldn't we
require the individual processing steps to obtain their own tokens instead?

It seems natural that the Abacus should use its own token - or tokens to
authenticate and authorize steps in the pipeline. After the submission
Abacus 'takes custody' of the submitted data so it should be solely
responsible for authrizing its processing.

Piotr

Piotr Przybylski | IBM Bluemix | piotrp(a)us.ibm.com | 650-645-8213


[image: Inactive hide details for Jean-Sebastien Delfino ---10/07/2015
09:21:05 AM---Hi all, A few more thoughts on a different, but re]Jean-Sebastien
Delfino ---10/07/2015 09:21:05 AM---Hi all, A few more thoughts on a
different, but related subject.

From: Jean-Sebastien Delfino <jsdelfino(a)gmail.com>
To: "Discussions about Cloud Foundry projects and the system overall." <
cf-dev(a)lists.cloudfoundry.org>
Date: 10/07/2015 09:21 AM
Subject: [cf-dev] Re: [abacus] Usage processing authorization, was: Usage
submission authorization
------------------------------



Hi all,

A few more thoughts on a different, but related subject.

We're using the resource provider's token and scopes to authorize usage
*submission* to our usage collector service. I agree with that and have
confirmed it in my answer to Piotr's question below.

It also looks like we're using the resource provider's token as well to
flow usage data through our usage *processing* pipeline (after we've
authorized the usage submission, validated that usage and taken
responsibility for it). I'm wondering if we couldn't find a better
approach, as we will run into a number of issues with this:

- processing delays in the pipeline can cause the token to expire, at that
point the resource provider is out of the loop, can't do anything about it,
and it won't make much sense anyway to ask the resource provider for a new
token way after it has submitted its usage;

- when restarting an Abacus service and recovering after a processing
interruption, we don't have a valid resource provider token either;

- more generally, I find a bit odd to use the resource provider's token in
usage processing steps down the Abacus pipeline, as they're really just
processing usage passed to them by the previous Abacus processing step (or
could have just read that usage from one of our usage DBs, and again the
resource provider wouldn't be so relevant as it wouldn't even have written
that usage to that DB itself.)

So, I'm wondering if it still makes sense to use the resource provider's
token inside our *asynchronous* usage *processing* pipeline. Shouldn't we
require the individual processing steps to obtain their own tokens instead?

Thoughts?

-- Jean-Sebastien


On Tue, Oct 6, 2015 at 10:36 PM, Jean-Sebastien Delfino <
*jsdelfino(a)gmail.com* <jsdelfino(a)gmail.com>> wrote:

Hi Piotr,

> what kind of authorization is required to submit usage to Abacus ?
> Is the oauth token used for submission [1] required to have
particular scope, specific to resource or resource provider ?

A resource provider is expected to present an OAuth token with the
usage it submits for a (service or runtime) resource.

That OAuth token should include:
- a user id uniquely identifying that resource provider;
- an OAuth scope named like abacus.usage.<resource_id>.write.

The precise naming syntax for that scope may still evolve in the next
few days as we progress with the implementation of user story 101703426 [1].

> Is there a different scope required to submit runtimes usage (like
cf bridge) versus other services or its possible to use single scope for
all the submissions

I'd like to handle runtimes and services consistently as they're
basically just different types of resources, i.e. one scope per 'service'
resource, one scope per 'runtime' resource.

We're still working on the detailed design and implementation, but I'm
not sure we'd want to share scopes across (service and runtime) resource
providers as that'd allow a resource provider to submit usage for resources
owned by another...

@assk / @sasrin, anything I missed? Thoughts?

-- Jean-Sebastien


On Tue, Oct 6, 2015 at 6:29 PM, Piotr Przybylski <*piotrp(a)us.ibm.com*
<piotrp(a)us.ibm.com>> wrote:
Hi,
what kind of authorization is required to submit usage to Abacus ?
Is the oauth token used for submission [1] required to have particular
scope, specific to resource or resource provider ? Is there a different
scope required to submit runtimes usage (like cf bridge) versus other
services or its possible to use single scope for all the submissions ?


[1] - *https://www.pivotaltracker.com/story/show/101703426*
<https://www.pivotaltracker.com/story/show/101703426>

Piotr

Hey Matt,

I'm trying to think of other things that would affect only the endpoints
that interact with UAA and would be fixed after a CC restart. I'm
wondering if it's possible there are a large number of connections being
kept-alive, or stuck in a wait state or something. Could you take a look
at the netstat information on the CC and UAA next time this happens?

-Zach and Swetha

I filed a GitHub issue:
https://github.com/cloudfoundry/cf-registrar/issues/7

What is happening was that
- NATS receives route registration from CC and Traffic Controller
- UAA not sending route registration to NATS, so UAA's routes are missing
from gorouter
- UAA's cf registrar stuck after following (instead of sending route
registration messages):
Connected to NATS - varz registration
Announcing start up vcap.component.announce

bosh vms, bosh cck, and monit all saying that everything running.

Any suggestion on how to troubleshoot this? What are the possible reasons
cf-registrar not sending route.register messages?

I was under the impression that you need to be able to connect directly,
but I can't say that definitively. Perhaps someone else can confirm.

Dan


On Thu, Oct 8, 2015 at 11:43 AM, Remi Tassing <tassingremi(a)gmail.com
<javascript:_e(%7B%7D,'cvml','tassingremi(a)gmail.com');>> wrote:

Hi Daniel,
10.0.16.103 is the internal address of that particular VM within the VPC
(I've deployed CF in AWS). So I can't access it directly.
I thought that was the point of using "bosh ssh", i.e., connecting to the
ha_proxy (which has a public address) first and then to the VM.

I have a feeling I've completed missed the point

Remi

I was under the impression that you need to be able to connect directly,
but I can't say that definitively. Perhaps someone else can confirm.

Dan


On Thu, Oct 8, 2015 at 11:43 AM, Remi Tassing <tassingremi(a)gmail.com>
wrote:

Hi Daniel,
10.0.16.103 is the internal address of that particular VM within the VPC
(I've deployed CF in AWS). So I can't access it directly.
I thought that was the point of using "bosh ssh", i.e., connecting to the
ha_proxy (which has a public address) first and then to the VM.

I have a feeling I've completed missed the point

Remi

Hi Daniel,
10.0.16.103 is the internal address of that particular VM within the VPC
(I've deployed CF in AWS). So I can't access it directly.
I thought that was the point of using "bosh ssh", i.e., connecting to the
ha_proxy (which has a public address) first and then to the VM.

I have a feeling I've completed missed the point

Remi

Hi,
I was trying "bosh ssh" in the interactive mode and after choosing the VM
it hangs for a bit then times out. I was following this tutorial:
http://docs.pivotal.io/pivotalcf/customizing/trouble-advanced.html

Console snippet:
....
13. stats_z1/0
Choose an instance: 1
Acting as user 'admin' on deployment 'cf' on 'microbosh'
Enter password (use it to sudo on remote host): *
Target deployment is `cf'

Setting up ssh artifacts

Director task 43

Task 43 done
Starting interactive shell on job nats_z1/0
ssh: connect to host 10.0.16.103 port 22: Connection timed out
...

Has anyone encountered this problem? Is there other alternative?

Remi