|
Re: Request for Multibuildpack Use Cases
Mike Youngstrom <youngm@...>
John,
It sounds like the buildpack team is thinking the multi buildpack feature
would only work for buildpacks they provide not a custom
"dependency-resolution" buildpack. Or at least that is how I understood
the message from Danny Rosen earlier in the thread.
Mike
toggle quoted message
Show quoted text
On Tue, Apr 12, 2016 at 10:45 AM, John Feminella <jxf(a)pivotal.io> wrote: Multibuildpack is absolutely useful and I'm excited for this proposal.
I encounter a lot of use cases for this. The most common is that an
application wants to pull in private dependencies during a future
dependency-resolution step of a later buildpack, but the dependency
resolver needs to be primed in some specific way. If you wait until
buildpack time it's too late.
On Heroku, for example, this is accomplished by having something like the
netrc buildpack (
https://github.com/timshadel/heroku-buildpack-github-netrc), adding a
GITHUB_TOKEN environment variable, and then running your "real" buildpack.
The netrc BP runs first, allowing Bundler to see the private dependencies.
best,
~ jf
On Tue, Apr 12, 2016 at 12:36 PM Jack Cai <greensight(a)gmail.com> wrote:
It would be more useful if the multi-buildpack can reference an admin
buildpack in addition to a remote git-hosted buildpack. :-)
Jack
On Tue, Apr 12, 2016 at 6:38 AM, David Illsley <davidillsley(a)gmail.com>
wrote:
In the past we've used the multi-buildpack to be able to use ruby sass
to compile SCSS for non-ruby projects (node and Java). In that case we used
the multi-buildpack and a .buildpacks file which worked reasonably well
(and was very clear).
On Mon, Apr 11, 2016 at 1:15 AM, Danny Rosen <drosen(a)pivotal.io> wrote:
Hi there,
The CF Buildpacks team is considering taking on a line of work to
provide more formal support for multibuildpacks. Before we start, we would
be interested in learning if any community users have compelling use cases
they could share with us.
For more information on multibuildpacks, see Heroku's documentation [1]
[1] -
https://devcenter.heroku.com/articles/using-multiple-buildpacks-for-an-app
|
|
|
Re: [cf-bosh] Re: Re: [Metron Agent] failed to generate job templates with metron agent on top of OpenStack Dynamic network
toggle quoted message
Show quoted text
On Mon, Apr 11, 2016 at 11:24 PM, Amit Gupta <agupta(a)pivotal.io> wrote: This will not work with dynamic networks. Many jobs in cf-release rely on
data from BOSH to determine their IP so that configuration files can be
rendered up-front by the director rather than at runtime, requiring system
calls to determine IP. metron_agent is one such job, and it tends to be
colocated with each other job (it is what allows all system component logs
to be aggregated through the loggregator system), so this would require all
Cloud Foundry VMs to be on a manual network. You don't need to manually
pick the IPs, you just need to tell BOSH which IPs in the network not to
use and specify these in the "reserved" range.
Since so many different components depend on being able to determine their
IP via BOSH data, there's no quick workaround if you want to stick to using
dynamic networks, but we're aware of this current limitation.
Best,
Amit
On Mon, Apr 11, 2016 at 7:23 PM, Yitao Jiang <jiangyt.cn(a)gmail.com> wrote:
Is it a bug of CF or Bosh ?
On Fri, Apr 8, 2016 at 12:08 PM, Ben R <vagcom.ben(a)gmail.com> wrote:
I have the same issue. It has to do with every release since bosh 248.
However, dynamic networks with older bosh releases + cf-231/cf-231 work.
This must be a bug.
Ben R
On Thu, Apr 7, 2016 at 8:55 PM, Yitao Jiang <jiangyt.cn(a)gmail.com>
wrote:
Hi,guys
When deploy CF on top of OpenStack with dynamic network, the jobs
failed with metron-agent
Error filling in template 'syslog_forwarder.conf.erb' (line 44:
undefined method `strip' for nil:NilClass)
here's related logs
​D​
etecting deployment changes
----------------------------
Releases
cf
version type changed: String -> Fixnum
- 233
+ 233
Compilation
No changes
Update
± canaries:
- 1
+ 0
Resource pools
No changes
Disk pools
No changes
Networks
dynamic-net
+ name: dynamic-net
subnets
10.0.0.0/24
cloud_properties
+ net_id: 0700ae03-4b38-464e-b40d-0a9c8dd18ff0
+ security_groups: ["Test OS SG_20160128T070152Z"]
+ dns: ["114.114.114.114", "8.8.8.8"]
+ range: 10.0.0.0/24
+ name: Test OS Sub Internal Network_20160128T070152Z
+ type: dynamic
Jobs stats_z1
± networks:
- {"name"=>"cf1"}
+ {"name"=>"dynamic-net"}
Properties No changes
Meta No changes
Please review all changes carefully
Deploying ---------
Are you sure you want to deploy? (type 'yes' to continue): yes
Director task 57 Started preparing deployment > Preparing deployment. Done (00:00:03)
Error 100: Unable to render instance groups for deployment. Errors
are: - Unable to render jobs for instance group 'stats_z1'. Errors are:
- Unable to render templates for job 'metron_agent'. Errors are:
- Error filling in template 'syslog_forwarder.conf.erb' (line
44: undefined method `strip' for nil:NilClass)
Task 57 error
For a more detailed error report, run: bosh task 57 --debug as the ip manged by OpenStack, bosh cannot get the actual ip address of
each vm until vm alive, this lead to the generated job spec doesn't contain
ip address infos
so, must i have to configure network type to manual?​
snippets of deployment yml
1001 - name: dynamic-net
1002 subnets:
1003 - cloud_properties:
1004 net_id: 0700ae03-4b38-464e-b40d-0a9c8dd18ff0
1005 security_groups:
1006 - Test OS SG_20160128T070152Z
1007 dns:
1010 - 114.114.114.114
1011 - 8.8.8.8
1012 range: 10.0.0.0/24
1013 name: Test OS Sub Internal Network_20160128T070152Z
1014 type: dynamic
​Rendered job spec
{"deployment"=>"staging-01", "job"=
{"name"=>"stats_z1", "templates"=>[{"name"=>"collector",
"version"=>"6c210292f18d129e9a037fe7053836db2d494344",
"sha1"=>"38927f47b15c2daf6c8a2e7c760e73e5ff90
dfd4", "blobstore_id"=>"23531029-0ee1-4267-8863-b5f931afaecb"},
{"name"=>"metron_agent",
"version"=>"2b80a211127fc642fc8bb0d14d7eb30c37730db3", "sha1"=>"150f2
7445c2ef960951c1f26606525d41ec629b2",
"blobstore_id"=>"e87174dc-f3f7-4768-94cd-74f299813528"}],
"template"=>"collector", "version"=>"6c210292f18d129e9a037fe70
53836db2d494344", "sha1"=>"38927f47b15c2daf6c8a2e7c760e73e5ff90dfd4",
"blobstore_id"=>"23531029-0ee1-4267-8863-b5f931afaecb"}, "index"=>0,
"bootstrap"=>true,
"name"=>"stats_z1", "id"=>"99f349d0-fb5d-4de7-9912-3de5559d2f19",
"az"=>nil,
*"networks"=>{"dynamic-net"=>{"type"=>"dynamic",
"cloud_properties"=>{"net_id"=>"0700ae03-4b38-464e-b40d-0a9c8dd18ff0",
"security_groups"=>["Test OS SG_20160128T070152Z"]},
"dns"=>["114.114.114.114", "8.8.8.8", "10.0.0.13"], "default"=>["dns",
"gateway"],
"dns_record_name"=>"0.stats-z1.dynamic-net.staging-01.microbosh"}}*,
"properties"=>{"collector"=>{"aws"=>{
"access_key_id"=>nil, "secret_access_key"=>nil},
"datadog"=>{"api_key"=>nil, "application_key"=>nil},
"deployment_name"=>nil, "logging_level"=>"info", "interv
als"=>{"discover"=>60, "healthz"=>30, "local_metrics"=>30,
"nats_ping"=>30, "prune"=>300, "varz"=>30}, "use_aws_cloudwatch"=>false,
"use_datadog"=>false, "use
_tsdb"=>false, "opentsdb"=>{"address"=>nil, "port"=>nil},
"use_graphite"=>false, "graphite"=>{"address"=>nil, "port"=>nil},
"memory_threshold"=>800}, "nats"=>
{"machines"=>["10.0.0.127"], "password"=>"NATS_PASSWORD",
"port"=>4222, "user"=>"NATS_USER"},
"syslog_daemon_config"=>{"address"=>nil, "port"=>nil, "transport
"=>"tcp", "fallback_addresses"=>[], "custom_rule"=>"",
"max_message_size"=>"4k"},
"metron_agent"=>{"dropsonde_incoming_port"=>3457, "preferred_protocol"=>"udp
", "tls"=>{"client_cert"=>"", "client_key"=>""}, "debug"=>false,
"zone"=>"z1", "deployment"=>"ya-staging-01",
"tcp"=>{"batching_buffer_bytes"=>10240, "batchin
g_buffer_flush_interval_milliseconds"=>100},
"logrotate"=>{"freq_min"=>5, "rotate"=>7, "size"=>"50M"},
"buffer_size"=>10000, "enable_buffer"=>false}, "metron_
endpoint"=>{"shared_secret"=>"LOGGREGATOR_ENDPOINT_SHARED_SECRET"},
"loggregator"=>{"tls"=>{"ca_cert"=>""}, "dropsonde_incoming_port"=>3457,
"etcd"=>{"machine
s"=>["10.0.0.133"], "maxconcurrentrequests"=>10}}},
"dns_domain_name"=>"microbosh", "links"=>{},
"address"=>"99f349d0-fb5d-4de7-9912-3de5559d2f19.stats-z1.dyn
amic-net.ya-staging-01.microbosh", "persistent_disk"=>0,
"resource_pool"=>"small_z1"}​
--
Regards,
Yitao
--
Regards,
Yitao
--
Jim Campbell | Product Manager | Cloud Foundry | Pivotal.io | 303.618.0963
|
|
|
Re: Request for Multibuildpack Use Cases
Another common use case for me is that an application is polyglot in some
way. For example, you have a blog you'd like to deploy that uses a static
site generator written in (say) Go. You would have the Go buildpack run to
bootstrap and generate the resulting files in some output directory, and
then the staticfile buildpack runs and serves the files from the output
directory.
best,
~ jf
toggle quoted message
Show quoted text
On Tue, Apr 12, 2016 at 12:45 PM John Feminella <jxf(a)pivotal.io> wrote: Multibuildpack is absolutely useful and I'm excited for this proposal.
I encounter a lot of use cases for this. The most common is that an
application wants to pull in private dependencies during a future
dependency-resolution step of a later buildpack, but the dependency
resolver needs to be primed in some specific way. If you wait until
buildpack time it's too late.
On Heroku, for example, this is accomplished by having something like the
netrc buildpack (
https://github.com/timshadel/heroku-buildpack-github-netrc), adding a
GITHUB_TOKEN environment variable, and then running your "real" buildpack.
The netrc BP runs first, allowing Bundler to see the private dependencies.
best,
~ jf
On Tue, Apr 12, 2016 at 12:36 PM Jack Cai <greensight(a)gmail.com> wrote:
It would be more useful if the multi-buildpack can reference an admin
buildpack in addition to a remote git-hosted buildpack. :-)
Jack
On Tue, Apr 12, 2016 at 6:38 AM, David Illsley <davidillsley(a)gmail.com>
wrote:
In the past we've used the multi-buildpack to be able to use ruby sass
to compile SCSS for non-ruby projects (node and Java). In that case we used
the multi-buildpack and a .buildpacks file which worked reasonably well
(and was very clear).
On Mon, Apr 11, 2016 at 1:15 AM, Danny Rosen <drosen(a)pivotal.io> wrote:
Hi there,
The CF Buildpacks team is considering taking on a line of work to
provide more formal support for multibuildpacks. Before we start, we would
be interested in learning if any community users have compelling use cases
they could share with us.
For more information on multibuildpacks, see Heroku's documentation [1]
[1] -
https://devcenter.heroku.com/articles/using-multiple-buildpacks-for-an-app
|
|
|
Re: Doppler/Firehose - Multiline Log Entry
Mike Youngstrom <youngm@...>
Jim, If I submitted a CLI PR to change the cf logs command to substitute /u2028 with /n could the loggregator team get behind that? Mike On Tue, Apr 12, 2016 at 10:20 AM, Jim CF Campbell <jcampbell(a)pivotal.io> wrote: Mike,
When you get a bit more desperate ;-) here is a nozzle plug in
<https://github.com/jtuchscherer/nozzle-plugin> for the CLI. It's
attaches to the firehose to display everything, but would be easy to modify
to just look at a single app, and sub out the magic token for newlines.
Jim
On Tue, Apr 12, 2016 at 9:56 AM, Mike Youngstrom <youngm(a)gmail.com> wrote:
Hi David,
The problem for me is that I'm searching for a solution that can works
for development (though less of a priority cause you can switch config
between dev and cf) and for viewing logs via "cf logs" in addition to a log
aggregator. I had hoped that /u2028 would work for viewing logs via "cf
logs" but it doesn't in bash. I'd need to write a plugin or something for
cf logs and train all my users to use it. Certainly possible but I'm not
that desperate yet. :)
Mike
On Tue, Apr 12, 2016 at 5:58 AM, David Laing <david(a)davidlaing.com>
wrote:
FWIW, the technique is to have your logging solution (eg, logback,
log4j) log a token (eg, \u2028) other than \n to denote line breaks in
your stack traces; and then have your log aggregation software replace that
token with a \n again when processing the log messages.
If \u2028 doesn't work in your environment; use something else; eg
NEWLINE
On Mon, 11 Apr 2016 at 21:12 Mike Youngstrom <youngm(a)gmail.com> wrote:
Finally got around to testing this. Preliminary testing show that "\u2028"
doesn't function as a new line character in bash and causes eclipse console
to wig out. I don't think "\u2028" is a viable long term solution.
Hope you make progress on a metric format available to an app in a
container. I too would like a tracker link to such a feature if there is
one.
Thanks,
Mike
On Mon, Mar 14, 2016 at 2:28 PM, Mike Youngstrom <youngm(a)gmail.com>
wrote:
Hi Jim,
So, to be clear what we're basically doing is using unicode newline
character to fool loggregator (which is looking for \n) into thinking that
it isn't a new log event right? Does \u2028 work as a new line character
when tailing logs in the CLI? Anyone tried this unicode new line character
in various consoles? IDE, xterm, etc? I'm wondering if developers will
need to have different config for development.
Mike
On Mon, Mar 14, 2016 at 12:17 PM, Jim CF Campbell <
jcampbell(a)pivotal.io> wrote:
Hi Mike and Alex,
Two things - for Java, we are working toward defining an enhanced
metric format that will support transport of Multi Lines.
The second is this workaround that David Laing suggested for
Logstash. Think you could use it for Splunk?
With the Java Logback library you can do this by adding
"%replace(%xException){'\n','\u2028'}%nopex" to your logging config[1] ,
and then use the following logstash conf.[2]
Replace the unicode newline character \u2028 with \n, which Kibana
will display as a new line.
mutate {
gsub => [ "[@message]", '\u2028', "
"]
^^^ Seems that passing a string with an actual newline in it is the
only way to make gsub work
}
to replace the token with a regular newline again so it displays
"properly" in Kibana.
[1] github.com/dpin...ication.yml#L12
<https://github.com/dpinto-pivotal/cf-SpringBootTrader-config/blob/master/application.yml#L12>
[2] github.com/logs...se.conf#L60-L64
<https://github.com/logsearch/logsearch-for-cloudfoundry/blob/master/src/logsearch-config/src/logstash-filters/snippets/firehose.conf#L60-L64>
On Mon, Mar 14, 2016 at 11:11 AM, Mike Youngstrom <youngm(a)gmail.com>
wrote:
I'll let the Loggregator team respond formally. But, in my
conversations with the Loggregator team I think we're basically stuck not
sure what the right thing to do is on the client side. How does the client
trigger in loggregator that this is a multi line log message or what is the
right way for loggregator to detect that the client is trying to send a
multi line log message? Any ideas?
Mike
On Mon, Mar 14, 2016 at 10:25 AM, Aliaksandr Prysmakou <
prysmakou(a)gmail.com> wrote:
Hi guys,
Are there any updates about "Multiline Log Entry" issue? How
correctly deal with stacktraces?
Links to the tracker to read?
----
Alex Prysmakou / Altoros
Tel: (617) 841-2121 ext. 5161 | Toll free: 855-ALTOROS
Skype: aliaksandr.prysmakou
www.altoros.com | blog.altoros.com | twitter.com/altoros
--
Jim Campbell | Product Manager | Cloud Foundry | Pivotal.io |
303.618.0963
--
Jim Campbell | Product Manager | Cloud Foundry | Pivotal.io | 303.618.0963
|
|
|
That other thread Rob and Alvaro just mentioned currently leaves open the question of why metron_agent worked when deployed from Micro BOSH on stemcell 3160 but started giving this error message on 3163. For sanity, I wlll continue the discussion on that other thread. Best, Amit On Tue, Apr 12, 2016 at 9:40 AM, Release Integration <cfruntime(a)gmail.com> wrote: As Amit wrote in the other, related thread -
https://lists.cloudfoundry.org/archives/list/cf-dev(a)lists.cloudfoundry.org/thread/RYMBWNUMMWSDPIE3DZ5JUVN2VNYXDWU6/
This will not work with dynamic networks. Many jobs in cf-release rely on data from BOSH to determine their IP so that configuration files can be
rendered up-front by the director rather than at runtime, requiring system
calls to determine IP. metron_agent is one such job, and it tends to be
colocated with each other job (it is what allows all system component logs
to be aggregated through the loggregator system), so this would require all
Cloud Foundry VMs to be on a manual network. You don't need to manually
pick the IPs, you just need to tell BOSH which IPs in the network not to
use and specify these in the "reserved" range.
Since so many different components depend on being able to determine their IP via BOSH data, there's no quick workaround if you want to stick to
using dynamic networks, but we're aware of this current limitation.
Thanks,
Rob & Alvaro
CF Release Integration
Pivotal
|
|
|
Re: Request for Multibuildpack Use Cases
Multibuildpack is absolutely useful and I'm excited for this proposal. I encounter a lot of use cases for this. The most common is that an application wants to pull in private dependencies during a future dependency-resolution step of a later buildpack, but the dependency resolver needs to be primed in some specific way. If you wait until buildpack time it's too late. On Heroku, for example, this is accomplished by having something like the netrc buildpack ( https://github.com/timshadel/heroku-buildpack-github-netrc), adding a GITHUB_TOKEN environment variable, and then running your "real" buildpack. The netrc BP runs first, allowing Bundler to see the private dependencies. best, ~ jf
On Tue, Apr 12, 2016 at 12:36 PM Jack Cai <greensight(a)gmail.com> wrote: It would be more useful if the multi-buildpack can reference an admin
buildpack in addition to a remote git-hosted buildpack. :-)
Jack
On Tue, Apr 12, 2016 at 6:38 AM, David Illsley <davidillsley(a)gmail.com>
wrote:
In the past we've used the multi-buildpack to be able to use ruby sass to
compile SCSS for non-ruby projects (node and Java). In that case we used
the multi-buildpack and a .buildpacks file which worked reasonably well
(and was very clear).
On Mon, Apr 11, 2016 at 1:15 AM, Danny Rosen <drosen(a)pivotal.io> wrote:
Hi there,
The CF Buildpacks team is considering taking on a line of work to
provide more formal support for multibuildpacks. Before we start, we would
be interested in learning if any community users have compelling use cases
they could share with us.
For more information on multibuildpacks, see Heroku's documentation [1]
[1] -
https://devcenter.heroku.com/articles/using-multiple-buildpacks-for-an-app
|
|
|
As Amit wrote in the other, related thread - https://lists.cloudfoundry.org/archives/list/cf-dev(a)lists.cloudfoundry.org/thread/RYMBWNUMMWSDPIE3DZ5JUVN2VNYXDWU6/ This will not work with dynamic networks. Many jobs in cf-release rely on data from BOSH to determine their IP so that configuration files can be rendered up-front by the director rather than at runtime, requiring system calls to determine IP. metron_agent is one such job, and it tends to be colocated with each other job (it is what allows all system component logs to be aggregated through the loggregator system), so this would require all Cloud Foundry VMs to be on a manual network. You don't need to manually pick the IPs, you just need to tell BOSH which IPs in the network not to use and specify these in the "reserved" range. Since so many different components depend on being able to determine their IP via BOSH data, there's no quick workaround if you want to stick to using dynamic networks, but we're aware of this current limitation. Thanks, Rob & Alvaro CF Release Integration Pivotal
|
|
|
Re: CF212 - golang app crash - how to perform the troubleshooting?
They are passed via environment variables and then your start command is used to start the app. This is an example of the startup script < https://github.com/cloudfoundry/dea_ng/blob/master/lib/dea/starting/startup_script_generator.rb> generated. Does this help? On Tue, Apr 12, 2016 at 7:52 AM, Rafal Radecki <radecki.rafal(a)gmail.com> wrote: Hi :)
I am starting an app and it crashes. In the meantime when it is trying to
start I login to appropriate warden container and after the app crashes I
try to start the go binary manually. The problem is that I am not able to
see in the environment variables like VCAP_SERVICES and I cannot pass its
content correctly to my app.
Can you tell me how to start my app inside the warden container once I
login through wsh with proper environment setup? I tried below:
root(a)19g7korv77m:/app# export
VCAP_SERVICES='{"mongodb26":[{"credentials":{"dbname":"...}
root(a)19g7korv77m:/app# bin/my_go_binary
but it is obviously not the proper way. How does cloudfoundry pass
variables to the process started in the warden container?
BR,
Rafal.
|
|
|
Re: Request for Multibuildpack Use Cases
It would be more useful if the multi-buildpack can reference an admin buildpack in addition to a remote git-hosted buildpack. :-) Jack On Tue, Apr 12, 2016 at 6:38 AM, David Illsley <davidillsley(a)gmail.com> wrote: In the past we've used the multi-buildpack to be able to use ruby sass to
compile SCSS for non-ruby projects (node and Java). In that case we used
the multi-buildpack and a .buildpacks file which worked reasonably well
(and was very clear).
On Mon, Apr 11, 2016 at 1:15 AM, Danny Rosen <drosen(a)pivotal.io> wrote:
Hi there,
The CF Buildpacks team is considering taking on a line of work to provide
more formal support for multibuildpacks. Before we start, we would be
interested in learning if any community users have compelling use cases
they could share with us.
For more information on multibuildpacks, see Heroku's documentation [1]
[1] -
https://devcenter.heroku.com/articles/using-multiple-buildpacks-for-an-app
|
|
|
Re: Doppler/Firehose - Multiline Log Entry
Mike, When you get a bit more desperate ;-) here is a nozzle plug in < https://github.com/jtuchscherer/nozzle-plugin> for the CLI. It's attaches to the firehose to display everything, but would be easy to modify to just look at a single app, and sub out the magic token for newlines. Jim
On Tue, Apr 12, 2016 at 9:56 AM, Mike Youngstrom <youngm(a)gmail.com> wrote: Hi David,
The problem for me is that I'm searching for a solution that can works for
development (though less of a priority cause you can switch config between
dev and cf) and for viewing logs via "cf logs" in addition to a log
aggregator. I had hoped that /u2028 would work for viewing logs via "cf
logs" but it doesn't in bash. I'd need to write a plugin or something for
cf logs and train all my users to use it. Certainly possible but I'm not
that desperate yet. :)
Mike
On Tue, Apr 12, 2016 at 5:58 AM, David Laing <david(a)davidlaing.com> wrote:
FWIW, the technique is to have your logging solution (eg, logback, log4j)
log a token (eg, \u2028) other than \n to denote line breaks in
your stack traces; and then have your log aggregation software replace that
token with a \n again when processing the log messages.
If \u2028 doesn't work in your environment; use something else; eg
NEWLINE
On Mon, 11 Apr 2016 at 21:12 Mike Youngstrom <youngm(a)gmail.com> wrote:
Finally got around to testing this. Preliminary testing show that "\u2028"
doesn't function as a new line character in bash and causes eclipse console
to wig out. I don't think "\u2028" is a viable long term solution.
Hope you make progress on a metric format available to an app in a
container. I too would like a tracker link to such a feature if there is
one.
Thanks,
Mike
On Mon, Mar 14, 2016 at 2:28 PM, Mike Youngstrom <youngm(a)gmail.com>
wrote:
Hi Jim,
So, to be clear what we're basically doing is using unicode newline
character to fool loggregator (which is looking for \n) into thinking that
it isn't a new log event right? Does \u2028 work as a new line character
when tailing logs in the CLI? Anyone tried this unicode new line character
in various consoles? IDE, xterm, etc? I'm wondering if developers will
need to have different config for development.
Mike
On Mon, Mar 14, 2016 at 12:17 PM, Jim CF Campbell <jcampbell(a)pivotal.io
wrote: Hi Mike and Alex,
Two things - for Java, we are working toward defining an enhanced
metric format that will support transport of Multi Lines.
The second is this workaround that David Laing suggested for Logstash.
Think you could use it for Splunk?
With the Java Logback library you can do this by adding
"%replace(%xException){'\n','\u2028'}%nopex" to your logging config[1] ,
and then use the following logstash conf.[2]
Replace the unicode newline character \u2028 with \n, which Kibana
will display as a new line.
mutate {
gsub => [ "[@message]", '\u2028', "
"]
^^^ Seems that passing a string with an actual newline in it is the
only way to make gsub work
}
to replace the token with a regular newline again so it displays
"properly" in Kibana.
[1] github.com/dpin...ication.yml#L12
<https://github.com/dpinto-pivotal/cf-SpringBootTrader-config/blob/master/application.yml#L12>
[2] github.com/logs...se.conf#L60-L64
<https://github.com/logsearch/logsearch-for-cloudfoundry/blob/master/src/logsearch-config/src/logstash-filters/snippets/firehose.conf#L60-L64>
On Mon, Mar 14, 2016 at 11:11 AM, Mike Youngstrom <youngm(a)gmail.com>
wrote:
I'll let the Loggregator team respond formally. But, in my
conversations with the Loggregator team I think we're basically stuck not
sure what the right thing to do is on the client side. How does the client
trigger in loggregator that this is a multi line log message or what is the
right way for loggregator to detect that the client is trying to send a
multi line log message? Any ideas?
Mike
On Mon, Mar 14, 2016 at 10:25 AM, Aliaksandr Prysmakou <
prysmakou(a)gmail.com> wrote:
Hi guys,
Are there any updates about "Multiline Log Entry" issue? How
correctly deal with stacktraces?
Links to the tracker to read?
----
Alex Prysmakou / Altoros
Tel: (617) 841-2121 ext. 5161 | Toll free: 855-ALTOROS
Skype: aliaksandr.prysmakou
www.altoros.com | blog.altoros.com | twitter.com/altoros
--
Jim Campbell | Product Manager | Cloud Foundry | Pivotal.io |
303.618.0963
--
Jim Campbell | Product Manager | Cloud Foundry | Pivotal.io | 303.618.0963
|
|
|
Hi, Trying to a fresh new installation of CF on AWS - I'm hitting the following issue when the deployment is trying to update the api_z1/0 jobs. I'm following the instruction - http://docs.cloudfoundry.org/deploying/aws/cf-stub.html and as well http://docs.cloudfoundry.org/deploying/common/consul-security.html to generate the consul certificates. According to the log - the consult_agent process failed to start. {"timestamp":"1460475448.173879385","source":"confab","message":"confab.agent-client.verify-joined.members.request.failed","log_level":2,"data":{"error":"Get http://127.0.0.1:8500/v1/agent/members: dial tcp 127.0.0.1:8
500: getsockopt: connection refused","wan":false}}
==> Starting Consul agent...
==> Error starting agent: Failed to start Consul client: Failed to load
cert/key pair: crypto/tls: failed to parse certificate PEM data
How can I debug this issue ? Cheers, sylvain
|
|
|
Re: Doppler/Firehose - Multiline Log Entry
Mike Youngstrom <youngm@...>
Hi David,
The problem for me is that I'm searching for a solution that can works for
development (though less of a priority cause you can switch config between
dev and cf) and for viewing logs via "cf logs" in addition to a log
aggregator. I had hoped that /u2028 would work for viewing logs via "cf
logs" but it doesn't in bash. I'd need to write a plugin or something for
cf logs and train all my users to use it. Certainly possible but I'm not
that desperate yet. :)
Mike
toggle quoted message
Show quoted text
On Tue, Apr 12, 2016 at 5:58 AM, David Laing <david(a)davidlaing.com> wrote: FWIW, the technique is to have your logging solution (eg, logback, log4j)
log a token (eg, \u2028) other than \n to denote line breaks in
your stack traces; and then have your log aggregation software replace that
token with a \n again when processing the log messages.
If \u2028 doesn't work in your environment; use something else; eg NEWLINE
On Mon, 11 Apr 2016 at 21:12 Mike Youngstrom <youngm(a)gmail.com> wrote:
Finally got around to testing this. Preliminary testing show that "\u2028"
doesn't function as a new line character in bash and causes eclipse console
to wig out. I don't think "\u2028" is a viable long term solution.
Hope you make progress on a metric format available to an app in a
container. I too would like a tracker link to such a feature if there is
one.
Thanks,
Mike
On Mon, Mar 14, 2016 at 2:28 PM, Mike Youngstrom <youngm(a)gmail.com>
wrote:
Hi Jim,
So, to be clear what we're basically doing is using unicode newline
character to fool loggregator (which is looking for \n) into thinking that
it isn't a new log event right? Does \u2028 work as a new line character
when tailing logs in the CLI? Anyone tried this unicode new line character
in various consoles? IDE, xterm, etc? I'm wondering if developers will
need to have different config for development.
Mike
On Mon, Mar 14, 2016 at 12:17 PM, Jim CF Campbell <jcampbell(a)pivotal.io>
wrote:
Hi Mike and Alex,
Two things - for Java, we are working toward defining an enhanced
metric format that will support transport of Multi Lines.
The second is this workaround that David Laing suggested for Logstash.
Think you could use it for Splunk?
With the Java Logback library you can do this by adding
"%replace(%xException){'\n','\u2028'}%nopex" to your logging config[1] ,
and then use the following logstash conf.[2]
Replace the unicode newline character \u2028 with \n, which Kibana will
display as a new line.
mutate {
gsub => [ "[@message]", '\u2028', "
"]
^^^ Seems that passing a string with an actual newline in it is the
only way to make gsub work
}
to replace the token with a regular newline again so it displays
"properly" in Kibana.
[1] github.com/dpin...ication.yml#L12
<https://github.com/dpinto-pivotal/cf-SpringBootTrader-config/blob/master/application.yml#L12>
[2] github.com/logs...se.conf#L60-L64
<https://github.com/logsearch/logsearch-for-cloudfoundry/blob/master/src/logsearch-config/src/logstash-filters/snippets/firehose.conf#L60-L64>
On Mon, Mar 14, 2016 at 11:11 AM, Mike Youngstrom <youngm(a)gmail.com>
wrote:
I'll let the Loggregator team respond formally. But, in my
conversations with the Loggregator team I think we're basically stuck not
sure what the right thing to do is on the client side. How does the client
trigger in loggregator that this is a multi line log message or what is the
right way for loggregator to detect that the client is trying to send a
multi line log message? Any ideas?
Mike
On Mon, Mar 14, 2016 at 10:25 AM, Aliaksandr Prysmakou <
prysmakou(a)gmail.com> wrote:
Hi guys,
Are there any updates about "Multiline Log Entry" issue? How
correctly deal with stacktraces?
Links to the tracker to read?
----
Alex Prysmakou / Altoros
Tel: (617) 841-2121 ext. 5161 | Toll free: 855-ALTOROS
Skype: aliaksandr.prysmakou
www.altoros.com | blog.altoros.com | twitter.com/altoros
--
Jim Campbell | Product Manager | Cloud Foundry | Pivotal.io |
303.618.0963
|
|
|
Re: CPU weight of application
Hi, Sam,
No, the LRP generated for the CF app will have its CPU weight set
proportionally to its desired memory. Keep in mind that for an app running
on Linux, the CPU weight corresponds to the cpu.shares cgroup, which
enforces a relative amount of CPU usage amongst processes only if there is
contention over CPU. If nothing else is competing for CPU on the VM, your
process can use as much as it is capable of using.
Thanks,
Eric, CF Runtime Diego PM
toggle quoted message
Show quoted text
|
|
|
Re: April CAB call next week on Wednesday April 13th, 2016
Final reminder, see below. If you are a PM, please update your team's highlights so community can review. Also, be ready to discuss more interesting items. Everyone else come with your questions and comments.
Talk soon. Best,
dr.max
ibm cloud labs
sillicon valley, ca
Sent from my iPhone
toggle quoted message
Show quoted text
On Apr 6, 2016, at 5:31 PM, Michael Maximilien <maxim(a)us.ibm.com> wrote:
Hi, all,
Quick reminder of the CAB call next Wednesday, April 13th @ 8a PDT. All info in link:
https://docs.google.com/document/d/1SCOlAquyUmNM-AQnekCOXiwhLs6gveTxAcduvDcW_xI/edit#heading=h.o44xhgvum2we
Remember, no more status update but rather discussions, so come ready with your questions.
Join the slack.cloudfoundry.org and join the #CAB channel for previous and future discussions.
Talk to you all next week. We'll send one more reminder on this list.
Best,
Chip, James, and Max
dr.max
ibm cloud labs
sillicon valley, ca
Sent from my iPhone
dr.max
ibm cloud labs
sillicon valley, ca
Sent from my iPhone
|
|
|
CF212 - golang app crash - how to perform the troubleshooting?
Hi :)
I am starting an app and it crashes. In the meantime when it is trying to start I login to appropriate warden container and after the app crashes I try to start the go binary manually. The problem is that I am not able to see in the environment variables like VCAP_SERVICES and I cannot pass its content correctly to my app.
Can you tell me how to start my app inside the warden container once I login through wsh with proper environment setup? I tried below:
root(a)19g7korv77m:/app# export VCAP_SERVICES='{"mongodb26":[{"credentials":{"dbname":"...}
root(a)19g7korv77m:/app# bin/my_go_binary
but it is obviously not the proper way. How does cloudfoundry pass variables to the process started in the warden container?
BR,
Rafal.
|
|
|
Changing Default Node version in Ruby Buildpack
Hi all, Currently the Ruby Buildpack vendors Node v0.12 to give Ruby applications access to Node. We are proposing to move the Ruby Buildpack's Node version from v0.12 to v4. More details can be found in the Github issues link below: https://github.com/cloudfoundry/ruby-buildpack/issues/53Please comment on the issue or this thread if you have any concerns or feedback. Thank you! Buildpack Team
|
|
|
Changing Node version of Ruby Buildpack
Hi all, Currently the Ruby Buildpack vendors Node v0.12 to give Ruby applications access to Node. We are proposing to move the Ruby Buildpack's Node version from v0.12 to v4. More details can be found in the Github issues link below: https://github.com/cloudfoundry/ruby-buildpack/issues/53Please comment on the issue or this thread if you have any concerns or feedback. Thank you! Buildpack Team
|
|
|
Re: Doppler/Firehose - Multiline Log Entry
FWIW, the technique is to have your logging solution (eg, logback, log4j)
log a token (eg, \u2028) other than \n to denote line breaks in your stack
traces; and then have your log aggregation software replace that token with
a \n again when processing the log messages.
If \u2028 doesn't work in your environment; use something else; eg NEWLINE
toggle quoted message
Show quoted text
On Mon, 11 Apr 2016 at 21:12 Mike Youngstrom <youngm(a)gmail.com> wrote: Finally got around to testing this. Preliminary testing show that "\u2028"
doesn't function as a new line character in bash and causes eclipse console
to wig out. I don't think "\u2028" is a viable long term solution. Hope
you make progress on a metric format available to an app in a container. I
too would like a tracker link to such a feature if there is one.
Thanks,
Mike
On Mon, Mar 14, 2016 at 2:28 PM, Mike Youngstrom <youngm(a)gmail.com> wrote:
Hi Jim,
So, to be clear what we're basically doing is using unicode newline
character to fool loggregator (which is looking for \n) into thinking that
it isn't a new log event right? Does \u2028 work as a new line character
when tailing logs in the CLI? Anyone tried this unicode new line character
in various consoles? IDE, xterm, etc? I'm wondering if developers will
need to have different config for development.
Mike
On Mon, Mar 14, 2016 at 12:17 PM, Jim CF Campbell <jcampbell(a)pivotal.io>
wrote:
Hi Mike and Alex,
Two things - for Java, we are working toward defining an enhanced metric
format that will support transport of Multi Lines.
The second is this workaround that David Laing suggested for Logstash.
Think you could use it for Splunk?
With the Java Logback library you can do this by adding
"%replace(%xException){'\n','\u2028'}%nopex" to your logging config[1] ,
and then use the following logstash conf.[2]
Replace the unicode newline character \u2028 with \n, which Kibana will
display as a new line.
mutate {
gsub => [ "[@message]", '\u2028', "
"]
^^^ Seems that passing a string with an actual newline in it is the only
way to make gsub work
}
to replace the token with a regular newline again so it displays
"properly" in Kibana.
[1] github.com/dpin...ication.yml#L12
<https://github.com/dpinto-pivotal/cf-SpringBootTrader-config/blob/master/application.yml#L12>
[2] github.com/logs...se.conf#L60-L64
<https://github.com/logsearch/logsearch-for-cloudfoundry/blob/master/src/logsearch-config/src/logstash-filters/snippets/firehose.conf#L60-L64>
On Mon, Mar 14, 2016 at 11:11 AM, Mike Youngstrom <youngm(a)gmail.com>
wrote:
I'll let the Loggregator team respond formally. But, in my
conversations with the Loggregator team I think we're basically stuck not
sure what the right thing to do is on the client side. How does the client
trigger in loggregator that this is a multi line log message or what is the
right way for loggregator to detect that the client is trying to send a
multi line log message? Any ideas?
Mike
On Mon, Mar 14, 2016 at 10:25 AM, Aliaksandr Prysmakou <
prysmakou(a)gmail.com> wrote:
Hi guys,
Are there any updates about "Multiline Log Entry" issue? How correctly
deal with stacktraces?
Links to the tracker to read?
----
Alex Prysmakou / Altoros
Tel: (617) 841-2121 ext. 5161 | Toll free: 855-ALTOROS
Skype: aliaksandr.prysmakou
www.altoros.com | blog.altoros.com | twitter.com/altoros
--
Jim Campbell | Product Manager | Cloud Foundry | Pivotal.io |
303.618.0963
|
|
|
Re: Request for Multibuildpack Use Cases
David Illsley <davidillsley@...>
In the past we've used the multi-buildpack to be able to use ruby sass to
compile SCSS for non-ruby projects (node and Java). In that case we used
the multi-buildpack and a .buildpacks file which worked reasonably well
(and was very clear).
toggle quoted message
Show quoted text
On Mon, Apr 11, 2016 at 1:15 AM, Danny Rosen <drosen(a)pivotal.io> wrote: Hi there,
The CF Buildpacks team is considering taking on a line of work to provide
more formal support for multibuildpacks. Before we start, we would be
interested in learning if any community users have compelling use cases
they could share with us.
For more information on multibuildpacks, see Heroku's documentation [1]
[1] -
https://devcenter.heroku.com/articles/using-multiple-buildpacks-for-an-app
|
|
|
Why hm9000 write message to /var/log/syslog
Hi,
We recently encountered a CF crash due to full disk issue of root file system of CC and HM9000. I found HM9000 wrote log message to /var/log/syslog. Does anybody know why? I think CF jobs should not use root file system since the stemcell only has 3G space.
CF version is 197.
Thanks,
Maggie
|
|
|
