Date   

cf stop not sending SIGTERM

Will Tran
 

I'm not seeing graceful shutdown behaviour when I cf stop a Java app. By the looks of Garden SIGTERM is sent and if the process is still alive after 10 seconds SIGKILL is sent: https://github.com/cloudfoundry-incubator/garden-linux/blob/8ab2cbf0696fd0b6bd1a0f15f2d3a463e4acbdac/linux_backend/skeleton/stop.sh#L54-L72

I don't see this in my dead simple app Spring Boot app however: https://github.com/william-tran/cf-shutdown-hook. When I SIGTERM locally it only takes a second to gracefully shutdown.

I'm seeing this on PCF 1.6.8-build.1 (cf 225.3, garden 0.330.0)


Re: cf v233 api_z1/api_z2 failing

Peter Goetz <peter.gtz@...>
 

Hi Ranga,

To trouble-shoot this issue could you also provide the contents of
/var/vcap/sys/log/cloud_controller_ng/cloud_controller_ng.log
and /var/vcap/sys/log/cloud_controller_worker/cloud_controller_worker.log?
This should give us more details about what's going on. The ctl script logs
do not provide enough details.

Thanks,
Peter

On Wed, Apr 6, 2016 at 6:12 PM Ranga Rajagopalan <
ranga.rajagopalan(a)gmail.com> wrote:

I tried v231. Unfortunately, same issue.


Re: [ Logging properties ] change buffer size for Containers SDOUT

Eric Malm <emalm@...>
 

Hi, Fabien,

If you're running the buildpack staging tasks on Diego cells, Jim is
correct that the executor's log-streaming system currently imposes a
hard-coded 4-KiB limit on each log line. The Diego team has a story (
https://www.pivotaltracker.com/story/show/111866195) to increase that to 60
KiB that we expect to get to in a few weeks. The original limit was
introduced very early in the life of the executor component, and was likely
based on some too-restrictive assumptions about the maximum size of UDP
datagrams. We've since validated that on Linux and on Windows these
datagrams can contain a little less than 64KiB of data in their payload, so
the 60-KiB limit should be fine.

Thanks,
Eric, CF Runtime Diego PM

On Thu, Apr 7, 2016 at 10:12 AM, Jim CF Campbell <jcampbell(a)pivotal.io>
wrote:

Hi Fabien,

It's a restriction that the Executor component of Diego implements. It's
not in the Loggregator code. I'm not sure of the where or why of it.

Jim


On Thu, Apr 7, 2016 at 6:44 AM, Fabien Guichard <
fabien.guichard(a)orange.com> wrote:

Hello,

while developping an extension ofr a buildpack, related to logging
event from applications, we found out that STDOUT logs where being flushed
with a fixed size (because we're receving troncated strings in our external
aggregating system).

it seems to us that the buffer size is around 4000 caracters, which is
not what we would like.

We went through doppler metron and loggregator configurations & codes,
without success, to find out anything related to fixed-size logging
property.

Do ou know where we should go to configure the size of the buffer being
flushed please ?

Many thanks for the time you took to read our very first question on this
ML.

Regards,
Fabien.


--
Jim Campbell | Product Manager | Cloud Foundry | Pivotal.io | 303.618.0963


Re: [ Logging properties ] change buffer size for Containers SDOUT

Jim CF Campbell
 

Hi Fabien,

It's a restriction that the Executor component of Diego implements. It's
not in the Loggregator code. I'm not sure of the where or why of it.

Jim


On Thu, Apr 7, 2016 at 6:44 AM, Fabien Guichard <fabien.guichard(a)orange.com>
wrote:

Hello,

while developping an extension ofr a buildpack, related to logging
event from applications, we found out that STDOUT logs where being flushed
with a fixed size (because we're receving troncated strings in our external
aggregating system).

it seems to us that the buffer size is around 4000 caracters, which is not
what we would like.

We went through doppler metron and loggregator configurations & codes,
without success, to find out anything related to fixed-size logging
property.

Do ou know where we should go to configure the size of the buffer being
flushed please ?

Many thanks for the time you took to read our very first question on this
ML.

Regards,
Fabien.


--
Jim Campbell | Product Manager | Cloud Foundry | Pivotal.io | 303.618.0963


Re: Proposal for Service Discovery within Elastic Clusters

Amit Kumar Gupta
 

Hi all,

Sorry for the confusion, I originally added some extraneous docs to the
Required Reading list that people have been trying and failing to access.
Those links have been removed, the most up-to-date information (about the
general Elastic Clusters feature narrative) is in what's now the first
link, titled "Cloud Foundry Elastic Clusters".

Cheers,
Amit

On Wed, Apr 6, 2016 at 6:52 PM, Amit Gupta <agupta(a)pivotal.io> wrote:

Hi all,

If you are familiar with the proposed Elastic Clusters feature narrative
(or even if you're not), I welcome you to take a look at the below proposal
for Consul-based service discovery for components within an Elastic
Clusters configuration.


https://docs.google.com/document/d/1aMpIXsPpB6O_oJsoazOl7kGajc3sRnlrflQFCtysfx8/edit

Feedback is eagerly solicited.

Links to background information on Elastic Clusters can be found within
the above doc.

Thanks,
Amit, CF Infrastructure team PM


Re: How can we customized "404 Not Found"

Mike Youngstrom <youngm@...>
 

We customized our 404 mapping wildcard routes to an app. We also display a
503 if the route exists.

Mike

On Apr 7, 2016 9:18 AM, "Stanley Shen" <meteorping(a)gmail.com> wrote:

When we accessing an APP which is not up or not existed, the CF will
display error message like:

404 Not Found: Requested route ('test.domain.name') does not exist.

Can we customize it, and how can we do that?


Thanks in advance.


How can we customized "404 Not Found"

Stanley Shen <meteorping@...>
 

When we accessing an APP which is not up or not existed, the CF will display error message like:

404 Not Found: Requested route ('test.domain.name') does not exist.

Can we customize it, and how can we do that?


Thanks in advance.


Re: OpenStack Leads

David <David@...>
 

Did you get a chance to review my previous email?



Let me know your thoughts!



~David



From: David [mailto:David(a)lead2tech.com]
Sent: Tuesday, April 05, 2016 8:42 AM
To: 'cf-dev(a)lists.cloudfoundry.org'
Subject: OpenStack Leads





Hi,



I hope this email finds you well.



As your organization is a part of " OpenStack Summit 2016 " I'm writing to
check if you would be interested in a leads of OpenStack users for marketing
campaign and lead generation initiative.



You may choose job titles such as: IT Architects, CTOs, IT Directors, Data
Centre Managers, Developers and more.



Each lead will be delivered to you with multi-channel data fields such as
Company name, Web address, Contact Name, Job Title, Phone Number, Direct
Business Emails, Postal address, Industry, employee size, Revenue size, SIC
code, Fax number, Application/Technology usage etc.



Appreciate your time and look forward to hearing from you.



Regards,



David Reid| Lead Generation Specialist
Lead2Tech LLC| Technology Division

If you do not wish to receive any more emails please reply to this email
with Leave out in the subject line.











<http://1816396.sigclick.mailinfo.com/sigclick/09000306/0000054D/0E0B0607/04
450118820393.jpg>


Automated Deployment of window cell for diego

Long Nguyen
 

Anyone know about automated deployment of window cell for diego? I've done it manually a few months back and it was very manual and error prone. Does anyone have anything that automate process?


Re: Internal/external domains

Geoff Franks <geoff@...>
 

We built the cf-haproxy-boshrelease for this (https://github.com/cloudfoundry-community/cf-haproxy-boshrelease). Check out https://blog.starkandwayne.com/2015/06/17/securing-private-domains-on-a-public-cloudfoundry/ for more details. The feature-set has been updated quite a bit since that post. It now supports enough features that should allow it to work behind tcp or http-mode load balancers (you can force x-forwarded-proto if behind a tcp-only loadbalancer that does ssl termination), handle websockets on standard http ports, serve multiple ssl certs, and do generic tcp proxying for things like app-ssh.

My ideal setup would be a set of http-mode load balancers in front of the 'public_haproxy_z*' nodes, which would filter out non-allowed domains, and send the remaining requests to the gorouters. For internal-only requests, I'd use a non-public http-mode load balancer directly in front of the gorouters.


Static IP setup for routers on AWS

Engelke, Johannes <info@...>
 

Hi,
does anybody know, why the routers got static ips in the cf-infrastructure-aws.yml file? https://github.com/cloudfoundry/cf-release/blob/master/templates/cf-infrastructure-aws.yml#L173 <https://github.com/cloudfoundry/cf-release/blob/master/templates/cf-infrastructure-aws.yml#L173>

Bosh is assigning the instances to ELB’s during deploy time, so there should be no need to have static addresses here.

If nobody know’s a good reason should we remove them ;-)

Cheers
Johannes


CF 212 - service in "delete failed" -> how to remove it properly?

Rafal Radecki
 

Hi :)

I have several organizations in which I have services in "delete failed" state. They prevent me from deleting the organization.

What is the best way to remove metadata about such service?

BR,
Rafal.


[ Logging properties ] change buffer size for Containers SDOUT

Fabien Guichard
 

Hello,

while developping an extension ofr a buildpack, related to logging event from applications, we found out that STDOUT logs where being flushed with a fixed size (because we're receving troncated strings in our external aggregating system).

it seems to us that the buffer size is around 4000 caracters, which is not what we would like.

We went through doppler metron and loggregator configurations & codes, without success, to find out anything related to fixed-size logging property.

Do ou know where we should go to configure the size of the buffer being flushed please ?

Many thanks for the time you took to read our very first question on this ML.

Regards,
Fabien.


Re: Internal/external domains

Matthew Sykes <matthew.sykes@...>
 

Container networking is aimed at your requirement. Once it's ready, you
would deploy your apps without external routes and rely on the private
network for communication.

We expect we'll have facilities in place to find apps via a DNS
implementation using app names or services names but we''re beginning to
move in a direction where apps are associated with explicit service names
that are orthogonal to the app name. We may also adopt a VIP pattern for
services to remove the need name resolution entirely.

For status, we have containers talking via n overlay with guardian and
we're in the process of integrating with diego. We hope to demonstrate what
we have at summit. It will be a while before things are "baked" but the
project exists and we're moving forward.

Finally, there are plans to implement several policy controls around
endpoint communication that enable users to express a supported
connectivity graph without relying on apps living in the same space for
communication.

Hope that helps. Let us know if you have more questions.

On Thu, Apr 7, 2016 at 1:28 AM, Krannich, Bernd <bernd.krannich(a)sap.com>
wrote:

Hi all,

Dr. Nic’s blog post [1] addresses a requirement that we keep hearing
frequently: To have certain applications that can only be accessed from
other apps running in the same Cloud Foundry deployment.

I wanted to check with the community here to see if anybody has come
across this requirement as well. Is this something one should actually be
advocating in terms of architecture?

Also, the solution suggested in [1] in our opinion is prone to bypassing
if implemented in the described way (see top-most comment in the same blog
post). Of course, one could work around that by terminating SSL at the two
load balancers, introspecting the header fields there. On the other hand,
this requires a much more complex “internal/external” DNS setup.

A logical next step would be to extend the requirement to only allow calls
from apps being deployed into certain orgs and spaces. This can no longer
be achieved by a setup “external” to CF given that apps from all types of
orgs and spaces are usually deployed on the very same runners with calls
being “indistinguishable” once a call has left the runner.

As alternatives, we came up with the following potential options:

1. Avoiding the notion of “private” services altogether, making sure that
each application is secured by - for example - a JWT token carrying a
shared secret.
2. Using messaging via a backing service instead of HTTP-based
communication.
3. Recently suggested/implemented CF functionality like “container
networking for applications” [2] (what’s the status on this one, actually?)
and “Context Path Routing” [3] (this was discussed on the mailing list some
time back, even though I’m not sure how you would actually use it to solve
the above scenario). Is any of that allowing to implement the above
requirement?

Any thoughts, opinions, suggestions are welcome.

Thanks in advance,
Bernd

[1]
https://blog.starkandwayne.com/2014/10/31/public-and-private-microservices-on-the-same-cloud-foundry
[2]
https://docs.google.com/document/d/1zQJqIEk4ldHH5iE5zat_oKIK8Ejogkgd_lySpg_oV_s/edit?usp=sharing
[3]
http://berlin2015.cfsummit.com/sites/berlin2015.cfsummit.com/files//pages/files/summit-berlin.pdf

Bernd Krannich
Products & Innovation Technology
SAP SE
Dietmar-Hopp-Allee 16
69190 Walldorf Germany
mailto:bernd.krannich(a)sap.com
www.sap.com

Pflichtangaben/Mandatory Disclosure Statements:
http://www.sap.com/company/legal/impressum.epx

Diese E-Mail kann Betriebs- oder Geschäftsgeheimnisse oder sonstige
vertrauliche Informationen enthalten. Sollten Sie diese E-Mail irrtümlich
erhalten haben, ist Ihnen eine Kenntnisnahme des Inhalts, eine
Vervielfältigung oder Weitergabe der E-Mail ausdrücklich untersagt. Bitte
benachrichtigen Sie uns und vernichten Sie die empfangene E-Mail. Vielen
Dank.

This e-mail may contain trade secrets or privileged, undisclosed, or
otherwise confidential information. If you have received this e-mail in
error, you are hereby notified that any review, copying, or distribution of
it is strictly prohibited. Please inform us immediately and destroy the
original transmittal. Thank you for your cooperation.


--
Matthew Sykes
matthew.sykes(a)gmail.com


HAproxy server connection limitation

MaggieMeng
 

Hi,

Is there connection limitation on HAproxy server? We had encountered a connection issue recently. HAproxy server failed to accept new connection request after connection number reached 2000+. Is there a way to change the connection limitation?

The CF version is 197 and below is the content of haproxy server configuration file.

global
log 127.0.0.1 syslog info
daemon
user vcap
group vcap
maxconn 64000
spread-checks 4

defaults
log global
timeout connect 30000ms

frontend http-in
mode http
bind :80
option httplog
option forwardfor
reqadd X-Forwarded-Proto:\ http
default_backend http-routers



frontend https-in
mode http
bind :443 ssl crt /var/vcap/jobs/haproxy/config/cert.pem no-sslv3 ciphers
option httplog
option forwardfor
option http-server-close
reqadd X-Forwarded-Proto:\ https
default_backend http-routers

frontend ssl-in
mode tcp
bind :4443 ssl crt /var/vcap/jobs/haproxy/config/cert.pem no-sslv3 ciphers
default_backend tcp-routers


backend http-routers
mode http
balance roundrobin

Thanks,
Maggie


Re: April CAB call next week on Wednesday April 13th, 2016

Dr Nic Williams <drnicwilliams@...>
 

Hurray I'll be in a useful timezone and can join!
"Living in Australia" and "CAB calls" are mutually exclusive :)
Nic

On Wed, Apr 6, 2016 at 5:36 PM -0700, "Michael Maximilien" <maxim(a)us.ibm.com> wrote:












Hi, all,

Quick reminder of the CAB call next Wednesday, April 13th @ 8a PDT. All info in link:

https://docs.google.com/document/d/1SCOlAquyUmNM-AQnekCOXiwhLs6gveTxAcduvDcW_xI/edit#heading=h.o44xhgvum2we

Remember, no more status update but rather discussions, so come ready with your questions.

Join the slack.cloudfoundry.org and join the #CAB channel for previous and future discussions.

Talk to you all next week. We'll send one more reminder on this list.

Best,

Chip, James, and Max

dr.max
ibm cloud labs
sillicon valley, ca

Sent from my iPhone


dr.max
ibm cloud labs
sillicon valley, ca

Sent from my iPhone


Internal/external domains

Krannich, Bernd <bernd.krannich@...>
 

Hi all,

Dr. Nic’s blog post [1] addresses a requirement that we keep hearing frequently: To have certain applications that can only be accessed from other apps running in the same Cloud Foundry deployment.

I wanted to check with the community here to see if anybody has come across this requirement as well. Is this something one should actually be advocating in terms of architecture?

Also, the solution suggested in [1] in our opinion is prone to bypassing if implemented in the described way (see top-most comment in the same blog post). Of course, one could work around that by terminating SSL at the two load balancers, introspecting the header fields there. On the other hand, this requires a much more complex “internal/external” DNS setup.

A logical next step would be to extend the requirement to only allow calls from apps being deployed into certain orgs and spaces. This can no longer be achieved by a setup “external” to CF given that apps from all types of orgs and spaces are usually deployed on the very same runners with calls being “indistinguishable” once a call has left the runner.

As alternatives, we came up with the following potential options:

1. Avoiding the notion of “private” services altogether, making sure that each application is secured by - for example - a JWT token carrying a shared secret.
2. Using messaging via a backing service instead of HTTP-based communication.
3. Recently suggested/implemented CF functionality like “container networking for applications” [2] (what’s the status on this one, actually?) and “Context Path Routing” [3] (this was discussed on the mailing list some time back, even though I’m not sure how you would actually use it to solve the above scenario). Is any of that allowing to implement the above requirement?

Any thoughts, opinions, suggestions are welcome.

Thanks in advance,
Bernd

[1] https://blog.starkandwayne.com/2014/10/31/public-and-private-microservices-on-the-same-cloud-foundry
[2] https://docs.google.com/document/d/1zQJqIEk4ldHH5iE5zat_oKIK8Ejogkgd_lySpg_oV_s/edit?usp=sharing
[3] http://berlin2015.cfsummit.com/sites/berlin2015.cfsummit.com/files//pages/files/summit-berlin.pdf

Bernd Krannich
Products & Innovation Technology
SAP SE
Dietmar-Hopp-Allee 16
69190 Walldorf Germany
mailto:bernd.krannich(a)sap.com
www.sap.com

Pflichtangaben/Mandatory Disclosure Statements: http://www.sap.com/company/legal/impressum.epx

Diese E-Mail kann Betriebs- oder Geschäftsgeheimnisse oder sonstige vertrauliche Informationen enthalten. Sollten Sie diese E-Mail irrtümlich erhalten haben, ist Ihnen eine Kenntnisnahme des Inhalts, eine Vervielfältigung oder Weitergabe der E-Mail ausdrücklich untersagt. Bitte benachrichtigen Sie uns und vernichten Sie die empfangene E-Mail. Vielen Dank.

This e-mail may contain trade secrets or privileged, undisclosed, or otherwise confidential information. If you have received this e-mail in error, you are hereby notified that any review, copying, or distribution of it is strictly prohibited. Please inform us immediately and destroy the original transmittal. Thank you for your cooperation.


dynamic networks

Vik R <vagcom.ben@...>
 

Are dynamic networks supported in the cf releases?

Ben


Re: Failed to upgrade cf from 212 to 233 (etcd_z1/1 failed)

王小锋 <zzuwxf at gmail.com...>
 

Many thanks !!

2016-04-07 9:55 GMT+08:00 Amit Gupta <agupta(a)pivotal.io>:

Hi there,

Since you've also asked this question on the cf-bosh mailing list:


https://lists.cloudfoundry.org/archives/list/cf-bosh(a)lists.cloudfoundry.org/thread/KPJNZSVAAOQEFLME4BERHL26DDBTDNXJ/

and have provided additional details there, I will ask the core team
responsible for etcd-release to answer your question on the cf-bosh mailing
list.

Cheers,
Amit

On Tue, Apr 5, 2016 at 12:26 AM, 王小锋 <zzuwxf(a)gmail.com> wrote:

Hi, there

I am trying to upgrade cf from version 212 to latest version 233 using
bosh, I am using spiff to generate the new manifest file. During upgrade,
it failed when updating job etcd_z1/1.

I login to etcd_z1/1 VM and found the following error message in etcd
error log.
Anyone meet the same issue? your help is greatly appreciated !!

2016/04/05 06:36:16 filePurge: successfully removed file
/var/vcap/store/etcd/member/snap/00000000000000d6-0000000003712a07.snap
2016/04/05 06:36:16 filePurge: successfully removed file
/var/vcap/store/etcd/member/wal/000000000000168e-0000000003712a08.wal
2016/04/05 06:46:09 rafthttp: server streaming to d7734b019d58ad5c at
term 214 has been stopped
2016/04/05 06:46:09 etcdserver: removed member d7734b019d58ad5c from
cluster f61148e36337417a
2016/04/05 06:46:10 etcdserver: reject message from removed member
d7734b019d58ad5c
2016/04/05 06:46:18 etcdserver: added member e3682036054bdae0 [
http://10.10.16.20:7001] to cluster f61148e36337417a
2016/04/05 06:46:18 sender: error posting to e3682036054bdae0: dial tcp
10.10.16.20:7001: connection refused
2016/04/05 06:46:18 sender: the connection with e3682036054bdae0 became
inactive
2016/04/05 06:46:20 sender: the connection with e3682036054bdae0 became
active
2016/04/05 06:46:20 rafthttp: path msgapp/e3682036054bdae0 cannot be
parsed
2016/04/05 06:46:20 rafthttp: path message/e3682036054bdae0 cannot be
parsed
2016/04/05 06:46:20 raft: ddea47ef60bbcdca received msgApp
rejection(lastindex: 0) from e3682036054bdae0 for index 57802518
2016/04/05 06:46:20 raft: ddea47ef60bbcdca decreased progress of
e3682036054bdae0 to [next = 1, match = 0, wait = 0]
2016/04/05 06:46:20 raft: ddea47ef60bbcdca [firstindex: 57797981, commit:
57802541] sent snapshot[index: 57797980, term: 214] to e3682036054bdae0
[next = 1, match = 0, wait = 0]
2016/04/05 06:46:20 rafthttp: server streaming to e3682036054bdae0 at
term 0 has been stopped
2016/04/05 06:46:20 rafthttp: path message/e3682036054bdae0 cannot be
parsed
2016/04/05 06:46:20 rafthttp: path msgapp/e3682036054bdae0 cannot be
parsed
2016/04/05 06:46:20 rafthttp: path message/e3682036054bdae0 cannot be
parsed


Re: Failed to upgrade cf from 212 to 233 (etcd_z1/1 failed)

Amit Kumar Gupta
 

Hi there,

Since you've also asked this question on the cf-bosh mailing list:

https://lists.cloudfoundry.org/archives/list/cf-bosh(a)lists.cloudfoundry.org/thread/KPJNZSVAAOQEFLME4BERHL26DDBTDNXJ/

and have provided additional details there, I will ask the core team
responsible for etcd-release to answer your question on the cf-bosh mailing
list.

Cheers,
Amit

On Tue, Apr 5, 2016 at 12:26 AM, 王小锋 <zzuwxf(a)gmail.com> wrote:

Hi, there

I am trying to upgrade cf from version 212 to latest version 233 using
bosh, I am using spiff to generate the new manifest file. During upgrade,
it failed when updating job etcd_z1/1.

I login to etcd_z1/1 VM and found the following error message in etcd
error log.
Anyone meet the same issue? your help is greatly appreciated !!

2016/04/05 06:36:16 filePurge: successfully removed file
/var/vcap/store/etcd/member/snap/00000000000000d6-0000000003712a07.snap
2016/04/05 06:36:16 filePurge: successfully removed file
/var/vcap/store/etcd/member/wal/000000000000168e-0000000003712a08.wal
2016/04/05 06:46:09 rafthttp: server streaming to d7734b019d58ad5c at term
214 has been stopped
2016/04/05 06:46:09 etcdserver: removed member d7734b019d58ad5c from
cluster f61148e36337417a
2016/04/05 06:46:10 etcdserver: reject message from removed member
d7734b019d58ad5c
2016/04/05 06:46:18 etcdserver: added member e3682036054bdae0 [
http://10.10.16.20:7001] to cluster f61148e36337417a
2016/04/05 06:46:18 sender: error posting to e3682036054bdae0: dial tcp
10.10.16.20:7001: connection refused
2016/04/05 06:46:18 sender: the connection with e3682036054bdae0 became
inactive
2016/04/05 06:46:20 sender: the connection with e3682036054bdae0 became
active
2016/04/05 06:46:20 rafthttp: path msgapp/e3682036054bdae0 cannot be parsed
2016/04/05 06:46:20 rafthttp: path message/e3682036054bdae0 cannot be
parsed
2016/04/05 06:46:20 raft: ddea47ef60bbcdca received msgApp
rejection(lastindex: 0) from e3682036054bdae0 for index 57802518
2016/04/05 06:46:20 raft: ddea47ef60bbcdca decreased progress of
e3682036054bdae0 to [next = 1, match = 0, wait = 0]
2016/04/05 06:46:20 raft: ddea47ef60bbcdca [firstindex: 57797981, commit:
57802541] sent snapshot[index: 57797980, term: 214] to e3682036054bdae0
[next = 1, match = 0, wait = 0]
2016/04/05 06:46:20 rafthttp: server streaming to e3682036054bdae0 at term
0 has been stopped
2016/04/05 06:46:20 rafthttp: path message/e3682036054bdae0 cannot be
parsed
2016/04/05 06:46:20 rafthttp: path msgapp/e3682036054bdae0 cannot be parsed
2016/04/05 06:46:20 rafthttp: path message/e3682036054bdae0 cannot be
parsed