Hi Ranga,

Looking at your logs we found an error that could possibly cause this and
it is related to the properties.apps_domain in the deployment manifest. By
setting it to 'b%()osh-lite.com' (using special characters), we could
reproduce the following error which we also found in your log file:

Encountered error: name
format\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/sequel-4.29.0/lib/sequel/model/base.rb:1543:in
`save'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/app/models/runtime/shared_domain.rb:35:in
`block in
find_or_create'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/sequel-4.29.0/lib/sequel/database/transactions.rb:134:in
`_transaction'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/sequel-4.29.0/lib/sequel/database/transactions.rb:108:in
`block in
transaction'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/sequel-4.29.0/lib/sequel/database/connecting.rb:249:in
`block in
synchronize'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/sequel-4.29.0/lib/sequel/connection_pool/threaded.rb:103:in
`hold'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/sequel-4.29.0/lib/sequel/database/connecting.rb:249:in
`synchronize'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/sequel-4.29.0/lib/sequel/database/transactions.rb:97:in
`transaction'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/app/models/runtime/shared_domain.rb:27:in
`find_or_create'\n/var/vcap/data/packages/cloud_controller_ng/da452b34d79be56a0784c7e88d6b9c0e1811a9d8.1-f934c136e6019cf54ab7aa04a3c153657226e729/cloud_controller_ng/lib/cloud_controller/seeds.rb:57:in
`block in
create_seed_domains'\n/var/vcap/data/packages/cloud_controller_ng/da452b34d79be56a0784c7e88d6b9c0e1811a9d8.1-f934c136e6019cf54ab7aa04a3c153657226e729/cloud_controller_ng/lib/cloud_controller/seeds.rb:56:in
`each'\n/var/vcap/data/packages/cloud_controller_ng/da452b34d79be56a0784c7e88d6b9c0e1811a9d8.1-f934c136e6019cf54ab7aa04a3c153657226e729/cloud_controller_ng/lib/cloud_controller/seeds.rb:56:in
`create_seed_domains'\n/var/vcap/data/packages/cloud_controller_ng/da452b34d79be56a0784c7e88d6b9c0e1811a9d8.1-f934c136e6019cf54ab7aa04a3c153657226e729/cloud_controller_ng/lib/cloud_controller/seeds.rb:9:in
`write_seed_data'\n/var/vcap/data/packages/cloud_controller_ng/da452b34d79be56a0784c7e88d6b9c0e1811a9d8.1-f934c136e6019cf54ab7aa04a3c153657226e729/cloud_controller_ng/lib/cloud_controller/runner.rb:93:in
`block in
run!'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/eventmachine-1.0.9.1/lib/eventmachine.rb:193:in
`call'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/eventmachine-1.0.9.1/lib/eventmachine.rb:193:in
`run_machine'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/eventmachine-1.0.9.1/lib/eventmachine.rb:193:in
`run'\n/var/vcap/data/packages/cloud_controller_ng/da452b34d79be56a0784c7e88d6b9c0e1811a9d8.1-f934c136e6019cf54ab7aa04a3c153657226e729/cloud_controller_ng/lib/cloud_controller/runner.rb:87:in
`run!'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/bin/cloud_controller:8:in
`<main>'

Can you check the apps_domain property and see if there is anything
suspicious with it?

Thanks,
Peter & Kara

On Thu, Apr 7, 2016 at 2:32 PM Ranga Rajagopalan <
ranga.rajagopalan(a)gmail.com> wrote:

Hi Peter,

Attaching /var/vcap/sys/log/cloud_controller_worker/cloud_controller_worker.log.gz
and /var/vcap/sys/log/cloud_controller_worker_ctl.log.gz. There isn't a
/var/vcap/sys/log/cloud_controller_worker/ directory on this node.

vcap(a)572afc33-0735-4727-8ff3-9dc6d7fa8af0:~$ ls /var/vcap/sys/log/
agent_ctl.err.log cloud_controller_worker_ctl.log
nginx_cc/
agent_ctl.log consul_agent/
nginx_ctl.err.log
cloud_controller_migration/ metron_agent/
nginx_ctl.log
cloud_controller_migration_ctl.err.log metron_agent_ctl.err.log
route_registrar/
cloud_controller_migration_ctl.log metron_agent_ctl.log
route_registrar_ctl.err.log
cloud_controller_ng/ monit/
route_registrar_ctl.log
cloud_controller_ng_ctl.err.log nfs_mounter/
statsd-injector/
cloud_controller_ng_ctl.log nfs_mounter_ctl.err.log
statsd-injector-ctl.err.log
cloud_controller_worker_ctl.err.log nfs_mounter_ctl.log
statsd-injector-ctl.log



On Thu, Apr 7, 2016 at 12:21 PM, Peter Goetz <peter.gtz(a)gmail.com> wrote:

Hi Ranga,

To trouble-shoot this issue could you also provide the contents of
/var/vcap/sys/log/cloud_controller_ng/cloud_controller_ng.log
and /var/vcap/sys/log/cloud_controller_worker/cloud_controller_worker.log?
This should give us more details about what's going on. The ctl script logs
do not provide enough details.

Thanks,
Peter

On Wed, Apr 6, 2016 at 6:12 PM Ranga Rajagopalan <
ranga.rajagopalan(a)gmail.com> wrote:

I tried v231. Unfortunately, same issue.

--
Thanks,

Ranga

`cf curl /v2/service_instances/#{service_guid}?purge=true -X DELETE` should
delete stuck service records.

`cf service --guid #{service_name}` will get you the service guid to use in
the above command.

Though note that this doesn't fix whatever caused the service to get into
the state that kept it from being deleted, and it probably still exists on
the system providing the service, so that needs to be dealt with
separately. (Or you might continue being charged for that service, for
instance.)

- Natalie

On Thu, Apr 7, 2016 at 6:27 AM Rafal Radecki <radecki.rafal(a)gmail.com>
wrote:

Hi :)

I have several organizations in which I have services in "delete failed"
state. They prevent me from deleting the organization.

What is the best way to remove metadata about such service?

BR,
Rafal.

Hi Peter,

Attaching /var/vcap/sys/log/cloud_controller_worker/cloud_controller_worker.log.gz
and /var/vcap/sys/log/cloud_controller_worker_ctl.log.gz. There isn't a
/var/vcap/sys/log/cloud_controller_worker/ directory on this node.

vcap(a)572afc33-0735-4727-8ff3-9dc6d7fa8af0:~$ ls /var/vcap/sys/log/
agent_ctl.err.log cloud_controller_worker_ctl.log
nginx_cc/
agent_ctl.log consul_agent/
nginx_ctl.err.log
cloud_controller_migration/ metron_agent/
nginx_ctl.log
cloud_controller_migration_ctl.err.log metron_agent_ctl.err.log
route_registrar/
cloud_controller_migration_ctl.log metron_agent_ctl.log
route_registrar_ctl.err.log
cloud_controller_ng/ monit/
route_registrar_ctl.log
cloud_controller_ng_ctl.err.log nfs_mounter/
statsd-injector/
cloud_controller_ng_ctl.log nfs_mounter_ctl.err.log
statsd-injector-ctl.err.log
cloud_controller_worker_ctl.err.log nfs_mounter_ctl.log
statsd-injector-ctl.log

--
Thanks,

Ranga

I'm not seeing graceful shutdown behaviour when I cf stop a Java app. By the looks of Garden SIGTERM is sent and if the process is still alive after 10 seconds SIGKILL is sent: https://github.com/cloudfoundry-incubator/garden-linux/blob/8ab2cbf0696fd0b6bd1a0f15f2d3a463e4acbdac/linux_backend/skeleton/stop.sh#L54-L72

I don't see this in my dead simple app Spring Boot app however: https://github.com/william-tran/cf-shutdown-hook. When I SIGTERM locally it only takes a second to gracefully shutdown.

I'm seeing this on PCF 1.6.8-build.1 (cf 225.3, garden 0.330.0)

Hi Ranga,

To trouble-shoot this issue could you also provide the contents of
/var/vcap/sys/log/cloud_controller_ng/cloud_controller_ng.log
and /var/vcap/sys/log/cloud_controller_worker/cloud_controller_worker.log?
This should give us more details about what's going on. The ctl script logs
do not provide enough details.

Thanks,
Peter

On Wed, Apr 6, 2016 at 6:12 PM Ranga Rajagopalan <
ranga.rajagopalan(a)gmail.com> wrote:

I tried v231. Unfortunately, same issue.

Hi, Fabien,

If you're running the buildpack staging tasks on Diego cells, Jim is
correct that the executor's log-streaming system currently imposes a
hard-coded 4-KiB limit on each log line. The Diego team has a story (
https://www.pivotaltracker.com/story/show/111866195) to increase that to 60
KiB that we expect to get to in a few weeks. The original limit was
introduced very early in the life of the executor component, and was likely
based on some too-restrictive assumptions about the maximum size of UDP
datagrams. We've since validated that on Linux and on Windows these
datagrams can contain a little less than 64KiB of data in their payload, so
the 60-KiB limit should be fine.

Thanks,
Eric, CF Runtime Diego PM

On Thu, Apr 7, 2016 at 10:12 AM, Jim CF Campbell <jcampbell(a)pivotal.io>
wrote:

Hi Fabien,

It's a restriction that the Executor component of Diego implements. It's
not in the Loggregator code. I'm not sure of the where or why of it.

Jim


On Thu, Apr 7, 2016 at 6:44 AM, Fabien Guichard <
fabien.guichard(a)orange.com> wrote:

Hello,

while developping an extension ofr a buildpack, related to logging
event from applications, we found out that STDOUT logs where being flushed
with a fixed size (because we're receving troncated strings in our external
aggregating system).

it seems to us that the buffer size is around 4000 caracters, which is
not what we would like.

We went through doppler metron and loggregator configurations & codes,
without success, to find out anything related to fixed-size logging
property.

Do ou know where we should go to configure the size of the buffer being
flushed please ?

Many thanks for the time you took to read our very first question on this
ML.

Regards,
Fabien.

--
Jim Campbell | Product Manager | Cloud Foundry | Pivotal.io | 303.618.0963

Hi Fabien,

It's a restriction that the Executor component of Diego implements. It's
not in the Loggregator code. I'm not sure of the where or why of it.

Jim


On Thu, Apr 7, 2016 at 6:44 AM, Fabien Guichard <fabien.guichard(a)orange.com>
wrote:

Hello,

while developping an extension ofr a buildpack, related to logging
event from applications, we found out that STDOUT logs where being flushed
with a fixed size (because we're receving troncated strings in our external
aggregating system).

it seems to us that the buffer size is around 4000 caracters, which is not
what we would like.

We went through doppler metron and loggregator configurations & codes,
without success, to find out anything related to fixed-size logging
property.

Do ou know where we should go to configure the size of the buffer being
flushed please ?

Many thanks for the time you took to read our very first question on this
ML.

Regards,
Fabien.

--
Jim Campbell | Product Manager | Cloud Foundry | Pivotal.io | 303.618.0963

Hi all,

Sorry for the confusion, I originally added some extraneous docs to the
Required Reading list that people have been trying and failing to access.
Those links have been removed, the most up-to-date information (about the
general Elastic Clusters feature narrative) is in what's now the first
link, titled "Cloud Foundry Elastic Clusters".

Cheers,
Amit

We customized our 404 mapping wildcard routes to an app. We also display a
503 if the route exists.

Mike

When we accessing an APP which is not up or not existed, the CF will display error message like:

404 Not Found: Requested route ('test.domain.name') does not exist.

Can we customize it, and how can we do that?


Thanks in advance.

Did you get a chance to review my previous email?



Let me know your thoughts!



~David



From: David [mailto:David(a)lead2tech.com]
Sent: Tuesday, April 05, 2016 8:42 AM
To: 'cf-dev(a)lists.cloudfoundry.org'
Subject: OpenStack Leads





Hi,



I hope this email finds you well.



As your organization is a part of " OpenStack Summit 2016 " I'm writing to
check if you would be interested in a leads of OpenStack users for marketing
campaign and lead generation initiative.



You may choose job titles such as: IT Architects, CTOs, IT Directors, Data
Centre Managers, Developers and more.



Each lead will be delivered to you with multi-channel data fields such as
Company name, Web address, Contact Name, Job Title, Phone Number, Direct
Business Emails, Postal address, Industry, employee size, Revenue size, SIC
code, Fax number, Application/Technology usage etc.



Appreciate your time and look forward to hearing from you.



Regards,



David Reid| Lead Generation Specialist
Lead2Tech LLC| Technology Division

If you do not wish to receive any more emails please reply to this email
with Leave out in the subject line.











<http://1816396.sigclick.mailinfo.com/sigclick/09000306/0000054D/0E0B0607/04
450118820393.jpg>

Anyone know about automated deployment of window cell for diego? I've done it manually a few months back and it was very manual and error prone. Does anyone have anything that automate process?

We built the cf-haproxy-boshrelease for this (https://github.com/cloudfoundry-community/cf-haproxy-boshrelease). Check out https://blog.starkandwayne.com/2015/06/17/securing-private-domains-on-a-public-cloudfoundry/ for more details. The feature-set has been updated quite a bit since that post. It now supports enough features that should allow it to work behind tcp or http-mode load balancers (you can force x-forwarded-proto if behind a tcp-only loadbalancer that does ssl termination), handle websockets on standard http ports, serve multiple ssl certs, and do generic tcp proxying for things like app-ssh.

My ideal setup would be a set of http-mode load balancers in front of the 'public_haproxy_z*' nodes, which would filter out non-allowed domains, and send the remaining requests to the gorouters. For internal-only requests, I'd use a non-public http-mode load balancer directly in front of the gorouters.

Hi,
does anybody know, why the routers got static ips in the cf-infrastructure-aws.yml file? https://github.com/cloudfoundry/cf-release/blob/master/templates/cf-infrastructure-aws.yml#L173 <https://github.com/cloudfoundry/cf-release/blob/master/templates/cf-infrastructure-aws.yml#L173>

Bosh is assigning the instances to ELB’s during deploy time, so there should be no need to have static addresses here.

If nobody know’s a good reason should we remove them ;-)

Cheers
Johannes

Hi :)

I have several organizations in which I have services in "delete failed" state. They prevent me from deleting the organization.

What is the best way to remove metadata about such service?

BR,
Rafal.

Hello,

while developping an extension ofr a buildpack, related to logging event from applications, we found out that STDOUT logs where being flushed with a fixed size (because we're receving troncated strings in our external aggregating system).

it seems to us that the buffer size is around 4000 caracters, which is not what we would like.

We went through doppler metron and loggregator configurations & codes, without success, to find out anything related to fixed-size logging property.

Do ou know where we should go to configure the size of the buffer being flushed please ?

Many thanks for the time you took to read our very first question on this ML.

Regards,
Fabien.

Container networking is aimed at your requirement. Once it's ready, you
would deploy your apps without external routes and rely on the private
network for communication.

We expect we'll have facilities in place to find apps via a DNS
implementation using app names or services names but we''re beginning to
move in a direction where apps are associated with explicit service names
that are orthogonal to the app name. We may also adopt a VIP pattern for
services to remove the need name resolution entirely.

For status, we have containers talking via n overlay with guardian and
we're in the process of integrating with diego. We hope to demonstrate what
we have at summit. It will be a while before things are "baked" but the
project exists and we're moving forward.

Finally, there are plans to implement several policy controls around
endpoint communication that enable users to express a supported
connectivity graph without relying on apps living in the same space for
communication.

Hope that helps. Let us know if you have more questions.

On Thu, Apr 7, 2016 at 1:28 AM, Krannich, Bernd <bernd.krannich(a)sap.com>
wrote:

Hi all,

Dr. Nic’s blog post [1] addresses a requirement that we keep hearing
frequently: To have certain applications that can only be accessed from
other apps running in the same Cloud Foundry deployment.

I wanted to check with the community here to see if anybody has come
across this requirement as well. Is this something one should actually be
advocating in terms of architecture?

Also, the solution suggested in [1] in our opinion is prone to bypassing
if implemented in the described way (see top-most comment in the same blog
post). Of course, one could work around that by terminating SSL at the two
load balancers, introspecting the header fields there. On the other hand,
this requires a much more complex “internal/external” DNS setup.

A logical next step would be to extend the requirement to only allow calls
from apps being deployed into certain orgs and spaces. This can no longer
be achieved by a setup “external” to CF given that apps from all types of
orgs and spaces are usually deployed on the very same runners with calls
being “indistinguishable” once a call has left the runner.

As alternatives, we came up with the following potential options:

1. Avoiding the notion of “private” services altogether, making sure that
each application is secured by - for example - a JWT token carrying a
shared secret.
2. Using messaging via a backing service instead of HTTP-based
communication.
3. Recently suggested/implemented CF functionality like “container
networking for applications” [2] (what’s the status on this one, actually?)
and “Context Path Routing” [3] (this was discussed on the mailing list some
time back, even though I’m not sure how you would actually use it to solve
the above scenario). Is any of that allowing to implement the above
requirement?

Any thoughts, opinions, suggestions are welcome.

Thanks in advance,
Bernd

[1]
https://blog.starkandwayne.com/2014/10/31/public-and-private-microservices-on-the-same-cloud-foundry
[2]
https://docs.google.com/document/d/1zQJqIEk4ldHH5iE5zat_oKIK8Ejogkgd_lySpg_oV_s/edit?usp=sharing
[3]
http://berlin2015.cfsummit.com/sites/berlin2015.cfsummit.com/files//pages/files/summit-berlin.pdf

Bernd Krannich
Products & Innovation Technology
SAP SE
Dietmar-Hopp-Allee 16
69190 Walldorf Germany
mailto:bernd.krannich(a)sap.com
www.sap.com

Pflichtangaben/Mandatory Disclosure Statements:
http://www.sap.com/company/legal/impressum.epx

Diese E-Mail kann Betriebs- oder Geschäftsgeheimnisse oder sonstige
vertrauliche Informationen enthalten. Sollten Sie diese E-Mail irrtümlich
erhalten haben, ist Ihnen eine Kenntnisnahme des Inhalts, eine
Vervielfältigung oder Weitergabe der E-Mail ausdrücklich untersagt. Bitte
benachrichtigen Sie uns und vernichten Sie die empfangene E-Mail. Vielen
Dank.

This e-mail may contain trade secrets or privileged, undisclosed, or
otherwise confidential information. If you have received this e-mail in
error, you are hereby notified that any review, copying, or distribution of
it is strictly prohibited. Please inform us immediately and destroy the
original transmittal. Thank you for your cooperation.

--
Matthew Sykes
matthew.sykes(a)gmail.com

Hi,

Is there connection limitation on HAproxy server? We had encountered a connection issue recently. HAproxy server failed to accept new connection request after connection number reached 2000+. Is there a way to change the connection limitation?

The CF version is 197 and below is the content of haproxy server configuration file.

global
log 127.0.0.1 syslog info
daemon
user vcap
group vcap
maxconn 64000
spread-checks 4

defaults
log global
timeout connect 30000ms

frontend http-in
mode http
bind :80
option httplog
option forwardfor
reqadd X-Forwarded-Proto:\ http
default_backend http-routers



frontend https-in
mode http
bind :443 ssl crt /var/vcap/jobs/haproxy/config/cert.pem no-sslv3 ciphers
option httplog
option forwardfor
option http-server-close
reqadd X-Forwarded-Proto:\ https
default_backend http-routers

frontend ssl-in
mode tcp
bind :4443 ssl crt /var/vcap/jobs/haproxy/config/cert.pem no-sslv3 ciphers
default_backend tcp-routers


backend http-routers
mode http
balance roundrobin

Thanks,
Maggie

Hurray I'll be in a useful timezone and can join!
"Living in Australia" and "CAB calls" are mutually exclusive :)
Nic

Hi all,

Dr. Nic’s blog post [1] addresses a requirement that we keep hearing frequently: To have certain applications that can only be accessed from other apps running in the same Cloud Foundry deployment.

I wanted to check with the community here to see if anybody has come across this requirement as well. Is this something one should actually be advocating in terms of architecture?

Also, the solution suggested in [1] in our opinion is prone to bypassing if implemented in the described way (see top-most comment in the same blog post). Of course, one could work around that by terminating SSL at the two load balancers, introspecting the header fields there. On the other hand, this requires a much more complex “internal/external” DNS setup.

A logical next step would be to extend the requirement to only allow calls from apps being deployed into certain orgs and spaces. This can no longer be achieved by a setup “external” to CF given that apps from all types of orgs and spaces are usually deployed on the very same runners with calls being “indistinguishable” once a call has left the runner.

As alternatives, we came up with the following potential options:

1. Avoiding the notion of “private” services altogether, making sure that each application is secured by - for example - a JWT token carrying a shared secret.
2. Using messaging via a backing service instead of HTTP-based communication.
3. Recently suggested/implemented CF functionality like “container networking for applications” [2] (what’s the status on this one, actually?) and “Context Path Routing” [3] (this was discussed on the mailing list some time back, even though I’m not sure how you would actually use it to solve the above scenario). Is any of that allowing to implement the above requirement?

Any thoughts, opinions, suggestions are welcome.

Thanks in advance,
Bernd

[1] https://blog.starkandwayne.com/2014/10/31/public-and-private-microservices-on-the-same-cloud-foundry
[2] https://docs.google.com/document/d/1zQJqIEk4ldHH5iE5zat_oKIK8Ejogkgd_lySpg_oV_s/edit?usp=sharing
[3] http://berlin2015.cfsummit.com/sites/berlin2015.cfsummit.com/files//pages/files/summit-berlin.pdf

Bernd Krannich
Products & Innovation Technology
SAP SE
Dietmar-Hopp-Allee 16
69190 Walldorf Germany
mailto:bernd.krannich(a)sap.com
www.sap.com

Pflichtangaben/Mandatory Disclosure Statements: http://www.sap.com/company/legal/impressum.epx

Diese E-Mail kann Betriebs- oder Geschäftsgeheimnisse oder sonstige vertrauliche Informationen enthalten. Sollten Sie diese E-Mail irrtümlich erhalten haben, ist Ihnen eine Kenntnisnahme des Inhalts, eine Vervielfältigung oder Weitergabe der E-Mail ausdrücklich untersagt. Bitte benachrichtigen Sie uns und vernichten Sie die empfangene E-Mail. Vielen Dank.

This e-mail may contain trade secrets or privileged, undisclosed, or otherwise confidential information. If you have received this e-mail in error, you are hereby notified that any review, copying, or distribution of it is strictly prohibited. Please inform us immediately and destroy the original transmittal. Thank you for your cooperation.