Date   

Re: Identifying all DEA's on a CF install

Matt Curry
 

I believe you could use the BOSH API:

https://bosh.io/docs/director-api-v1.html#list-vms-detailed

From: john mcteague <john.mcteague(a)gmail.com<mailto:john.mcteague(a)gmail.com>>
Reply-To: "Discussions about Cloud Foundry projects and the system overall." <cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org>>
Date: Monday, December 7, 2015 at 3:29 PM
To: "Discussions about Cloud Foundry projects and the system overall." <cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org>>
Subject: [cf-dev] Re: Re: Identifying all DEA's on a CF install

I hadnt considered that. I see that there are now docs for the BOSH api, however is the BOSH api suited to concurrent access by multiple processes? We are writing a component that will need to frequently validate that a particular machine is a DEA.

On Mon, Dec 7, 2015 at 10:26 PM, Amit Gupta <agupta(a)pivotal.io<mailto:agupta(a)pivotal.io>> wrote:
What about with BOSH?

On Mon, Dec 7, 2015 at 2:17 PM, john mcteague <john.mcteague(a)gmail.com<mailto:john.mcteague(a)gmail.com>> wrote:
I am trying to enumerate all the ways I could identify all the DEA's in a CF cluster in an IAAS agnostic way (e.g. not querying Openstack to read metadata on the VM's that list the job type).

I think the only reliable way is to listen on NATS and look for the DEA advertisement messages. Am I correct or is there a method I am missing (I'd rather call and API rather than subscribe to NATS)?

Thanks,
John


Re: Identifying all DEA's on a CF install

john mcteague <john.mcteague@...>
 

I hadnt considered that. I see that there are now docs for the BOSH api,
however is the BOSH api suited to concurrent access by multiple processes?
We are writing a component that will need to frequently validate that a
particular machine is a DEA.

On Mon, Dec 7, 2015 at 10:26 PM, Amit Gupta <agupta(a)pivotal.io> wrote:

What about with BOSH?

On Mon, Dec 7, 2015 at 2:17 PM, john mcteague <john.mcteague(a)gmail.com>
wrote:

I am trying to enumerate all the ways I could identify all the DEA's in a
CF cluster in an IAAS agnostic way (e.g. not querying Openstack to read
metadata on the VM's that list the job type).

I think the only reliable way is to listen on NATS and look for the DEA
advertisement messages. Am I correct or is there a method I am missing (I'd
rather call and API rather than subscribe to NATS)?

Thanks,
John


Re: Identifying all DEA's on a CF install

Amit Kumar Gupta
 

What about with BOSH?

On Mon, Dec 7, 2015 at 2:17 PM, john mcteague <john.mcteague(a)gmail.com>
wrote:

I am trying to enumerate all the ways I could identify all the DEA's in a
CF cluster in an IAAS agnostic way (e.g. not querying Openstack to read
metadata on the VM's that list the job type).

I think the only reliable way is to listen on NATS and look for the DEA
advertisement messages. Am I correct or is there a method I am missing (I'd
rather call and API rather than subscribe to NATS)?

Thanks,
John


Re: v226 release notes

Amit Kumar Gupta
 

Hi John,

Sorry for the delay. The notes are in draft mode until all sections have
been populated.

I will ask the respective Product Managers for the remaining section to add
their notes.

Thanks,
Amit

On Mon, Dec 7, 2015 at 2:19 PM, john mcteague <john.mcteague(a)gmail.com>
wrote:

v226 was cut 4 days ago, is there any sign of the release notes?

Thanks,
John


v226 release notes

john mcteague <john.mcteague@...>
 

v226 was cut 4 days ago, is there any sign of the release notes?

Thanks,
John


Identifying all DEA's on a CF install

john mcteague <john.mcteague@...>
 

I am trying to enumerate all the ways I could identify all the DEA's in a
CF cluster in an IAAS agnostic way (e.g. not querying Openstack to read
metadata on the VM's that list the job type).

I think the only reliable way is to listen on NATS and look for the DEA
advertisement messages. Am I correct or is there a method I am missing (I'd
rather call and API rather than subscribe to NATS)?

Thanks,
John


Re: - Reducing number of instances on a cloud foundry deployment

Amit Kumar Gupta
 

Hey Kinjal,

Your best bet, if you choose to use cf-boshworkspace, would be to open up
issues against their GitHub repo. The maintainers there will be able to
tell you if their stuff is up-to-date and expected to work with v222, what
sort of properties you need to put in your stub, etc. For starters, it
looks like you're missing values for "bulk_api_password",
"staging_upload_user", "db_encryption_key", and "staging_upload_password"
under the "cc" section of "properties".

Amit

On Fri, Dec 4, 2015 at 7:43 AM, Kinjal Doshi <kindoshi(a)gmail.com> wrote:

Thanks a lot for directing me towards this project. When I am trying to
deploy CF using this project, there are some issues as below:


1. 'bosh prepare deployment' fails because of checksum failure on
package buildpack_python
2. Alternatively I tried to upload release as directed in the 'Read
Me' of this project but there is a version mismatch, the read me suggests
uploading version 196 whereas the deployment manifest mentions version 222.
3. There are also some spiff merge issue as I get the below error on
using 'bosh deploy'

I have been able to resolve errors 1 and 2 but need help for point 3 as am
not well versed with spiff. Kindly help me resolve this error:

ubuntu(a)ip-172-31-38-159:~/cf-boshworkspace$ bosh deploy
Generating deployment manifest
Command failed: 'spiff merge
/home/ubuntu/cf-boshworkspace/templates/cf/cf-deployment.yml
/home/ubuntu/cf-boshworkspace/templates/cf/cf-resource-pools.yml
/home/ubuntu/cf-boshworkspace/templates/tiny/cf-tiny-scalable.yml
/home/ubuntu/cf-boshworkspace/templates/cf-uaa-port.yml
/home/ubuntu/cf-boshworkspace/templates/cf-allow-services-access.yml
/home/ubuntu/cf-boshworkspace/templates/cf/cf-properties.yml
/home/ubuntu/cf-boshworkspace/templates/cf/cf-infrastructure-aws.yml
/home/ubuntu/cf-boshworkspace/templates/cf-properties.yml
/home/ubuntu/cf-boshworkspace/templates/tiny/cf-jobs-nfs.yml
/home/ubuntu/cf-boshworkspace/templates/tiny/cf-jobs-uaa.yml
/home/ubuntu/cf-boshworkspace/templates/tiny/cf-jobs-base.yml
/home/ubuntu/cf-boshworkspace/templates/parallel.yml
/home/ubuntu/cf-boshworkspace/templates/cf-no-ssl.yml
/home/ubuntu/cf-boshworkspace/templates/cf-secrets.yml
/home/ubuntu/cf-boshworkspace/templates/tiny/cf-use-nfs.yml
/home/ubuntu/cf-boshworkspace/templates/tiny/cf-use-haproxy.yml
/home/ubuntu/cf-boshworkspace/templates/cf-networking.yml
/home/ubuntu/cf-boshworkspace/.stubs/cf-aws-tiny.yml 2>&1'
2015/12/04 14:02:53 error generating manifest: unresolved nodes:
(( merge )) in
/home/ubuntu/cf-boshworkspace/templates/cf/cf-properties.yml
properties.cc.staging_upload_user
(( merge )) in
/home/ubuntu/cf-boshworkspace/templates/cf/cf-properties.yml
properties.cc.staging_upload_password
(( bulk_api_password )) in dynaml
properties.cc.internal_api_password
(( merge )) in
/home/ubuntu/cf-boshworkspace/templates/cf/cf-properties.yml
properties.cc.db_encryption_key
(( merge )) in
/home/ubuntu/cf-boshworkspace/templates/cf/cf-properties.yml
properties.cc.bulk_api_password

Thanks in Advance,
Kinjal


On Thu, Dec 3, 2015 at 12:37 AM, Amit Gupta <agupta(a)pivotal.io> wrote:

The cf-boshworkspace project from Stark and Wayne:
https://github.com/cloudfoundry-community/cf-boshworkspace provides an
cf-aws-tiny.yml template, which I believe gets it down to 4 instances.

In principle, everything should be able to run on 1 VM, but my
understanding is there are port collisions that aren't configurable (yet),
and so 4 was the lower bound S&W found.

On Wed, Dec 2, 2015 at 4:33 AM, Kinjal Doshi <kindoshi(a)gmail.com> wrote:

Hi,

I am looking at miminal-aws.yml in the release
https://github.com/cloudfoundry/cf-release and realized that this
deployment will consume 13 instances on AWS.

Is it possible to reduce the number of instances for this deployment?

Thanks a lot for the help in advance.

Regards,
Kinjal


Re: doppler issue which fails to emit logs with syslog protocol on CFv212

Amit Kumar Gupta
 

Hey Masumi,

Glad you've gotten further, thanks for the update.

Best,
Amit

On Mon, Dec 7, 2015 at 6:20 AM, Masumi Ito <msmi10f(a)gmail.com> wrote:

Hi Amit,

This could be due to several reasons -- network issues, etcd slowness,
etcd
down, etc.

All right. I might ask questions again if the error messages are frequently
encountered however this is a separated issue and not highly prioritized so
far . So I am going to close this thread because the original issue has
been
resolved thanks to you. Thanks for your help.

Regards,
Masumi



--
View this message in context:
http://cf-dev.70369.x6.nabble.com/doppler-issue-which-fails-to-emit-logs-with-syslog-protocol-on-CFv212-tp2418p2979.html
Sent from the CF Dev mailing list archive at Nabble.com.


Bosh version and stemcell for 225

Mike Youngstrom
 

We are preparing to release 225 and noticed the release notes don't list a
bosh and stemcell version. Does anyone have that info?

Mike


Re: [cf-env] [abacus] Changing how resources are organized

Jean-Sebastien Delfino
 

Hi Daniel,

Is that related to Github issue #38?

https://github.com/cloudfoundry-incubator/cf-abacus/issues/38

Thanks

- Jean-Sebastien

On Fri, Dec 4, 2015 at 11:22 AM, dmangin <dmangin(a)us.ibm.com> wrote:

With the current way we have resource_ids being used for metering,
accumulation, and aggregation, we have to have a resource definition for
every resource_id that people are using. Then when we have custom
buildpacks, we will have to start creating a resource definition for each
one of them, inflating the amount of the resource definitions that we have.
So we will be adding a new field called resource_type_id ontop of the
resource_id that will be the resource definition abacus uses, allowing
custom buildpacks to fall under the resource type. So we were thinking of
having the hierarchy changed so resource_id is underneath resource_type_id.

When we implement this, we will have to change abacus to use the
resource_type_id rather than the resource_id to do the calculations. What
changes will this cause in abacus right now and how will this affect the
previous reports abacus has created?

Reagrds,
Daniel Mangin



--
View this message in context:
http://cf-dev.70369.x6.nabble.com/cf-env-abacus-Changing-how-resources-are-organized-tp2971.html
Sent from the CF Dev mailing list archive at Nabble.com.


- Urgent - Cloud Foundry Deployment is failing on dea.yml.erb

Kinjal Doshi
 

Hi,

I am trying to deploy cloud foundry with the
stemcell light-bosh-stemcell-3147-aws-xen-hvm-ubuntu-trusty-go_agent.tgz
and cloud foundry release manifest cf-226yml

I am also using the minimal-aws.yml for configuration data.

During 'bosh deploy' command, I run into the following deployment error:

Started preparing deployment
Started preparing deployment > Binding releases. Done (00:00:00)
Started preparing deployment > Binding existing deployment. Done
(00:00:01)
Started preparing deployment > Binding resource pools. Done (00:00:00)
Started preparing deployment > Binding stemcells. Done (00:00:00)
Started preparing deployment > Binding templates. Done (00:00:00)
Started preparing deployment > Binding properties. Done (00:00:00)
Started preparing deployment > Binding unallocated VMs. Done (00:00:00)
Started preparing deployment > Binding instance networks. Done (00:00:00)

Started preparing package compilation > Finding packages to compile. Done
(00:00:00)

Started preparing dns > Binding DNS. Done (00:00:00)

Started preparing configuration > Binding configuration. Failed: Error
filling in template `dea.yml.erb' for `runner_z1/0' (line 86: bad
component(expected user component): Oro(a)1602) (00:00:01)

Error 100: Error filling in template `dea.yml.erb' for `runner_z1/0' (line
86: bad component(expected user component): Oro(a)1602)

I noticed that the property cc.internal_api_user is missing from the global
properties and have added the same to minimal-aws.yml but the deployment
still fails.

I need to have the CF deployment up and running tonight. Would be great if
some one can please help me with this on priority?

Regards,
Kinjal


How to estimate reconnection / failover time between gorouter and nats

Masumi Ito
 

Hi,

Can anyone explain about the expected reconnection / failover time for
gorouter when one of the nats VMs hangs up accidentally?

The background of this question is that I found the gorouter had some
timeframe to return "404 Not found Err" for app requests temporarily when
one of the clusted nats was not responsive. This happened after about 2 min
and then recovered in another 2-3min. I understand it is mainly due to
pruning stale routes and reconnection / failover time to a healthy nats by
gorouter. First 2 min can be explained as droplet_stale_threshold value.
However I am wondering if what exactly happened in another 2-3min.

Note that bosh health monitor detected an unresponsive nats and recreated it
finally however the gorouter had received "router.register" from DEAs before
the recreation was complete. Therefore I think this indicates the failover
to the other nats rather than reconnecting to the recreated nats which was
previously down.

I believe some connection parameters in the yagnats and apcera/nats client
are keys for this.

- Timeout: timeout to create a new connection
- ReconnectWait: wait time before reconnect happens
- MaxReconnect: unlimited reconnect times if this value is -1
- PingInterval: interval of each pinging to check if a connection is healthy
- MaxPingOut: trial times of pinging before determining reconnection is
necessary

1. When one of nats hangs up, the connection might still exist until TCP
timeout has been reached.

2. PingTimer periodically sends ping to check if the connection is stale
totally (PingInterval * MaxPingOut) times and concluds it is necessary to
reconnect to the next nats server.

3. Before reconecting it, the gorouter waits in ReconnectWait.

4. Create a new connection for the next nats server within Timeout.

5. After that, the gorouter starts to register app routes from DEAs through
the nats connected.

Therefore my rough estimation is:
PingInterval(2 min) * MaxPingOut(2) + ReconnectWait(500 millisec) +
Timeout(2 sec)

I would appreciate if someone could correct this rough explanation or give
some more details.

Regards,
Masumi



--
View this message in context: http://cf-dev.70369.x6.nabble.com/How-to-estimate-reconnection-failover-time-between-gorouter-and-nats-tp2980.html
Sent from the CF Dev mailing list archive at Nabble.com.


Re: doppler issue which fails to emit logs with syslog protocol on CFv212

Masumi Ito
 

Hi Amit,

This could be due to several reasons -- network issues, etcd slowness, etcd
down, etc.

All right. I might ask questions again if the error messages are frequently
encountered however this is a separated issue and not highly prioritized so
far . So I am going to close this thread because the original issue has been
resolved thanks to you. Thanks for your help.

Regards,
Masumi



--
View this message in context: http://cf-dev.70369.x6.nabble.com/doppler-issue-which-fails-to-emit-logs-with-syslog-protocol-on-CFv212-tp2418p2979.html
Sent from the CF Dev mailing list archive at Nabble.com.


Re: Cloud Foundry Elastic Clusters - Proposal

Dieu Cao <dcao@...>
 

Hi All,

I've updated the Elastic Clusters proposal [1] with more detail based on
additional discussions we've had among CAPI, Routing, Diego, Loggregator,
and UAA.

Please have a look at this updated version and use the google doc to
provide feedback and ideas.

There are still a few open questions but I think the proposal is much more
fleshed out now and based on feedback we can hopefully get started on the
work soon.

-Dieu
CF CAPI PM / CF Runtime PMC Lead

[1]
https://docs.google.com/document/d/1BZL4GwluaRtUVfeW-j8_ezBJQmwnhGqcMNvQIy9ltZE/edit?usp=sharing

On Fri, Oct 2, 2015 at 7:19 PM, Mark Kropf <markkropf(a)gmail.com> wrote:

Hello CF Community,

I would like to share a proposal that I have been thinking through with a
couple of my colleagues here at Pivotal. Attached below, you'll find a
high level view of a feature I'm referring to you as Elastic Clusters.
Please use the google doc as a place to provide feedback and ideas for this
feature.


https://docs.google.com/document/d/1BZL4GwluaRtUVfeW-j8_ezBJQmwnhGqcMNvQIy9ltZE/edit?usp=sharing

Thank you,

Mark Kropf


CF CAB call for December is this Wednesday Dec. 9th, 2015 @ 8a PDT

Michael Maximilien
 

Hi, all, 您好 Nín hǎo,

Quick reminder that the last CAB call for 2015 is this week Wednesday December 9th @ 8a PDT. I'll be calling from Beijing where it will be 1a, so I plan to start and end on time :)

Product managers, please add project updates to Agenda here:

https://docs.google.com/document/d/1SCOlAquyUmNM-AQnekCOXiwhLs6gveTxAcduvDcW_xI/edit#heading=h.o44xhgvum2we

Everyone, if you have something to share, please also add an entry at the end in Other section.

Best,

Chip, James, and Max

PS: call info and details are in the link above; just visit it

dr.max
ibm cloud labs
sillicon valley, ca

Sent from my iPhone


Replacing nfs with WebDAV

Dieu Cao <dcao@...>
 

Hello All,

The CAPI team has been investigating replacing the nfs as the default
blobstore option in cf-release with WebDAV.

There are a number of reasons for doing this.
The flakiness of the nfs_mounter job.
The requirement for rpc-bind and nfs-common packages in the bosh stemcell
and thus open ports on jobs that didn't need them [1]
There is a feature on the bosh backlog [2] to have the agent poll process
state on "STOP" messages which is blocked by cloud controller's use of nfs
because cc would get stuck if the nfs job should change IP.

We successfully completed a spike to investigate if we could replace nfs in
a backwards compatible way without requiring migration of blobs with webdav
building on some work that the bosh team did.

We'll be working on the bosh packaging for webdav and making sure that we
can provide instructions to make this change as seamless as possible and
hopefully we'll have this available "soon".

-Dieu
CF CAPI PM

[1] https://www.pivotaltracker.com/story/show/88810548
[2] https://www.pivotaltracker.com/story/show/62354622


Re: [abacus] Refactor Aggregated Usage and Aggregated Rated Usage data model

Saravanakumar A. Srinivasan
 

c) a middle-ground approach where we'll store the aggregated usage per app in separate docs, but maintain the aggregated usage at the upper levels (org, space, resource, plan) in the parent doc linking the app usage docs together, and explore what constrains or limitations that would impose on our ability to trigger real time usage limit alerts at any org, space, resource, plan, app etc level.
As a first step (refer to [1] for more details) to refactor the usage data model using middle-ground approach, we have removed Usage Rating Service from Abacus pipeline (refer to commit at [2]) and moved entire rating implementation from Usage Rating Service to Usage Aggregator (refer to commit at [3])

With these commits, If you are using Abacus, be aware that the Abacus pipeline has become shorter and you have one less application (Usage Rating Service) to manage.

[1] https://github.com/cloudfoundry-incubator/cf-abacus/issues/184
[2] https://github.com/cloudfoundry-incubator/cf-abacus/commit/1488e1ae2e4547a010151ad2245f3a3f1ff2e488
[3] https://github.com/cloudfoundry-incubator/cf-abacus/commit/c661b7bdd35e70e985583570cb9920b90ced44a8


Re: Dev and Production environment inconsistent

CF Runtime
 

Hi Juan,

That is correct. The "production" flag on apps in the CC should not be
used. Instead you should push apps to different spaces with separate
domains associated to each app/space to create a staging/production pattern
as you described.

Best,
Zak Auerbach, CF Release Integration

On Fri, Nov 27, 2015 at 4:31 AM, Juan Antonio Breña Moral <
bren(a)juanantonio.info> wrote:

Hi Alex,

when I get the summary from an app:
http://apidocs.cloudfoundry.org/214/apps/get_app_summary.html

I see a deprecated element: (example: production: false) so the best
practice is the development of virtual environments using spaces?

Example:

+ CERT
+ PRE
+ PRO

Is the idea with spaces?

Juan Antonio


Re: Passwords visible in infrastructure logs

Amit Kumar Gupta
 

Hey Momchil,

Do you know whether it's the DEA or Warden that's logging that sensitive
data when you say "runner"?

I would recommended opening issues against the relevant projects:

API: https://github.com/cloudfoundry/cloud_controller_ng/issues
DEA or Warden: https://github.com/cloudfoundry/dea_ng/issues or
https://github.com/cloudfoundry/warden/issues

As for NATS, you may be able to change the logging level? Alternatively,
NATS is not a Cloud Foundry project but you could ask over there about
encrypting log output: https://github.com/nats-io/gnatsd

In Pivotal's production environments, we run 100% on Diego, so we are not
concerned with DEA/Warden logging, and this move also removes NATS from the
flows like create-user-provided-service. CC is likely still an issue, so
it would be a good one to raise against their GitHub project.

Best,
Amit

On Fri, Dec 4, 2015 at 2:09 AM, Momchil Atanassov <momchil.atanassov(a)sap.com
wrote:
Hi,

We are using the `syslog_deamon_config` proprty to stream all of our CF
infrastructure logs to an external stack for later processing.

We have noticed that operations like `cf create-user-provided-service`,
`cf bind-service`, and others are logged by multiple components in CF. That
would normally not be a problem, except that these commands often involve
passwords and those passwords get logged as well, ending up in the log
files on the VM and the target log processing stack, which allows operators
of the system to view end-user passwords.

We have noticed that the following jobs are responsible for the logs:

* api
* runner
* nats

Increasing the log level from the default `debug` / `debug2` to `info`
solves the problem for the first two, at the cost of making troubleshooting
tasks more difficult on the system.
The last one can only be solved by removing the `nats_stream_forwarder`
component from the `nats` job, again making troubleshooting more difficult.

I believe the ideal solution is to have those components not log the
payload of commands holding confidential information. Maybe they could
replace it with some pattern.
This would help for the first two but might not help for nats, where some
other means would be needed (encryption of the private content?).

How are you solving this issue on your productive system? What are your
thoughts on this matter?

Thanks in advance!

Regards,
Momchil Atanassov


Re: Garden Port Assignment Story

Mike Youngstrom
 

Thanks for the update Will. I'll keep waiting patiently. :)

On Fri, Dec 4, 2015 at 10:44 AM, Will Pragnell <wpragnell(a)pivotal.io> wrote:

Hi Mike,

Just to let you know we've not forgotten this. We're giving the matter
some thought, and we'll report back here with a proposal once we've figured
out some of the fiddly details.

Thanks,
Will

On 24 November 2015 at 15:19, Mike Youngstrom <youngm(a)gmail.com> wrote:

Yes Will, that summary is essentially correct. But, for even more
clarity let me restate the complete story again and reason I want 92085170
to work across stemcell upgrades. :)

Today if NATS goes down after 2 minutes the routers will drop their
routing tables and my entire CF deployment goes down. The routers behave
this way because of an experience Dieu had [0]. I don't like this I would
prefer for routers to not drop routing tables if it cannot connect to
Nats. Therefore, the routing team is adding 'prune_on_config_unavailable'.
I plan to set this to false to make my deployment less sensitive to NATS
failure. In doing so I am incurring more risk of mis routed stale routes.
I am hoping that 92085170 will help reduce some of that risk. Since one of
the times I personally have experienced stale route routing was during a
deploy I hope that Garden will consider a port selection technique that
will help ensure uniqueness across stemcell upgrades, something we
frequently do as part of a deploy.

Consequently a stateless solution like random assignment or a consistent
hash will work across stemcell upgrades.

Thanks,
Mike

[0]
https://groups.google.com/a/cloudfoundry.org/d/msg/vcap-dev/yuVYCZkMLG8/7t8FHnFzWEsJ

On Tue, Nov 24, 2015 at 3:44 AM, Will Pragnell <wpragnell(a)pivotal.io>
wrote:

Hi Mike,

What I think you're saying is that once the new
`prune_on_config_unavailable` property is available in the router, and if
it's set to `false`, there's a case when NATs is not reachable from the
router in which potentially stale routes will continue to exist until the
router can reach NATs again. Is that correct?

(Sorry to repeat you back at yourself, just want to make sure I've
understood you correctly.)

Will

On 23 November 2015 at 19:02, Mike Youngstrom <youngm(a)gmail.com> wrote:

Hi Will,

Though I see the main reason for the issue assuming a healthy running
environment I've also experienced a deploy related issue that more unique
port assignment could help defend against. During one of our deploys the
routers finished deployed before the DEAs. When the DEAs started rolling,
for some reason some of our routers stopped getting route updates from
NATs. This caused their route tables to go stale and as apps started
rolling new apps started getting assigned ports previously held by other
apps. Which caused a number of our hosts to be mis-routed.

Though the root cause was probably some bug in the Nats client in
GoRouter the runtime team had apparently experienced a similar issue in the
past [0] which caused them to implement code that would delete stale routes
even then a router couldn't connect to NATs. The Router team is now
planning to optionally remove this failsafe [1]. I'm hoping that with the
removal of this failsafe (which I'm planning to take advantage of) this
tracker story will help protect us from the problem we experienced before
from happening again.

If the ports simply reset on a stemcell upgrade this issue provides no
defense for the problem we had before.

Does that make sense Will?

Mike

[0]
https://groups.google.com/a/cloudfoundry.org/d/msg/vcap-dev/yuVYCZkMLG8/7t8FHnFzWEsJ
[1] https://www.pivotaltracker.com/story/show/108659764

On Mon, Nov 23, 2015 at 11:11 AM, Will Pragnell <wpragnell(a)pivotal.io>
wrote:

Hi Mike,

What's the motivation for wanting rolling port assignment to persist
across e.g. stemcell upgrade? The motivation for this story is to prevent
stale routes from sending traffic to the wrong containers. Our assumption
is that stale routes won't ever exist for anything close to the amount of
time it takes BOSH to destroy and recreate a VM. Have we missed something
in making that assumption?

On your second point, I see your concern. We've talked about the
possibility of implementing FIFO semantics on free ports (when a port that
was in use becomes free, it goes to the end of the queue of available
ports) to decrease the chances of traffic reaching the wrong container as
far as possible. It's possible that the rolling ports approach is "good
enough" though. We're still trying to understand whether that's actually
the case.

The consistent hashing idea is interesting, but a few folks have
suggested that with a relatively small range of available ports (5000 by
default) that the chances of collision are actually higher than we'd want.
I'll see if someone wants to lay down some maths to give that idea some
credence.

Cheers,
Will

On 23 November 2015 at 08:47, Mike Youngstrom <youngm(a)gmail.com>
wrote:

Since I cannot comment in tracker I'm starting this thread to discuss
story:
https://www.pivotaltracker.com/n/projects/1158420/stories/92085170

Some comments I have:

* Although I can see how a rolling port assignment could be
maintained across garden/diego restarts I'd also like the story to ensure
that the rolling port assignments get maintained across a Stemcell upgrade
without the need for persistent disks on each cell. Perhaps etcd?

* Another thing to keep in mind. Although a rolling port value may
not duplicate ports 100% of the time in a short lived container in a long
lived container it seems to me that a rolling port assignment becomes no
more successful than a random port assignment if the container lives long
enough for the port assignment loop to loop a few times.

* Has there been any consideration to using an incremental consistent
hash of the app_guid to assign ports? A consistent hash would have the
benefit of being stateless. It also would have the benefit of increasing
the likely hood that if a request is sent to a stale route it may be to the
correct app anyway.

Thoughts?

Mike

6401 - 6420 of 9398