What about with BOSH?

On Mon, Dec 7, 2015 at 2:17 PM, john mcteague <john.mcteague(a)gmail.com>
wrote:

I am trying to enumerate all the ways I could identify all the DEA's in a
CF cluster in an IAAS agnostic way (e.g. not querying Openstack to read
metadata on the VM's that list the job type).

I think the only reliable way is to listen on NATS and look for the DEA
advertisement messages. Am I correct or is there a method I am missing (I'd
rather call and API rather than subscribe to NATS)?

Thanks,
John

Thanks a lot for directing me towards this project. When I am trying to
deploy CF using this project, there are some issues as below:


1. 'bosh prepare deployment' fails because of checksum failure on
package buildpack_python
2. Alternatively I tried to upload release as directed in the 'Read
Me' of this project but there is a version mismatch, the read me suggests
uploading version 196 whereas the deployment manifest mentions version 222.
3. There are also some spiff merge issue as I get the below error on
using 'bosh deploy'

I have been able to resolve errors 1 and 2 but need help for point 3 as am
not well versed with spiff. Kindly help me resolve this error:

ubuntu(a)ip-172-31-38-159:~/cf-boshworkspace$ bosh deploy
Generating deployment manifest
Command failed: 'spiff merge
/home/ubuntu/cf-boshworkspace/templates/cf/cf-deployment.yml
/home/ubuntu/cf-boshworkspace/templates/cf/cf-resource-pools.yml
/home/ubuntu/cf-boshworkspace/templates/tiny/cf-tiny-scalable.yml
/home/ubuntu/cf-boshworkspace/templates/cf-uaa-port.yml
/home/ubuntu/cf-boshworkspace/templates/cf-allow-services-access.yml
/home/ubuntu/cf-boshworkspace/templates/cf/cf-properties.yml
/home/ubuntu/cf-boshworkspace/templates/cf/cf-infrastructure-aws.yml
/home/ubuntu/cf-boshworkspace/templates/cf-properties.yml
/home/ubuntu/cf-boshworkspace/templates/tiny/cf-jobs-nfs.yml
/home/ubuntu/cf-boshworkspace/templates/tiny/cf-jobs-uaa.yml
/home/ubuntu/cf-boshworkspace/templates/tiny/cf-jobs-base.yml
/home/ubuntu/cf-boshworkspace/templates/parallel.yml
/home/ubuntu/cf-boshworkspace/templates/cf-no-ssl.yml
/home/ubuntu/cf-boshworkspace/templates/cf-secrets.yml
/home/ubuntu/cf-boshworkspace/templates/tiny/cf-use-nfs.yml
/home/ubuntu/cf-boshworkspace/templates/tiny/cf-use-haproxy.yml
/home/ubuntu/cf-boshworkspace/templates/cf-networking.yml
/home/ubuntu/cf-boshworkspace/.stubs/cf-aws-tiny.yml 2>&1'
2015/12/04 14:02:53 error generating manifest: unresolved nodes:
(( merge )) in
/home/ubuntu/cf-boshworkspace/templates/cf/cf-properties.yml
properties.cc.staging_upload_user
(( merge )) in
/home/ubuntu/cf-boshworkspace/templates/cf/cf-properties.yml
properties.cc.staging_upload_password
(( bulk_api_password )) in dynaml
properties.cc.internal_api_password
(( merge )) in
/home/ubuntu/cf-boshworkspace/templates/cf/cf-properties.yml
properties.cc.db_encryption_key
(( merge )) in
/home/ubuntu/cf-boshworkspace/templates/cf/cf-properties.yml
properties.cc.bulk_api_password

Thanks in Advance,
Kinjal


On Thu, Dec 3, 2015 at 12:37 AM, Amit Gupta <agupta(a)pivotal.io> wrote:

The cf-boshworkspace project from Stark and Wayne:
https://github.com/cloudfoundry-community/cf-boshworkspace provides an
cf-aws-tiny.yml template, which I believe gets it down to 4 instances.

In principle, everything should be able to run on 1 VM, but my
understanding is there are port collisions that aren't configurable (yet),
and so 4 was the lower bound S&W found.

On Wed, Dec 2, 2015 at 4:33 AM, Kinjal Doshi <kindoshi(a)gmail.com> wrote:

Hi,

I am looking at miminal-aws.yml in the release
https://github.com/cloudfoundry/cf-release and realized that this
deployment will consume 13 instances on AWS.

Is it possible to reduce the number of instances for this deployment?

Thanks a lot for the help in advance.

Regards,
Kinjal

Hi Amit,

This could be due to several reasons -- network issues, etcd slowness,

etcd
down, etc.

All right. I might ask questions again if the error messages are frequently
encountered however this is a separated issue and not highly prioritized so
far . So I am going to close this thread because the original issue has
been
resolved thanks to you. Thanks for your help.

Regards,
Masumi



--
View this message in context:
http://cf-dev.70369.x6.nabble.com/doppler-issue-which-fails-to-emit-logs-with-syslog-protocol-on-CFv212-tp2418p2979.html
Sent from the CF Dev mailing list archive at Nabble.com.

With the current way we have resource_ids being used for metering,
accumulation, and aggregation, we have to have a resource definition for
every resource_id that people are using. Then when we have custom
buildpacks, we will have to start creating a resource definition for each
one of them, inflating the amount of the resource definitions that we have.
So we will be adding a new field called resource_type_id ontop of the
resource_id that will be the resource definition abacus uses, allowing
custom buildpacks to fall under the resource type. So we were thinking of
having the hierarchy changed so resource_id is underneath resource_type_id.

When we implement this, we will have to change abacus to use the
resource_type_id rather than the resource_id to do the calculations. What
changes will this cause in abacus right now and how will this affect the
previous reports abacus has created?

Reagrds,
Daniel Mangin



--
View this message in context:
http://cf-dev.70369.x6.nabble.com/cf-env-abacus-Changing-how-resources-are-organized-tp2971.html
Sent from the CF Dev mailing list archive at Nabble.com.

Hello CF Community,

I would like to share a proposal that I have been thinking through with a
couple of my colleagues here at Pivotal. Attached below, you'll find a
high level view of a feature I'm referring to you as Elastic Clusters.
Please use the google doc as a place to provide feedback and ideas for this
feature.


https://docs.google.com/document/d/1BZL4GwluaRtUVfeW-j8_ezBJQmwnhGqcMNvQIy9ltZE/edit?usp=sharing

Thank you,

Mark Kropf

wrote:

Hi,

We are using the `syslog_deamon_config` proprty to stream all of our CF
infrastructure logs to an external stack for later processing.

We have noticed that operations like `cf create-user-provided-service`,
`cf bind-service`, and others are logged by multiple components in CF. That
would normally not be a problem, except that these commands often involve
passwords and those passwords get logged as well, ending up in the log
files on the VM and the target log processing stack, which allows operators
of the system to view end-user passwords.

We have noticed that the following jobs are responsible for the logs:

* api
* runner
* nats

Increasing the log level from the default `debug` / `debug2` to `info`
solves the problem for the first two, at the cost of making troubleshooting
tasks more difficult on the system.
The last one can only be solved by removing the `nats_stream_forwarder`
component from the `nats` job, again making troubleshooting more difficult.

I believe the ideal solution is to have those components not log the
payload of commands holding confidential information. Maybe they could
replace it with some pattern.
This would help for the first two but might not help for nats, where some
other means would be needed (encryption of the private content?).

How are you solving this issue on your productive system? What are your
thoughts on this matter?

Thanks in advance!

Regards,
Momchil Atanassov

Hi Mike,

Just to let you know we've not forgotten this. We're giving the matter
some thought, and we'll report back here with a proposal once we've figured
out some of the fiddly details.

Thanks,
Will

On 24 November 2015 at 15:19, Mike Youngstrom <youngm(a)gmail.com> wrote:

Yes Will, that summary is essentially correct. But, for even more
clarity let me restate the complete story again and reason I want 92085170
to work across stemcell upgrades. :)

Today if NATS goes down after 2 minutes the routers will drop their
routing tables and my entire CF deployment goes down. The routers behave
this way because of an experience Dieu had [0]. I don't like this I would
prefer for routers to not drop routing tables if it cannot connect to
Nats. Therefore, the routing team is adding 'prune_on_config_unavailable'.
I plan to set this to false to make my deployment less sensitive to NATS
failure. In doing so I am incurring more risk of mis routed stale routes.
I am hoping that 92085170 will help reduce some of that risk. Since one of
the times I personally have experienced stale route routing was during a
deploy I hope that Garden will consider a port selection technique that
will help ensure uniqueness across stemcell upgrades, something we
frequently do as part of a deploy.

Consequently a stateless solution like random assignment or a consistent
hash will work across stemcell upgrades.

Thanks,
Mike

[0]
https://groups.google.com/a/cloudfoundry.org/d/msg/vcap-dev/yuVYCZkMLG8/7t8FHnFzWEsJ

On Tue, Nov 24, 2015 at 3:44 AM, Will Pragnell <wpragnell(a)pivotal.io>
wrote:

Hi Mike,

What I think you're saying is that once the new
`prune_on_config_unavailable` property is available in the router, and if
it's set to `false`, there's a case when NATs is not reachable from the
router in which potentially stale routes will continue to exist until the
router can reach NATs again. Is that correct?

(Sorry to repeat you back at yourself, just want to make sure I've
understood you correctly.)

Will

On 23 November 2015 at 19:02, Mike Youngstrom <youngm(a)gmail.com> wrote:

Hi Will,

Though I see the main reason for the issue assuming a healthy running
environment I've also experienced a deploy related issue that more unique
port assignment could help defend against. During one of our deploys the
routers finished deployed before the DEAs. When the DEAs started rolling,
for some reason some of our routers stopped getting route updates from
NATs. This caused their route tables to go stale and as apps started
rolling new apps started getting assigned ports previously held by other
apps. Which caused a number of our hosts to be mis-routed.

Though the root cause was probably some bug in the Nats client in
GoRouter the runtime team had apparently experienced a similar issue in the
past [0] which caused them to implement code that would delete stale routes
even then a router couldn't connect to NATs. The Router team is now
planning to optionally remove this failsafe [1]. I'm hoping that with the
removal of this failsafe (which I'm planning to take advantage of) this
tracker story will help protect us from the problem we experienced before
from happening again.

If the ports simply reset on a stemcell upgrade this issue provides no
defense for the problem we had before.

Does that make sense Will?

Mike

[0]
https://groups.google.com/a/cloudfoundry.org/d/msg/vcap-dev/yuVYCZkMLG8/7t8FHnFzWEsJ
[1] https://www.pivotaltracker.com/story/show/108659764

On Mon, Nov 23, 2015 at 11:11 AM, Will Pragnell <wpragnell(a)pivotal.io>
wrote:

Hi Mike,

What's the motivation for wanting rolling port assignment to persist
across e.g. stemcell upgrade? The motivation for this story is to prevent
stale routes from sending traffic to the wrong containers. Our assumption
is that stale routes won't ever exist for anything close to the amount of
time it takes BOSH to destroy and recreate a VM. Have we missed something
in making that assumption?

On your second point, I see your concern. We've talked about the
possibility of implementing FIFO semantics on free ports (when a port that
was in use becomes free, it goes to the end of the queue of available
ports) to decrease the chances of traffic reaching the wrong container as
far as possible. It's possible that the rolling ports approach is "good
enough" though. We're still trying to understand whether that's actually
the case.

The consistent hashing idea is interesting, but a few folks have
suggested that with a relatively small range of available ports (5000 by
default) that the chances of collision are actually higher than we'd want.
I'll see if someone wants to lay down some maths to give that idea some
credence.

Cheers,
Will

On 23 November 2015 at 08:47, Mike Youngstrom <youngm(a)gmail.com>
wrote:

Since I cannot comment in tracker I'm starting this thread to discuss
story:
https://www.pivotaltracker.com/n/projects/1158420/stories/92085170

Some comments I have:

* Although I can see how a rolling port assignment could be
maintained across garden/diego restarts I'd also like the story to ensure
that the rolling port assignments get maintained across a Stemcell upgrade
without the need for persistent disks on each cell. Perhaps etcd?

* Another thing to keep in mind. Although a rolling port value may
not duplicate ports 100% of the time in a short lived container in a long
lived container it seems to me that a rolling port assignment becomes no
more successful than a random port assignment if the container lives long
enough for the port assignment loop to loop a few times.

* Has there been any consideration to using an incremental consistent
hash of the app_guid to assign ports? A consistent hash would have the
benefit of being stateless. It also would have the benefit of increasing
the likely hood that if a request is sent to a stale route it may be to the
correct app anyway.

Thoughts?

Mike