You have a lot of questions here and some of them may depend on how you’re deploying Cloud Foundry. I’ll add some answers inline.

On Dec 16, 2015, at 11:05 AM, ravi malhotra <ravi.malhotra(a)bnymellon.com> wrote:

I did not find detailed documentation so I have created a set of assumptions below on how CF Networking works. Are these correct? Can I get answers to the questions? Thank you!

High Level Architecture/Scaling:
1. APP Containers and CF Routers are created/destroyed dynamically to support application needs.

A container is created for each desired instance of an application and they can be destroyed for various reasons. The most common reasons are because you’ve scaled down the number of desired instances, your application instance has crashed, or the DEA/Cell hosting the application instance is evacuating application instances for maintenance.

Routers are not tied to an application container lifecycle but routes are.

2. A Router can LB to any DEA in the environment (or, are there Availability Zones which prescribe sets of Routers and DEAs?)

Today routers will forward requests to any application instance that advertises a route which matches a request. Routers do not care about DEAs or anything else.

3. DEAs cannot talk directly to each other; APP1 to APP2 communication must go through a Router?

There’s nothing that prevents DEAs from communicating with each other but, in general, they do not; there’s no need to. This is not the same as saying application instances can talk to each other...

When it comes to application instances that are hosted by a DEA, in the default configuration, application instances are not able to communicate directly with each other without going through the router. This is due to the presence of application security groups and networking sandboxing. While strongly discouraged, warden can be configured to allow access to the DEA network stack and security groups can be modified to allow direct access to other DEAs if needed.

A container networking project has just been proposed at the Runtime PMC with the goal of enabling container to container communication with appropriate access control and policies.

4. If I deploy my own LB solution -- how do I dynamically update Router addresses in my LB (as Routers are created/destroyed)?

It will likely require engineering effort on your part. Some organizations have had success (LDS presented ‘norouter’ at one of the summits) but routing is a bit of a moving target. The routing team is better positioned to answer this.

Communication from Router to App:
1. Router can use some algorithm (like round-robin) to direct traffic to a DEA.

The current implementation uses simple round robin.

2. Router to DEA traffic: is there an overlay network? or are we just utilizing the native network?

Generally speaking, it’s the native network but, ultimately, it depends on how and where you’ve deployed Cloud Foundry.

3. Router to DEA traffic: is the Router just changing the destination address of the request to the address of the DEA and forwarding the request with the source address intact?

The router is targeting the ip and port advertised in the route. For application instances hosted by a DEA, that will be the DEA’s address and a port associated with a DNAT rule to the application’s endpoint in the container’s network namespace.

4. Router to DEA traffic: let's say the Router dies half way through; can we mirror state to another Router?

I don’t follow the question.

5. If a Router dies – all the DEAs can still be accessed via other Routers; is this right?

In general, yes, but it ultimately depends on how you’ve deployed.

Communication from point of view of App/Container:
1. An APP (container) cannot directly talk to another APP (container) even in the same DEA. This communication must go through a Router. Is this accurate?

Answered previously.

2. The container is in a Network Name Space which is bridged to a Linux Bridge that then joins to physical NIC.

No. In warden it’s a veth pair with a private IP that is only accessible to the host.

3. Containers are isolated from each other because they are in different Name Spaces and because of IPTables rules.

Yes.

3. IPTables rules allow the container to communicate with all Routers.

In general, no. The iptables rules allow access to the load balancer sitting in front of the go routers. But, again, this depends on your application security groups.

4. IPTables rules bar the container from directly talking to anything that is not a Router.

Depends on your application security groups.

East-West traffic between Containers:
1. E-W traffic must go through a Router.
2. APP1 will seek out a Router (which one?)
3. The Router will direct the request to APP2 on some DEA using some algorithm (say, round-robin).
4. The reverse traffic from APP2 to APP1 would need to be NATTED to the Router address. Also, we need a destination NAT. Not sure how the NAT function would do this work.

These have generally been answered already.

Management:
1. Is there ability to define network policy in WARDEN to shut an APP?

I can’t parse the question.

2. We may want to define policy based on bandwidth usage.

Warden and DEA support bandwidth limits; Diego and Garden do not (yet).

3. Can we configure QoS bits on an application?

No.

Troubleshooting:
1. Is there a promiscuous APP on a container that can sniff all traffic so we can troubleshoot?
2. Use case for above: let's say an APP appears to freeze -- having a packet capture from the DEA node could help diagnose the problem.

If you’re dealing with your own deployment, you can use standard network troubleshooting techniques from the DEA.

Performance:
1. When is a new CF Router instance spun up? Can I set up a rule in BOSH to spin up new router when a certain traffic threshold is exceeded?

Routers are started when you change the number of instances in your bosh manifest and deploy. Bosh does not do auto scaling.

2. Similarly, when are new APP instances spun up?

App instances are started when the number of desired instances changes. Cloud Foundry does not do auto scaling but many vendors have ways of doing this.

3. Is there any performance data available on the CF Router?

There are a number of metrics managed by the go router and the access logs provide response time information.

DEA:
1. can DEA's be multihomed on Public and private networks?

Depends on your deployment but I believe the DNAT rules for app instances will only be configured for one interface.

2. the BOSH agent on each DEA – what are all its functions? Is it collecting health data used by the router in the LB decision?

You should look at the bosh documentation. The agent functions are not specific to DEAs or any other job type in a Cloud Foundry deployment.

Packet walk (please include LB and overlay technologies involved):
1. From App to App within a droplet?
2. From App to App between droplets on the same host?
3. From App to App between droplets on different hosts?
4. From App to App between Availability Zones? (is this allowed?)
5. From web server (outside CF environment) to App.

This seems like an exercise, not a question.

IP Addressing:
1. The containers all take addresses from a NATTED range (say, 10.254.0.0/16). Don’t I also need to NAT my source address? Example, I am coming from an Apache web server to a CF App. The source address of the Apache web server cannot be from the 10.254.0.0/16 range (if it were, we would need to NAT the source).
2. Are the container addresses further subnetted (say, /24 per host?)

IP Multicast: Assuming there is no requirement for IP multicast in this space.

Details: Commands to check which containers are up? What are their addresses? Which DEAs a router knows about? What tcp sessions are active? Where can I find the detailed documentation?