I did not find detailed documentation so I have created a set of assumptions below on how CF Networking works. Are these correct? Can I get answers to the questions? Thank you!
High Level Architecture/Scaling:
1. APP Containers and CF Routers are created/destroyed dynamically to support application needs.
2. A Router can LB to any DEA in the environment (or, are there Availability Zones which prescribe sets of Routers and DEAs?)
3. DEAs cannot talk directly to each other; APP1 to APP2 communication must go through a Router?
4. If I deploy my own LB solution -- how do I dynamically update Router addresses in my LB (as Routers are created/destroyed)?
Communication from Router to App:
1. Router can use some algorithm (like round-robin) to direct traffic to a DEA.
2. Router to DEA traffic: is there an overlay network? or are we just utilizing the native network?
3. Router to DEA traffic: is the Router just changing the destination address of the request to the address of the DEA and forwarding the request with the source address intact?
4. Router to DEA traffic: let's say the Router dies half way through; can we mirror state to another Router?
5. If a Router dies – all the DEAs can still be accessed via other Routers; is this right?
Communication from point of view of App/Container:
1. An APP (container) cannot directly talk to another APP (container) even in the same DEA. This communication must go through a Router. Is this accurate?
2. The container is in a Network Name Space which is bridged to a Linux Bridge that then joins to physical NIC.
3. Containers are isolated from each other because they are in different Name Spaces and because of IPTables rules.
3. IPTables rules allow the container to communicate with all Routers.
4. IPTables rules bar the container from directly talking to anything that is not a Router.
East-West traffic between Containers:
1. E-W traffic must go through a Router.
2. APP1 will seek out a Router (which one?)
3. The Router will direct the request to APP2 on some DEA using some algorithm (say, round-robin).
4. The reverse traffic from APP2 to APP1 would need to be NATTED to the Router address. Also, we need a destination NAT. Not sure how the NAT function would do this work.
1. Is there ability to define network policy in WARDEN to shut an APP?
2. We may want to define policy based on bandwidth usage.
3. Can we configure QoS bits on an application?
1. Is there a promiscuous APP on a container that can sniff all traffic so we can troubleshoot?
2. Use case for above: let's say an APP appears to freeze -- having a packet capture from the DEA node could help diagnose the problem.
1. When is a new CF Router instance spun up? Can I set up a rule in BOSH to spin up new router when a certain traffic threshold is exceeded?
2. Similarly, when are new APP instances spun up?
3. Is there any performance data available on the CF Router?
1. can DEA's be multihomed on Public and private networks?
2. the BOSH agent on each DEA – what are all its functions? Is it collecting health data used by the router in the LB decision?
Packet walk (please include LB and overlay technologies involved):
1. From App to App within a droplet?
2. From App to App between droplets on the same host?
3. From App to App between droplets on different hosts?
4. From App to App between Availability Zones? (is this allowed?)
5. From web server (outside CF environment) to App.
1. The containers all take addresses from a NATTED range (say, 10.254.0.0/16). Don’t I also need to NAT my source address? Example, I am coming from an Apache web server to a CF App. The source address of the Apache web server cannot be from the 10.254.0.0/16 range (if it were, we would need to NAT the source).
2. Are the container addresses further subnetted (say, /24 per host?)
IP Multicast: Assuming there is no requirement for IP multicast in this space.
Details: Commands to check which containers are up? What are their addresses? Which DEAs a router knows about? What tcp sessions are active? Where can I find the detailed documentation?