Matt and Amit,
Great feedback from you! Thanks so much for taking so much time.

I think I understand some more pieces of the puzzle, but, some more statements/questions -- if you could validate/comment, as you did earlier. (Amit, You had asked where I am looking up stuff. Mainly using cloudfoundry.org. There are also some blogs -- I saw one from someone in Pivotal.)

NATS BUS:
a) DEAs will publish App instances using their (routable) IP Address and a tcp Port (the Port is in turn mapped by the DEA to a private container ip and port).
b) Routers subscribe to this bus (since external traffic needs to be load balanced over the available App instances).
c) DEAs currently do not subscribe to this bus (may change with Diego).

ARCHITECTURE OF CONTAINER NETWORK (ON A DEA):
a) Each DEA may define one or more container networks.
b) Each container network is local to the DEA.
c) The container network could be associated with a subnet (say, 10.254.1.0/24, 10.254/2.0/24, or just point-to-point /30 or /31, etc)
d) A container/app will derive an address from its container subnet.
e) The DEA interface on the container network is configured with an IP address which represents the default gateway for the containers.
f) The DEA interface on the container network also provides the NAT function for traffic outbound from Container to anywhere outside the DEA.
g) This NAT function is provisioned via IPTABLES rules.

TRAFFIC FROM AN EXTERNAL WEB SERVER TO APP INSTANCE:
a) The router will target the DEA Address/Port it saw on the NATS BUS.
b) The source ip address of the web server will not change in this flow from Router to DEA to Container.
c) The flow from the Container to the web server will be NATTED by the DEA (to its external IP address -- the same address that the Web Server targeted).
d) There is mention of a NAT VM in the CF documentation. Not sure how that fits into the architecture!

TRAFFIC FROM APP to APP:
a) A container will use DNS and will receive the address of a Router or the LB in the DNS reply message (as Asit mentioned in his reply).
b) If two containers on different DEAs are talking to each other -- the flow would need to go through a router.
c) If two containers on the same DEA are talking to each other -- the flow would still need to go through a router.
d) In other words, Container-P and Container-Q on container subnet A could talk to each other over this virtual network but since they will never see each others private addresses all communications would need to go through a router.

NAT VM
a) I do not understand how this fits the architecture.

SECURITY
a) Need to understand how security is set up (assuming it is all via IPTABLES rules but need to look up the methods available to set up rules).