1. Topics
  2. App Container IP Address assignment on vSphere

App Container IP Address assignment on vSphere

Daya Shetty <daya.shetty@...>
 

Hello,

On vSphere I'm seeing the app containers are assigned 10.254.0.x, 10.254.2.x, 10.254.3.x addresses though the default value of the "pool_network" is 10.254.0.0/24 and the pool_size is 256. Any reason why?

Also, if we have DEA's on 2 AZ's, do they share this network_pool or will they independently allocate from their own NAT'ed pool.

Thanks
Daya
Gwenn Etourneau
 

This pool network is for Warden / Garden (Diego) is not share accross VM
...

On Wed, Dec 9, 2015 at 10:04 AM, Daya Shetty <daya.shetty(a)bnymellon.com>
wrote:

Hello,

On vSphere I'm seeing the app containers are assigned 10.254.0.x,
10.254.2.x, 10.254.3.x addresses though the default value of the
"pool_network" is 10.254.0.0/24 and the pool_size is 256. Any reason why?

Also, if we have DEA's on 2 AZ's, do they share this network_pool or will
they independently allocate from their own NAT'ed pool.

Thanks
Daya

?
attachment.html

Will Pragnell <wpragnell@...>
 

Hi Daya,

I think the reason that you're getting IPs you don't expect might be
because you're getting the property names wrong. It's not `pool_network`,
it's `network_pool` [1]. There is no `pool_size` property.

Will

[1]:
https://github.com/cloudfoundry-incubator/garden-linux-release/blob/develop/jobs/garden/spec#L68

On 9 December 2015 at 02:50, Gwenn Etourneau <getourneau(a)pivotal.io> wrote:

This pool network is for Warden / Garden (Diego) is not share accross VM
...

On Wed, Dec 9, 2015 at 10:04 AM, Daya Shetty <daya.shetty(a)bnymellon.com>
wrote:

Hello,

On vSphere I'm seeing the app containers are assigned 10.254.0.x,
10.254.2.x, 10.254.3.x addresses though the default value of the
"pool_network" is 10.254.0.0/24 and the pool_size is 256. Any reason why?

Also, if we have DEA's on 2 AZ's, do they share this network_pool or
will they independently allocate from their own NAT'ed pool.

Thanks
Daya

?
attachment.html

Daya Shetty <daya.shetty@...>
 

Thanks Gwenn!
Daya Shetty <daya.shetty@...>
 

Will,

We are using warden containers in our deployment and I was referring to the attributes defined in

./cf-release/jobs/dea_next/templates/warden.yml.erb

network:
pool_start_address: 10.254.0.0
pool_size: 256

and in ./cf-release/src/warden/warden/lib/warden/config.rb

def self.network_defaults
{
"pool_network" => "10.254.0.0/24",
"deny_networks" => [],
"allow_networks" => [],
"allow_host_access" => false,
"mtu" => 1500,
}
end

def self.network_schema
::Membrane::SchemaParser.parse do
{
# Preferred way to specify networks to pool
optional("pool_network") => String,

# Present for Backwards compatibility
optional("pool_start_address") => String,
optional("pool_size") => Integer,
optional("release_delay") => Integer,
optional("mtu") => Integer,

"deny_networks" => [String],
"allow_networks" => [String],
optional("allow_host_access") => bool,
}
end

Thanks
Daya
Will Pragnell <wpragnell@...>
 

Ah, sorry, my bad! I assumed Garden for some reason.

On 9 December 2015 at 21:15, Daya Shetty <daya.shetty(a)bnymellon.com> wrote:

Will,

We are using warden containers in our deployment and I was referring to
the attributes defined in

./cf-release/jobs/dea_next/templates/warden.yml.erb

network:
pool_start_address: 10.254.0.0
pool_size: 256

and in ./cf-release/src/warden/warden/lib/warden/config.rb

def self.network_defaults
{
"pool_network" => "10.254.0.0/24",
"deny_networks" => [],
"allow_networks" => [],
"allow_host_access" => false,
"mtu" => 1500,
}
end

def self.network_schema
::Membrane::SchemaParser.parse do
{
# Preferred way to specify networks to pool
optional("pool_network") => String,

# Present for Backwards compatibility
optional("pool_start_address") => String,
optional("pool_size") => Integer,
optional("release_delay") => Integer,
optional("mtu") => Integer,

"deny_networks" => [String],
"allow_networks" => [String],
optional("allow_host_access") => bool,
}
end

Thanks
Daya

?
attachment.html

Eric Malm <emalm@...>
 

Hi, Daya,

Based on
https://github.com/cloudfoundry/warden/blob/master/warden/lib/warden/config.rb#L207-L216,
the warden server uses the values of the network.pool_start_address and
network.pool_size properties from the rendered warden.yml config file to
construct a value for the pool_network property. Warden allocates a /30
subnet for each container, to have room for both the host-side and
container-side IP addresses in the veth pair, as well as the broadcast
address on the subnet. With the default values of 10.254.0.0 for the pool
start address and 256 (= 2^8) for the pool size, warden then calculates the
pool network to be 10.254.0.0/22. This /22 subnet includes the 10.254.2.x
and 10.254.3.x addresses you have observed on your DEAs.

In any case, these 10.254.x.y IP addresses are used only internally on each
DEA or Diego cell VM, so there's no conflict between these IP addresses on
other VMs that run warden/garden containers. If you examine the 'nat' table
in the iptables config, you'll see that for each container, warden creates
a NAT rule that directs inbound traffic from a particular port on the host
VM's eth0 interface to that same port on the container's host-side veth
interface (the one with offset 2 in the container's /30 subnet). The DEA
then provides this port as the value of the $PORT environment variable, so
the CF app process running in the container can listen on that port for its
web traffic.

Thanks,
Eric

On Wed, Dec 9, 2015 at 11:25 PM, Will Pragnell <wpragnell(a)pivotal.io> wrote:

Ah, sorry, my bad! I assumed Garden for some reason.

On 9 December 2015 at 21:15, Daya Shetty <daya.shetty(a)bnymellon.com>
wrote:

Will,

We are using warden containers in our deployment and I was referring to
the attributes defined in

./cf-release/jobs/dea_next/templates/warden.yml.erb

network:
pool_start_address: 10.254.0.0
pool_size: 256

and in ./cf-release/src/warden/warden/lib/warden/config.rb

def self.network_defaults
{
"pool_network" => "10.254.0.0/24",
"deny_networks" => [],
"allow_networks" => [],
"allow_host_access" => false,
"mtu" => 1500,
}
end

def self.network_schema
::Membrane::SchemaParser.parse do
{
# Preferred way to specify networks to pool
optional("pool_network") => String,

# Present for Backwards compatibility
optional("pool_start_address") => String,
optional("pool_size") => Integer,
optional("release_delay") => Integer,
optional("mtu") => Integer,

"deny_networks" => [String],
"allow_networks" => [String],
optional("allow_host_access") => bool,
}
end

Thanks
Daya

?
attachment.html

Daya Shetty <daya.shetty@...>
 

Hi Eric,

Thanks for the detailed explanation! Makes perfect sense as the network pool was calculated to be 10.254.0.0/22 and not 10.254.0.0/24.

Regards
Daya
1 - 8 of 8