I’ve been trying to configure the CC security groups to get this stuff working and haven’t been able to do it. Currently I’m trying to use just one security group to allow anything:
Name allow_all
Rules
[
{
"destination": "0.0.0.0-255.255.255.255",
"protocol": "all"
},
{
"destination": "0.0.0.0/0",
"ports": "53",
"protocol": "tcp"
},
{
"destination": "0.0.0.0/0",
"ports": "53",
"protocol": "udp"
}
]

But I still can’t get to anything from inside the container. Is there something else I have to configure for this?

From: CF Runtime <cfruntime(a)gmail.com<mailto:cfruntime(a)gmail.com>>
Reply-To: "Discussions about Cloud Foundry projects and the system overall." <cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org>>
Date: Thursday, September 17, 2015 at 1:28 PM
To: "Discussions about Cloud Foundry projects and the system overall." <cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org>>
Subject: [cf-dev] Re: Re: Re: Re: Re: Re: Re: Re: DEA/Warden staging error

It certainly could be. By default the contains reject all egress traffic. CC security groups configure iptables rules that allow traffic out.

One of the default security groups in the BOSH templates allows access on port 53. If you have no security groups, the containers will not be able to make any outgoing requests.

Joseph & Natalie
CF Release Integration Team

After more tinkering, I got security group rules set up that allowed the app to download the buildpack and push succesfully. However after about 30 seconds the container loses network connectivity and external requests to it fail. I can still get into the container with the warden shell but any network requests fail. Would something be blocking the container’s network access shortly after it come up (mistakenly thinking it’s unresponsive)?

From: Kyle Havlovitz <kyhavlov(a)cisco.com<mailto:kyhavlov(a)cisco.com>>
Reply-To: "Discussions about Cloud Foundry projects and the system overall." <cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org>>
Date: Thursday, September 17, 2015 at 3:30 PM
To: "Discussions about Cloud Foundry projects and the system overall." <cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org>>
Subject: [cf-dev] DEA/Warden staging error

I’ve been trying to configure the CC security groups to get this stuff working and haven’t been able to do it. Currently I’m trying to use just one security group to allow anything:
Name allow_all
Rules
[
{
"destination": "0.0.0.0-255.255.255.255",
"protocol": "all"
},
{
"destination": "0.0.0.0/0",
"ports": "53",
"protocol": "tcp"
},
{
"destination": "0.0.0.0/0",
"ports": "53",
"protocol": "udp"
}
]

But I still can’t get to anything from inside the container. Is there something else I have to configure for this?

From: CF Runtime <cfruntime(a)gmail.com<mailto:cfruntime(a)gmail.com>>
Reply-To: "Discussions about Cloud Foundry projects and the system overall." <cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org>>
Date: Thursday, September 17, 2015 at 1:28 PM
To: "Discussions about Cloud Foundry projects and the system overall." <cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org>>
Subject: [cf-dev] Re: Re: Re: Re: Re: Re: Re: Re: DEA/Warden staging error

It certainly could be. By default the contains reject all egress traffic. CC security groups configure iptables rules that allow traffic out.

One of the default security groups in the BOSH templates allows access on port 53. If you have no security groups, the containers will not be able to make any outgoing requests.

Joseph & Natalie
CF Release Integration Team

The Warden logs (and possibly the DEA logs) would probably be helpful in
figuring out what is going on there.

Joseph
CF Release Integration Team

On Thu, Sep 17, 2015 at 1:23 PM, Kyle Havlovitz (kyhavlov) <
kyhavlov(a)cisco.com> wrote:

After more tinkering, I got security group rules set up that allowed the
app to download the buildpack and push succesfully. However after about 30
seconds the container loses network connectivity and external requests to
it fail. I can still get into the container with the warden shell but any
network requests fail. Would something be blocking the container’s network
access shortly after it come up (mistakenly thinking it’s unresponsive)?

From: Kyle Havlovitz <kyhavlov(a)cisco.com>
Reply-To: "Discussions about Cloud Foundry projects and the system
overall." <cf-dev(a)lists.cloudfoundry.org>
Date: Thursday, September 17, 2015 at 3:30 PM
To: "Discussions about Cloud Foundry projects and the system overall." <
cf-dev(a)lists.cloudfoundry.org>
Subject: [cf-dev] DEA/Warden staging error

I’ve been trying to configure the CC security groups to get this stuff
working and haven’t been able to do it. Currently I’m trying to use just
one security group to allow anything:
Name allow_all
Rules
[
{
"destination": "0.0.0.0-255.255.255.255",
"protocol": "all"
},
{
"destination": "0.0.0.0/0",
"ports": "53",
"protocol": "tcp"
},
{
"destination": "0.0.0.0/0",
"ports": "53",
"protocol": "udp"
}
]

But I still can’t get to anything from inside the container. Is there
something else I have to configure for this?

From: CF Runtime <cfruntime(a)gmail.com>
Reply-To: "Discussions about Cloud Foundry projects and the system
overall." <cf-dev(a)lists.cloudfoundry.org>
Date: Thursday, September 17, 2015 at 1:28 PM
To: "Discussions about Cloud Foundry projects and the system overall." <
cf-dev(a)lists.cloudfoundry.org>
Subject: [cf-dev] Re: Re: Re: Re: Re: Re: Re: Re: DEA/Warden staging error

It certainly could be. By default the contains reject all egress traffic.
CC security groups configure iptables rules that allow traffic out.

One of the default security groups in the BOSH templates allows access on
port 53. If you have no security groups, the containers will not be able to
make any outgoing requests.

Joseph & Natalie
CF Release Integration Team

On Thu, Sep 17, 2015 at 8:44 AM, Kyle Havlovitz (kyhavlov) <
kyhavlov(a)cisco.com> wrote:

On running git clone inside the container via the warden shell, I get:
"Cloning into 'staticfile-buildpack'...
fatal: unable to access '
https://github.com/cloudfoundry/staticfile-buildpack/': Could not
resolve host: github.com".
So the container can't get to anything outside of it (I also tried
pinging some external IPs to make sure it wasn't a DNS thing). Would this
be caused by cloud controller security group settings?

Hey Kyle,

Did you make any progress?

Zak & Mikhail
CF Release Integration Team

I didn't; I'm still having this problem. Even adding this lenient security group didn't let me get any traffic out of the VM:

[{"name":"allow_all","rules":[{"protocol":"all","destination":"0.0.0.0/0"},{"protocol":"tcp","destination":"0.0.0.0/0","ports":"1-65535"},{"protocol":"udp","destination":"0.0.0.0/0","ports":"1-65535"}]}]

The only way I was able to get traffic out was by manually removing the reject/drop iptables rules that warden set up, and even with that the container still lost all connectivity after 30 seconds.

From: CF Runtime <cfruntime(a)gmail.com<mailto:cfruntime(a)gmail.com>>
Reply-To: "Discussions about Cloud Foundry projects and the system overall." <cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org>>
Date: Tuesday, September 22, 2015 at 12:50 PM
To: "Discussions about Cloud Foundry projects and the system overall." <cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org>>
Subject: [cf-dev] Re: Re: Re: Re: Re: Re: Re: Re: DEA/Warden staging error

Hey Kyle,

Did you make any progress?

Zak & Mikhail
CF Release Integration Team

Based on your description, it doesn't sound like warden networking or the
warden iptables chains are your problem. Are you able to share all of your
routes and chains via a gist?

route -n
ifconfig -a
iptables -L -n -v -t filter
iptables -L -n -v -t nat
iptables -L -n -v -t mangle

Any kernel messages that look relevant in the message buffer (dmesg)?

Have you tried doing a network capture to verify the packets are look the
way you expect? Are you sure your host routing rules are good? Do the
warden subnets overlap with any network accessible to the host?

Based on previous notes, it doesn't sound like this is a standard
deployment so it's hard to say what could be impacting you.

On Tue, Sep 22, 2015 at 1:08 PM, Kyle Havlovitz (kyhavlov) <
kyhavlov(a)cisco.com> wrote:

I didn’t; I’m still having this problem. Even adding this lenient security
group didn’t let me get any traffic out of the VM:

[{"name":"allow_all","rules":[{"protocol":"all","destination":"0.0.0.0/0
"},{"protocol":"tcp","destination":"0.0.0.0/0
","ports":"1-65535"},{"protocol":"udp","destination":"0.0.0.0/0
","ports":"1-65535"}]}]

The only way I was able to get traffic out was by manually removing the
reject/drop iptables rules that warden set up, and even with that the
container still lost all connectivity after 30 seconds.

From: CF Runtime <cfruntime(a)gmail.com>
Reply-To: "Discussions about Cloud Foundry projects and the system
overall." <cf-dev(a)lists.cloudfoundry.org>
Date: Tuesday, September 22, 2015 at 12:50 PM
To: "Discussions about Cloud Foundry projects and the system overall." <
cf-dev(a)lists.cloudfoundry.org>
Subject: [cf-dev] Re: Re: Re: Re: Re: Re: Re: Re: DEA/Warden staging error

Hey Kyle,

Did you make any progress?

Zak & Mikhail
CF Release Integration Team

On Thu, Sep 17, 2015 at 10:28 AM, CF Runtime <cfruntime(a)gmail.com> wrote:

It certainly could be. By default the contains reject all egress traffic.
CC security groups configure iptables rules that allow traffic out.

One of the default security groups in the BOSH templates allows access on
port 53. If you have no security groups, the containers will not be able to
make any outgoing requests.

Joseph & Natalie
CF Release Integration Team

On Thu, Sep 17, 2015 at 8:44 AM, Kyle Havlovitz (kyhavlov) <
kyhavlov(a)cisco.com> wrote:

On running git clone inside the container via the warden shell, I get:
"Cloning into 'staticfile-buildpack'...
fatal: unable to access '
https://github.com/cloudfoundry/staticfile-buildpack/': Could not
resolve host: github.com".
So the container can't get to anything outside of it (I also tried
pinging some external IPs to make sure it wasn't a DNS thing). Would this
be caused by cloud controller security group settings?

--
Matthew Sykes
matthew.sykes(a)gmail.com

Here's the output from those commands:
https://gist.github.com/MrEnzyme/36592831b1c46d44f007
Soon after running those I noticed that the container loses its IPv4
address shortly after coming up and ifconfig looks like this:

root(a)cf-build:/home/cloud-user/test# ifconfig -a

docker0 Link encap:Ethernet HWaddr 56:84:7a:fe:97:99
inet addr:172.17.42.1 Bcast:0.0.0.0 Mask:255.255.0.0
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
eth0 Link encap:Ethernet HWaddr fa:16:3e:cd:f3:0a
inet addr:172.25.1.52 Bcast:172.25.1.127 Mask:255.255.255.128
inet6 addr: fe80::f816:3eff:fecd:f30a/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:515749 errors:0 dropped:0 overruns:0 frame:0
TX packets:295471 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1162366659 (1.1 GB) TX bytes:59056756 (59.0 MB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:45057315 errors:0 dropped:0 overruns:0 frame:0
TX packets:45057315 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:18042315375 (18.0 GB) TX bytes:18042315375 (18.0 GB)
w-190db6c54la-0 Link encap:Ethernet HWaddr 12:dc:ba:da:38:5b
inet6 addr: fe80::10dc:baff:feda:385b/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1454 Metric:1
RX packets:12 errors:0 dropped:0 overruns:0 frame:0
TX packets:227 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:872 (872.0 B) TX bytes:35618 (35.6 KB)

Any idea what would be causing that?


On Tue, Sep 22, 2015 at 10:31 PM, Matthew Sykes <matthew.sykes(a)gmail.com>
wrote:

Based on your description, it doesn't sound like warden networking or the
warden iptables chains are your problem. Are you able to share all of your
routes and chains via a gist?

route -n
ifconfig -a
iptables -L -n -v -t filter
iptables -L -n -v -t nat
iptables -L -n -v -t mangle

Any kernel messages that look relevant in the message buffer (dmesg)?

Have you tried doing a network capture to verify the packets are look the
way you expect? Are you sure your host routing rules are good? Do the
warden subnets overlap with any network accessible to the host?

Based on previous notes, it doesn't sound like this is a standard
deployment so it's hard to say what could be impacting you.

On Tue, Sep 22, 2015 at 1:08 PM, Kyle Havlovitz (kyhavlov) <
kyhavlov(a)cisco.com> wrote:

I didn’t; I’m still having this problem. Even adding this lenient
security group didn’t let me get any traffic out of the VM:

[{"name":"allow_all","rules":[{"protocol":"all","destination":"0.0.0.0/0
"},{"protocol":"tcp","destination":"0.0.0.0/0
","ports":"1-65535"},{"protocol":"udp","destination":"0.0.0.0/0
","ports":"1-65535"}]}]

The only way I was able to get traffic out was by manually removing the
reject/drop iptables rules that warden set up, and even with that the
container still lost all connectivity after 30 seconds.

From: CF Runtime <cfruntime(a)gmail.com>
Reply-To: "Discussions about Cloud Foundry projects and the system
overall." <cf-dev(a)lists.cloudfoundry.org>
Date: Tuesday, September 22, 2015 at 12:50 PM
To: "Discussions about Cloud Foundry projects and the system overall." <
cf-dev(a)lists.cloudfoundry.org>
Subject: [cf-dev] Re: Re: Re: Re: Re: Re: Re: Re: DEA/Warden staging
error

Hey Kyle,

Did you make any progress?

Zak & Mikhail
CF Release Integration Team

On Thu, Sep 17, 2015 at 10:28 AM, CF Runtime <cfruntime(a)gmail.com> wrote:

It certainly could be. By default the contains reject all egress
traffic. CC security groups configure iptables rules that allow traffic
out.

One of the default security groups in the BOSH templates allows access
on port 53. If you have no security groups, the containers will not be able
to make any outgoing requests.

Joseph & Natalie
CF Release Integration Team

On Thu, Sep 17, 2015 at 8:44 AM, Kyle Havlovitz (kyhavlov) <
kyhavlov(a)cisco.com> wrote:

On running git clone inside the container via the warden shell, I get:
"Cloning into 'staticfile-buildpack'...
fatal: unable to access '
https://github.com/cloudfoundry/staticfile-buildpack/': Could not
resolve host: github.com".
So the container can't get to anything outside of it (I also tried
pinging some external IPs to make sure it wasn't a DNS thing). Would this
be caused by cloud controller security group settings?

--
Matthew Sykes
matthew.sykes(a)gmail.com

Ok, after more investigating the problem was that network manager was
running on the machine and was trying to take control of new network
interfaces after they came up, so it would cause problems with the
interface that Warden created for the container. With network manager
disabled I can push the app and everything is fine.

Thanks for your help everyone.