Hey John,

Yes, that's come up a number of places now, and we plan on upping the
(userland) nf_conntrack_max parameter on the DEA as well, we're still
trying to figure out a sane default for that:

https://www.pivotaltracker.com/story/show/104554676

The tcp_fin_timeout and tcp_tw_reuse are separate kernel parameters that
need to be tuned as well, and currently the DEA (and a couple other
components) do this. But we think any BOSH-deployed service will need
something like this, hence the proposal to put it in the stemcell.

Thanks,
Amit

On Thu, Oct 1, 2015 at 1:07 PM, john mcteague <john.mcteague(a)gmail.com>
wrote:

My knowledge of linux networking could fit on a postage stamp, but this
thread reminded me of something we came across recently during performance
testing: "ip_conntrack: table full, dropping packet" errors on the DEA's.
No requests would make it to any app for a period and then a small portion
would make it through once closed tcp connections were released.

As a very short term fix we upped the ip_conntrack_max value on the dea's
to continue (we were focusing on the Gorouters and HAProxies primarily at
this time). We were running a small number of DEA's, far smaller than we
would run in production, so apps were concentrated on DEA's more than they
should be, and as such we have it logged as something to focus on in more
detail in the near future, but I am interested to see where network
performance tuning discussions go.

John

On Thu, Oct 1, 2015 at 2:05 AM, Joshua McKenty <jmckenty(a)pivotal.io>
wrote:

Amit - I worry about changes to the former in the context of HTTP 1.0
and 1.1, especially without pipelining. What problem are you trying to
solve?

If you’re having trouble initiating new sockets, there are other kernel
params we should adjust.


On Sep 29, 2015, at 5:17 PM, Amit Gupta <agupta(a)pivotal.io> wrote:

Hi all,

I'd like to propose tuning a couple kernel parameters related to tcp
performance:

# TCP_FIN_TIMEOUT
# This setting determines the time that must elapse before TCP/IP can
release a closed connection and reuse
# its resources. During this TIME_WAIT state, reopening the connection
to the client costs less than establishing
# a new connection. By reducing the value of this entry, TCP/IP can
release closed connections faster, making more
# resources available for new connections. Adjust this in the presence
of many connections sitting in the
# TIME_WAIT state:

echo 5 > /proc/sys/net/ipv4/tcp_fin_timeout

# TCP_TW_REUSE
# This allows reusing sockets in TIME_WAIT state for new connections
when it is safe from protocol viewpoint.
# Default value is 0 (disabled). It is generally a safer alternative to
tcp_tw_recycle

echo 1 > /proc/sys/net/ipv4/tcp_tw_reuse

Currently, these parameters are set by certain jobs in cf-release,
diego-release, and perhaps others. Any VM needing to establish a high
number of incoming/outgoing tcp connections in a short period of time will
be unable to establish new connections without changing these parameters.

We believe these parameters are safe to change across the board, and
will be generally beneficial. The existing defaults made sense for much
older networks, but can be greatly optimized for modern systems.

Please share with the mailing lists if you have any questions or
feedback about this proposal. If you maintain a bosh release and would
like to see how these changes would affect your release, you can create a
job which simply does the above in its startup scripts, and colocate that
job with all the other jobs in a deployment of your release.

Thanks,

Amit Gupta
Cloud Foundry PM, OSS Release Integration team