Re: DNS caching and forwarding in Cloud Foundry with Consul Agents


Hector Rivas Gandara
 

This all sounds great. Sounds like you just want options to (1) configure
which port Consul DNS binds to, and (2) toggle whether or not it makes
changes to /etc/resolv.conf, at minimum.
Yeah, that would be the minimum required.

Then, I would add:

(3) override the recursors to be used by consul.

With those 3x options, consul can be configured to support any
scenario without having to fork any release.

Is this something you'd want to configure Consul to do, or
would you configure your upstream component like dnsmasq or bind to only hit
the Consul DNS server port when it receives a query for that search domain?
Yes, that should be the pattern.

You also mentioned you don't like Consul's recursor behaviour, and want to
"turn it off". Independently, other people have asked for Consul to be able
to do more recursing, by being able to provide it a list of IPs to recurse
to *in addition* to the ones it pulls out of /etc/resolv.conf. So I see a
few use cases people might have:

(a) make Consul not do any recursing at all, and let the logic live upstream
(in something like dnsmasq or bind) to try a different server when Consul
isn't able to resolve an address
(b) doesn't matter at all what Consul recursing does, because you want to
configure something like dnsmasq to only send queries to Consul when it
matches a certain domain, so no matter how Consul is configured to recurse,
in practice this would never happen
(c) doesn't matter at all what Consul recursing does, because you want to
configure Consul itself to only handle queries which match a certain domain,
so no matter how Consul is configured to recurse, in practice this would
never happen
(d) be able to override the set of recursors Consul uses (instead of what it
reads out of /etc/resolv.conf), so that you could set it to an empty array
or whatever you like -- this is one way to generalize use case (a)
(e) be able to add a set of recursors to the ones Consul reads out of
/etc/resolv.conf, because the stuff in /etc/resolv.conf is determined by the
BOSH Director, and not necessarily something you want to mess with, but may
be something you want to add to, to get over limitations like "MAXNS 3".
See: https://www.pivotaltracker.com/story/show/121335409

If consul-release makes it possible to change the binding port and toggle it
not to modify /etc/resolv.conf, then would you say your use cases falls into
bucket (b)? If so, that's good because it requires fewer additional changes
and also supports consul-release being able to support use case (e) as well
at some point, without conflict.
Yes, I think that would be the case.

IMO I think it is enough to allow the cases b and d. That can be done
by implementing the features 1, 2 and 3 mention before.


(e) be able to add a set of recursors to the ones Consul reads out of
/etc/resolv.conf, because the stuff in /etc/resolv.conf is determined by the
BOSH Director, and not necessarily something you want to mess with, but may
be something you want to add to, to get over limitations like "MAXNS 3".
See: https://www.pivotaltracker.com/story/show/121335409
Rare case and suboptimal.

Be aware that in the current consul implementation, each recursor will
be queried in round robin, with a hardcoded timeout of 2 seconds. So,
if you have 4x recursors, you query might tike 8s to complete! and
given that the default timeout in linux is 5 seconds with 2x retries,
you will have to override that as well.

As I commented in a previous mail, the consul recursor logic is not
smart at all, compared with other agents like dnsmasq, pdnsd or bind.
If somebody wants to customise and improve the resolution logic, they
should look at those agents, not at consul.

Because that, I don't think it makes sense to put an option of "add
new recursors to the list". Instead limit it to optionally specify the
full list of DNS to be configured as recursors (and empty to disable).
"Add recursors" is a rare case and I think it is OK to have to a
little bit of duplication of defining the DNS nameservers twice.



--
Regards
Hector Rivas | GDS / Multi-Cloud PaaS

Join cf-dev@lists.cloudfoundry.org to automatically receive all group messages.