End-user troubleshooting of bad c-ares interaction with router

[Nicholas Chammas]

Hello,

I am trying to troubleshoot a problem as an end-user of c-ares. I use a library (Apache Spark Connect <https://spark.apache.org/docs/latest/spark-connect-overview.html>) that uses gRPC, which in turn uses c-ares. I am two levels removed from c-ares itself and am a little out of my depth.

I have a little Python script that connects to a remote Apache Spark cluster via Spark Connect and runs a test query. When I run this script on my home network, it takes over 20 seconds to run. When I tether my workstation to my phone (which is connected via LTE), the same script runs in a second or two. In both cases the script runs successfully.

I enabled some gRPC debug flags which print out a lot of information. This led me to c-ares, as I believe the difference in runtime is related somehow to DNS.

I’ve extracted the log lines output by gRPC related to c-ares <https://gist.github.com/nchammas/a4c9873d8158c323796e9b47c064e63a>. (Be sure to scroll down to see both files; there is one for home and one for LTE.) The gRPC codebase is hosted on GitHub, where you can find the grpc_ares_wrapper.cc <https://github.com/grpc/grpc/blob/v1.60.0/src/core/ext/filters/client_channel/resolver/dns/c_ares/grpc_ares_wrapper.cc> file mentioned in the log files.

I tried changing the DNS servers configured in my home router but that didn’t seem to help. Interestingly, however, if I set the same DNS servers already configured in my home router directly on the network interface I’m using, the 20 second delay disappears:

```
networksetup -setdnsservers “My Network" 1.1.1.1 1.0.0.1
```

But this setting doesn’t persist across restarts, and only Spark Connect seems to have this problem. It seems there is some kind of bad interaction between c-ares and my router.

How can I dig deeper to understand what’s going wrong with my home network? I checked the c-ares docs <https://c-ares.org/docs.html> but I don’t see a way for an end-user to enable debug output from c-ares, e.g. via an environment variable.

Any suggestions? I’m running macOS 14.2.1. The router is an Apple AirPort.

Nick

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.haxx.se/pipermail/c-ares/attachments/20240117/7bb6bb0b/attachment.htm>