<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">Thank you for all the troubleshooting help, Brad.<div><br></div><div>I am using gRPC via Apache Spark Connect (a Python library), so I am two levels removed from c-ares itself. Looking in the Python virtual environment where gRPC is installed, I’m not sure what file to run otool on. The only seemingly relevant file I could find is called cygrpc.cpython-311-darwin.so, and otool didn’t turn up anything interesting on it.</div><div><br></div><div>I will take this issue up with the gRPC folks.</div><div><br></div><div>I see in several places that the gRPC folks are using ares_gethostbyname:</div><div><ul class="MailOutline"><li><a href="https://github.com/grpc/grpc/blob/v1.60.0/src/core/lib/event_engine/ares_resolver.cc#L287-L293">https://github.com/grpc/grpc/blob/v1.60.0/src/core/lib/event_engine/ares_resolver.cc#L287-L293</a></li><li><a href="https://github.com/grpc/grpc/blob/v1.60.0/src/core/ext/filters/client_channel/resolver/dns/c_ares/grpc_ares_wrapper.cc#L748-L758">https://github.com/grpc/grpc/blob/v1.60.0/src/core/ext/filters/client_channel/resolver/dns/c_ares/grpc_ares_wrapper.cc#L748-L758</a></li><li><a href="https://github.com/grpc/grpc/blob/v1.60.0/src/core/ext/filters/client_channel/resolver/dns/c_ares/grpc_ares_wrapper.cc#L1075-L1086">https://github.com/grpc/grpc/blob/v1.60.0/src/core/ext/filters/client_channel/resolver/dns/c_ares/grpc_ares_wrapper.cc#L1075-L1086</a></li></ul></div><div><br></div><div><div><br><blockquote type="cite"><div>On Jan 22, 2024, at 1:39 PM, Brad House <brad@brad-house.com> wrote:</div><br class="Apple-interchange-newline"><div>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<div><p>Are you using gRPC installed via homebrew or is it bundled with
something else? Usually package maintainers like homebrew will
dynamically link to the system versions of dependencies so they
can be updated independently. You might be able to run otool -L
on grpc to see what c-ares library its picking up (and if none are
listed, it might be compiled in statically).</p><p>That said, according to your grpc logs, it appears that grpc may
be itself performing both A and AAAA queries and expect responses
to both of those. I see the "A" reply comes back but the "AAAA"
reply never comes and it bails at that point. Many years ago
c-ares didn't have a way to request both A and AAAA records with
one query, but does these days via ares_getaddrinfo(), and it was
recently enhanced with logic to assist in the exact scenario you
are seeing, basically it will stop retrying when at least one
address family is returned. <br>
</p><p>You might need to escalate this to the gRPC folks.<br>
</p>
<div class="moz-cite-prefix">On 1/22/24 12:10 PM, Nicholas Chammas
wrote:<br>
</div>
<blockquote type="cite" cite="mid:CC4282A8-A3BD-448D-A6D6-B03602393F0A@gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
Here’s <a href="https://gist.github.com/nchammas/a4c9873d8158c323796e9b47c064e63a#file-adig-ahost-txt" moz-do-not-send="true">the output of adig and ahost</a>, both
with and without the DNS servers set directly on the network
interface (vs. just on the router).
<div><br>
</div>
<div>I also learned that gRPC 1.60.0 may be using <a href="https://github.com/grpc/grpc/tree/v1.60.0/third_party/cares" moz-do-not-send="true">c-ares 1.19.1</a>, though again that’s
just via looking at the gRPC source and not via some runtime
query.</div>
<div><br id="lineBreakAtBeginningOfMessage">
<div><br>
<blockquote type="cite">
<div>On Jan 21, 2024, at 7:34 AM, Brad House
<a class="moz-txt-link-rfc2396E" href="mailto:brad@brad-house.com"><brad@brad-house.com></a> wrote:</div>
<br class="Apple-interchange-newline">
<div>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<div><p>I think homebrew distributes the 'adig' and 'ahost'
utilities from c-ares. Can you try using those to do
the same lookup so we can see the results?<br>
</p>
<div class="moz-cite-prefix">On 1/19/24 11:01 AM,
Nicholas Chammas wrote:<br>
</div>
<blockquote type="cite" cite="mid:D952E55A-2CAA-4E2C-A0DB-549CE2FA936A@gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div><br>
</div>
<div>
<blockquote type="cite">
<div>On Jan 17, 2024, at 3:38 PM, Brad House <a class="moz-txt-link-rfc2396E" href="mailto:brad@brad-house.com" moz-do-not-send="true"><brad@brad-house.com></a>
wrote:</div>
<div>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<div><p>What version of c-ares is installed?</p>
</div>
</div>
</blockquote>
<div>Sorry about the delay in responding. Answering
this question is more difficult than I expected.</div>
<div><br>
</div>
<div>I know that Spark Connect is running gRPC
1.160.0. Looking through the gRPC repo, I see
mention of <a href="https://github.com/grpc/grpc/blob/v1.60.0/cmake/cares.cmake#L42" moz-do-not-send="true">c-ares 1.13.0</a>, but I
don’t know how that translates to my runtime.
Homebrew tells me I have c-ares 1.25.0 installed,
but again, I’m not sure if that’s what I’m
actually running.</div>
<div><br>
</div>
<div>Is there a way I can directly query the version
of c-ares being run via Spark Connect / gRPC? I
asked <a href="https://groups.google.com/g/grpc-io/c/3tZCa48Xvh8" moz-do-not-send="true">this question on the gRPC
forum</a> but no response yet.</div>
<div><br>
</div>
<div>For the record, I know that c-ares is involved
because if I tell gRPC to not use it (via <a href="https://github.com/grpc/grpc/blob/b34d98fbd47834845e3f9cdaa4aa706f1aa4eddb/doc/environment_variables.md" moz-do-not-send="true">GRPC_DNS_RESOLVER=native</a>)
then my problem disappears.</div>
<blockquote type="cite">
<div>
<div><p>What DNS servers are configured on your
MacOS system when its not operating
properly? The output of "scutil --dns"
would be helpful here.<br>
</p>
</div>
</div>
</blockquote>
</div>
<a href="https://gist.github.com/nchammas/a4c9873d8158c323796e9b47c064e63a#file-scutil-dns-txt" moz-do-not-send="true">Here’s that output.</a> I
believe 192.168.1.1 is just my local router, and on
there is where I have the default DNS servers set to
1.1.1.1 and 1.0.0.1.
<div><br>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
</div>
</div></blockquote></div><br></div></body></html>