<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">To close the loop on this discussion, I’ve filed the following issue with the gRPC folks:<div><br></div><div><a href="https://github.com/grpc/grpc/issues/35638">https://github.com/grpc/grpc/issues/35638</a></div><div><br></div><div>Thank you again for all of your help. I would not have been able to understand what’s going on without it.</div><div><br id="lineBreakAtBeginningOfMessage"><div><br><blockquote type="cite"><div>On Jan 23, 2024, at 11:43 AM, Brad House <brad@brad-house.com> wrote:</div><br class="Apple-interchange-newline"><div>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<div><p>Yeah, it does clearly show them enqueuing IPv4 and IPv6 requests
separately. So either they need to add logic similar to c-ares
has internally with <a class="moz-txt-link-freetext" href="https://github.com/c-ares/c-ares/pull/551">https://github.com/c-ares/c-ares/pull/551</a> or
just use ares_getaddrinfo() instead of ares_gethostbyname() with
address family AF_UNSPEC and let c-ares do the right thing.</p><p><br>
</p>
<div class="moz-cite-prefix">On 1/23/24 11:25 AM, Nicholas Chammas
wrote:<br>
</div>
<blockquote type="cite" cite="mid:75C8F199-009F-4B78-A5B9-907D85559932@gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
Thank you for all the troubleshooting help, Brad.
<div><br>
</div>
<div>I am using gRPC via Apache Spark Connect (a Python library),
so I am two levels removed from c-ares itself. Looking in the
Python virtual environment where gRPC is installed, I’m not sure
what file to run otool on. The only seemingly relevant file I
could find is called cygrpc.cpython-311-darwin.so, and otool
didn’t turn up anything interesting on it.</div>
<div><br>
</div>
<div>I will take this issue up with the gRPC folks.</div>
<div><br>
</div>
<div>I see in several places that the gRPC folks are using
ares_gethostbyname:</div>
<div>
<ul class="MailOutline">
<li><a href="https://github.com/grpc/grpc/blob/v1.60.0/src/core/lib/event_engine/ares_resolver.cc#L287-L293" moz-do-not-send="true" class="moz-txt-link-freetext">https://github.com/grpc/grpc/blob/v1.60.0/src/core/lib/event_engine/ares_resolver.cc#L287-L293</a></li>
<li><a href="https://github.com/grpc/grpc/blob/v1.60.0/src/core/ext/filters/client_channel/resolver/dns/c_ares/grpc_ares_wrapper.cc#L748-L758" moz-do-not-send="true" class="moz-txt-link-freetext">https://github.com/grpc/grpc/blob/v1.60.0/src/core/ext/filters/client_channel/resolver/dns/c_ares/grpc_ares_wrapper.cc#L748-L758</a></li>
<li><a href="https://github.com/grpc/grpc/blob/v1.60.0/src/core/ext/filters/client_channel/resolver/dns/c_ares/grpc_ares_wrapper.cc#L1075-L1086" moz-do-not-send="true" class="moz-txt-link-freetext">https://github.com/grpc/grpc/blob/v1.60.0/src/core/ext/filters/client_channel/resolver/dns/c_ares/grpc_ares_wrapper.cc#L1075-L1086</a></li>
</ul>
</div>
<div><br>
</div>
<div>
<div><br>
<blockquote type="cite">
<div>On Jan 22, 2024, at 1:39 PM, Brad House
<a class="moz-txt-link-rfc2396E" href="mailto:brad@brad-house.com"><brad@brad-house.com></a> wrote:</div>
<br class="Apple-interchange-newline">
<div>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<div><p>Are you using gRPC installed via homebrew or is it
bundled with something else? Usually package
maintainers like homebrew will dynamically link to the
system versions of dependencies so they can be updated
independently. You might be able to run otool -L on
grpc to see what c-ares library its picking up (and if
none are listed, it might be compiled in statically).</p><p>That said, according to your grpc logs, it appears
that grpc may be itself performing both A and AAAA
queries and expect responses to both of those. I see
the "A" reply comes back but the "AAAA" reply never
comes and it bails at that point. Many years ago
c-ares didn't have a way to request both A and AAAA
records with one query, but does these days via
ares_getaddrinfo(), and it was recently enhanced with
logic to assist in the exact scenario you are seeing,
basically it will stop retrying when at least one
address family is returned. <br>
</p><p>You might need to escalate this to the gRPC folks.<br>
</p>
<div class="moz-cite-prefix">On 1/22/24 12:10 PM,
Nicholas Chammas wrote:<br>
</div>
<blockquote type="cite" cite="mid:CC4282A8-A3BD-448D-A6D6-B03602393F0A@gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
Here’s <a href="https://gist.github.com/nchammas/a4c9873d8158c323796e9b47c064e63a#file-adig-ahost-txt" moz-do-not-send="true">the output of adig and ahost</a>,
both with and without the DNS servers set directly on
the network interface (vs. just on the router).
<div><br>
</div>
<div>I also learned that gRPC 1.60.0 may be using <a href="https://github.com/grpc/grpc/tree/v1.60.0/third_party/cares" moz-do-not-send="true">c-ares 1.19.1</a>, though
again that’s just via looking at the gRPC source and
not via some runtime query.</div>
<div><br id="lineBreakAtBeginningOfMessage">
<div><br>
<blockquote type="cite">
<div>On Jan 21, 2024, at 7:34 AM, Brad House <a class="moz-txt-link-rfc2396E" href="mailto:brad@brad-house.com" moz-do-not-send="true"><brad@brad-house.com></a>
wrote:</div>
<br class="Apple-interchange-newline">
<div>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<div><p>I think homebrew distributes the 'adig'
and 'ahost' utilities from c-ares. Can
you try using those to do the same lookup
so we can see the results?<br>
</p>
<div class="moz-cite-prefix">On 1/19/24
11:01 AM, Nicholas Chammas wrote:<br>
</div>
<blockquote type="cite" cite="mid:D952E55A-2CAA-4E2C-A0DB-549CE2FA936A@gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div><br>
</div>
<div>
<blockquote type="cite">
<div>On Jan 17, 2024, at 3:38 PM, Brad
House <a class="moz-txt-link-rfc2396E" href="mailto:brad@brad-house.com" moz-do-not-send="true"><brad@brad-house.com></a>
wrote:</div>
<div>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<div><p>What version of c-ares is
installed?</p>
</div>
</div>
</blockquote>
<div>Sorry about the delay in
responding. Answering this question is
more difficult than I expected.</div>
<div><br>
</div>
<div>I know that Spark Connect is
running gRPC 1.160.0. Looking through
the gRPC repo, I see mention of <a href="https://github.com/grpc/grpc/blob/v1.60.0/cmake/cares.cmake#L42" moz-do-not-send="true">c-ares 1.13.0</a>,
but I don’t know how that translates
to my runtime. Homebrew tells me I
have c-ares 1.25.0 installed, but
again, I’m not sure if that’s what I’m
actually running.</div>
<div><br>
</div>
<div>Is there a way I can directly query
the version of c-ares being run via
Spark Connect / gRPC? I asked <a href="https://groups.google.com/g/grpc-io/c/3tZCa48Xvh8" moz-do-not-send="true">this question
on the gRPC forum</a> but no
response yet.</div>
<div><br>
</div>
<div>For the record, I know that c-ares
is involved because if I tell gRPC to
not use it (via <a href="https://github.com/grpc/grpc/blob/b34d98fbd47834845e3f9cdaa4aa706f1aa4eddb/doc/environment_variables.md" moz-do-not-send="true">GRPC_DNS_RESOLVER=native</a>)
then my problem disappears.</div>
<blockquote type="cite">
<div>
<div><p>What DNS servers are configured
on your MacOS system when its
not operating properly? The
output of "scutil --dns" would
be helpful here.<br>
</p>
</div>
</div>
</blockquote>
</div>
<a href="https://gist.github.com/nchammas/a4c9873d8158c323796e9b47c064e63a#file-scutil-dns-txt" moz-do-not-send="true">Here’s that
output.</a> I believe 192.168.1.1 is
just my local router, and on there is
where I have the default DNS servers set
to 1.1.1.1 and 1.0.0.1.
<div><br>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
</div>
</div></blockquote></div><br></div></body></html>