<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Just glanced at the issue, looks like you summarized it well.<br>
</p>
<div class="moz-cite-prefix">On 1/23/24 12:58 PM, Nicholas Chammas
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:8B63880C-773E-440A-AEAA-DE3611DC4A4D@gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
To close the loop on this discussion, I’ve filed the following
issue with the gRPC folks:
<div><br>
</div>
<div><a href="https://github.com/grpc/grpc/issues/35638"
moz-do-not-send="true" class="moz-txt-link-freetext">https://github.com/grpc/grpc/issues/35638</a></div>
<div><br>
</div>
<div>Thank you again for all of your help. I would not have been
able to understand what’s going on without it.</div>
<div><br id="lineBreakAtBeginningOfMessage">
<div><br>
<blockquote type="cite">
<div>On Jan 23, 2024, at 11:43 AM, Brad House
<a class="moz-txt-link-rfc2396E" href="mailto:brad@brad-house.com"><brad@brad-house.com></a> wrote:</div>
<br class="Apple-interchange-newline">
<div>
<meta http-equiv="Content-Type"
content="text/html; charset=UTF-8">
<div>
<p>Yeah, it does clearly show them enqueuing IPv4 and
IPv6 requests separately. So either they need to add
logic similar to c-ares has internally with <a
class="moz-txt-link-freetext"
href="https://github.com/c-ares/c-ares/pull/551"
moz-do-not-send="true">https://github.com/c-ares/c-ares/pull/551</a>
or just use ares_getaddrinfo() instead of
ares_gethostbyname() with address family AF_UNSPEC and
let c-ares do the right thing.</p>
<p><br>
</p>
<div class="moz-cite-prefix">On 1/23/24 11:25 AM,
Nicholas Chammas wrote:<br>
</div>
<blockquote type="cite"
cite="mid:75C8F199-009F-4B78-A5B9-907D85559932@gmail.com">
<meta http-equiv="content-type"
content="text/html; charset=UTF-8">
Thank you for all the troubleshooting help, Brad.
<div><br>
</div>
<div>I am using gRPC via Apache Spark Connect (a
Python library), so I am two levels removed from
c-ares itself. Looking in the Python virtual
environment where gRPC is installed, I’m not sure
what file to run otool on. The only seemingly
relevant file I could find is
called cygrpc.cpython-311-darwin.so, and otool
didn’t turn up anything interesting on it.</div>
<div><br>
</div>
<div>I will take this issue up with the gRPC folks.</div>
<div><br>
</div>
<div>I see in several places that the gRPC folks are
using ares_gethostbyname:</div>
<div>
<ul class="MailOutline">
<li><a
href="https://github.com/grpc/grpc/blob/v1.60.0/src/core/lib/event_engine/ares_resolver.cc#L287-L293"
moz-do-not-send="true"
class="moz-txt-link-freetext">https://github.com/grpc/grpc/blob/v1.60.0/src/core/lib/event_engine/ares_resolver.cc#L287-L293</a></li>
<li><a
href="https://github.com/grpc/grpc/blob/v1.60.0/src/core/ext/filters/client_channel/resolver/dns/c_ares/grpc_ares_wrapper.cc#L748-L758"
moz-do-not-send="true"
class="moz-txt-link-freetext">https://github.com/grpc/grpc/blob/v1.60.0/src/core/ext/filters/client_channel/resolver/dns/c_ares/grpc_ares_wrapper.cc#L748-L758</a></li>
<li><a
href="https://github.com/grpc/grpc/blob/v1.60.0/src/core/ext/filters/client_channel/resolver/dns/c_ares/grpc_ares_wrapper.cc#L1075-L1086"
moz-do-not-send="true"
class="moz-txt-link-freetext">https://github.com/grpc/grpc/blob/v1.60.0/src/core/ext/filters/client_channel/resolver/dns/c_ares/grpc_ares_wrapper.cc#L1075-L1086</a></li>
</ul>
</div>
<div><br>
</div>
<div>
<div><br>
<blockquote type="cite">
<div>On Jan 22, 2024, at 1:39 PM, Brad House <a
class="moz-txt-link-rfc2396E"
href="mailto:brad@brad-house.com"
moz-do-not-send="true"><brad@brad-house.com></a>
wrote:</div>
<br class="Apple-interchange-newline">
<div>
<meta http-equiv="Content-Type"
content="text/html; charset=UTF-8">
<div>
<p>Are you using gRPC installed via homebrew
or is it bundled with something else?
Usually package maintainers like homebrew
will dynamically link to the system
versions of dependencies so they can be
updated independently. You might be able
to run otool -L on grpc to see what c-ares
library its picking up (and if none are
listed, it might be compiled in
statically).</p>
<p>That said, according to your grpc logs,
it appears that grpc may be itself
performing both A and AAAA queries and
expect responses to both of those. I see
the "A" reply comes back but the "AAAA"
reply never comes and it bails at that
point. Many years ago c-ares didn't have
a way to request both A and AAAA records
with one query, but does these days via
ares_getaddrinfo(), and it was recently
enhanced with logic to assist in the exact
scenario you are seeing, basically it will
stop retrying when at least one address
family is returned. <br>
</p>
<p>You might need to escalate this to the
gRPC folks.<br>
</p>
<div class="moz-cite-prefix">On 1/22/24
12:10 PM, Nicholas Chammas wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CC4282A8-A3BD-448D-A6D6-B03602393F0A@gmail.com">
<meta http-equiv="content-type"
content="text/html; charset=UTF-8">
Here’s <a
href="https://gist.github.com/nchammas/a4c9873d8158c323796e9b47c064e63a#file-adig-ahost-txt"
moz-do-not-send="true">the output of
adig and ahost</a>, both with and
without the DNS servers set directly on
the network interface (vs. just on the
router).
<div><br>
</div>
<div>I also learned that gRPC 1.60.0 may
be using <a
href="https://github.com/grpc/grpc/tree/v1.60.0/third_party/cares"
moz-do-not-send="true">c-ares 1.19.1</a>,
though again that’s just via looking at
the gRPC source and not via some runtime
query.</div>
<div><br
id="lineBreakAtBeginningOfMessage">
<div><br>
<blockquote type="cite">
<div>On Jan 21, 2024, at 7:34 AM,
Brad House <a
class="moz-txt-link-rfc2396E"
href="mailto:brad@brad-house.com" moz-do-not-send="true"><brad@brad-house.com></a>
wrote:</div>
<br
class="Apple-interchange-newline">
<div>
<meta http-equiv="Content-Type"
content="text/html; charset=UTF-8">
<div>
<p>I think homebrew distributes
the 'adig' and 'ahost'
utilities from c-ares. Can
you try using those to do the
same lookup so we can see the
results?<br>
</p>
<div class="moz-cite-prefix">On
1/19/24 11:01 AM, Nicholas
Chammas wrote:<br>
</div>
<blockquote type="cite"
cite="mid:D952E55A-2CAA-4E2C-A0DB-549CE2FA936A@gmail.com">
<meta
http-equiv="content-type"
content="text/html; charset=UTF-8">
<div><br>
</div>
<div>
<blockquote type="cite">
<div>On Jan 17, 2024, at
3:38 PM, Brad House <a
class="moz-txt-link-rfc2396E" href="mailto:brad@brad-house.com"
moz-do-not-send="true"><brad@brad-house.com></a>
wrote:</div>
<div>
<meta
http-equiv="Content-Type" content="text/html; charset=UTF-8">
<div>
<p>What version of
c-ares is installed?</p>
</div>
</div>
</blockquote>
<div>Sorry about the delay
in responding. Answering
this question is more
difficult than I expected.</div>
<div><br>
</div>
<div>I know that Spark
Connect is running gRPC
1.160.0. Looking through
the gRPC repo, I see
mention of <a
href="https://github.com/grpc/grpc/blob/v1.60.0/cmake/cares.cmake#L42"
moz-do-not-send="true">c-ares
1.13.0</a>, but I don’t
know how that translates
to my runtime. Homebrew
tells me I have c-ares
1.25.0 installed, but
again, I’m not sure if
that’s what I’m actually
running.</div>
<div><br>
</div>
<div>Is there a way I can
directly query the version
of c-ares being run via
Spark Connect / gRPC? I
asked <a
href="https://groups.google.com/g/grpc-io/c/3tZCa48Xvh8"
moz-do-not-send="true">this
question on the gRPC
forum</a> but no
response yet.</div>
<div><br>
</div>
<div>For the record, I know
that c-ares is involved
because if I tell gRPC to
not use it (via <a
href="https://github.com/grpc/grpc/blob/b34d98fbd47834845e3f9cdaa4aa706f1aa4eddb/doc/environment_variables.md"
moz-do-not-send="true">GRPC_DNS_RESOLVER=native</a>)
then my problem
disappears.</div>
<blockquote type="cite">
<div>
<div>
<p>What DNS servers
are configured on
your MacOS system
when its not
operating properly?
The output of
"scutil --dns" would
be helpful here.<br>
</p>
</div>
</div>
</blockquote>
</div>
<a
href="https://gist.github.com/nchammas/a4c9873d8158c323796e9b47c064e63a#file-scutil-dns-txt"
moz-do-not-send="true">Here’s
that output.</a> I believe
192.168.1.1 is just my local
router, and on there is where
I have the default DNS servers
set to 1.1.1.1 and 1.0.0.1.
<div><br>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
</body>
</html>