End-user troubleshooting of bad c-ares interaction with router

[Brad House]

Are you using gRPC installed via homebrew or is it bundled with 
something else?  Usually package maintainers like homebrew will 
dynamically link to the system versions of dependencies so they can be 
updated independently.  You might be able to run otool -L on grpc to see 
what c-ares library its picking up (and if none are listed, it might be 
compiled in statically).

That said, according to your grpc logs, it appears that grpc may be 
itself performing both A and AAAA queries and expect responses to both 
of those.  I see the "A" reply comes back but the "AAAA" reply never 
comes and it bails at that point.  Many years ago c-ares didn't have a 
way to request both A and AAAA records with one query, but does these 
days via ares_getaddrinfo(), and it was recently enhanced with logic to 
assist in the exact scenario you are seeing, basically it will stop 
retrying when at least one address family is returned.

You might need to escalate this to the gRPC folks.

On 1/22/24 12:10 PM, Nicholas Chammas wrote:
> Here’s the output of adig and ahost 
> <https://gist.github.com/nchammas/a4c9873d8158c323796e9b47c064e63a#file-adig-ahost-txt>, 
> both with and without the DNS servers set directly on the network 
> interface (vs. just on the router).
>
> I also learned that gRPC 1.60.0 may be using c-ares 1.19.1 
> <https://github.com/grpc/grpc/tree/v1.60.0/third_party/cares>, though 
> again that’s just via looking at the gRPC source and not via some 
> runtime query.
>
>
>> On Jan 21, 2024, at 7:34 AM, Brad House <brad at brad-house.com> wrote:
>>
>> I think homebrew distributes the 'adig' and 'ahost' utilities from 
>> c-ares.  Can you try using those to do the same lookup so we can see 
>> the results?
>>
>> On 1/19/24 11:01 AM, Nicholas Chammas wrote:
>>>
>>>> On Jan 17, 2024, at 3:38 PM, Brad House <brad at brad-house.com> wrote:
>>>>
>>>> What version of c-ares is installed?
>>>>
>>> Sorry about the delay in responding. Answering this question is more 
>>> difficult than I expected.
>>>
>>> I know that Spark Connect is running gRPC 1.160.0. Looking through 
>>> the gRPC repo, I see mention of c-ares 1.13.0 
>>> <https://github.com/grpc/grpc/blob/v1.60.0/cmake/cares.cmake#L42>, 
>>> but I don’t know how that translates to my runtime. Homebrew tells 
>>> me I have c-ares 1.25.0 installed, but again, I’m not sure if that’s 
>>> what I’m actually running.
>>>
>>> Is there a way I can directly query the version of c-ares being run 
>>> via Spark Connect / gRPC? I asked this question on the gRPC forum 
>>> <https://groups.google.com/g/grpc-io/c/3tZCa48Xvh8> but no response yet.
>>>
>>> For the record, I know that c-ares is involved because if I tell 
>>> gRPC to not use it (via GRPC_DNS_RESOLVER=native 
>>> <https://github.com/grpc/grpc/blob/b34d98fbd47834845e3f9cdaa4aa706f1aa4eddb/doc/environment_variables.md>) 
>>> then my problem disappears.
>>>>
>>>> What DNS servers are configured on your MacOS system when its not 
>>>> operating properly?  The output of "scutil --dns" would be helpful 
>>>> here.
>>>>
>>> Here’s that output. 
>>> <https://gist.github.com/nchammas/a4c9873d8158c323796e9b47c064e63a#file-scutil-dns-txt> I 
>>> believe 192.168.1.1 is just my local router, and on there is where I 
>>> have the default DNS servers set to 1.1.1.1 and 1.0.0.1.
>>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.haxx.se/pipermail/c-ares/attachments/20240122/8c5ba5b2/attachment.htm>