[Daniel's week] January 17, 2025

[Daniel Stenberg]

Hello,

I survived another week. Things happened:

## curl talks

I did an in-person talk this week at a private "cyber security" networking
event here in Stockholm, Sweden where I talked about curl and Open Source and
how all modern digital infrastructure runs on Open Source. It was fun and I
got a lot of good questions and positive feedback.

I appreciate the occasional chance to get out in the real world and meet
people and hear how things work "out there" in the reality. Working from home
is great, but it is nice to sometimes get away from my chair.

Oh, and I did it in all in Swedish, which is rare for me. I mean, not to speak
Swedish because that's my native language, but to do a curl or Open Source
related presentation using it.

I rushed home, only to get online and join the wolfSSL team meeting happening
right now on the North-western coast of the US and did a second but shorter
presentation of the day about "the curl and wolfSSL connection". A brief intro
into how wolfSSL is being used in curl.

## next curl talk

While up to speed I announced my talk for next week: "curl from start to to
end" [1] that I will do on Tuesday January 21st live-streamed on Twitch. No
registration needed, just show up. It will of course also be recorded and made
available after the fact.

This is the first time I do this talk, which is based around the single image
I created a few years ago that tries to explain the entire life-time and
procedures of a curl transfer - in one multi-step illustration.

## HTTPS RR

RFC 9460 [2] documents the HTTPS DNS Resource Record. It works a little
similar to the alt-svc HTTP header [3] and provides a way for a client to
figure out which ALPN ids, HTTP versions really, to try. It saves the client
from having to wait for HTTP headers to return before getting that knowledge
and thus it can in the ideal case try HTTP/3 already in the first request.

Stephen Farrell already added experimental partial HTTPS RR support a while
back as part of his work on experimental ECH support. For simplicity and as a
start he added the HTTPS RR support in the DoH resolver only.

Adding HTTPS RR support for the other resolver backends we have in curl is
certainly more work, in particular for the by-default selected threaded
resolver as that uses the standard getaddrinfo() function which as you might
know does not expose the info nor does it provide any way to get it. For curl
builds using the threaded resolver, we thus need to add a separate request for
the HTTPS RR in parallel, using a separate DNS library (I'll use c-ares [6]).

I suppose we might also need to add a way to explicitly tell curl to *not* use
particular bits of the HTTPS RR data if we would enable this by default.

This week I extended the DoH version of the HTTPS RR support so that it would
parse the received ALPN list and Stefan Eissing tweaked the connection code so
that it can use this externally provided list of ALPN IDs as a clue for how to
do the connection attempts. It seems to work. At least the basics. There is
more to do before it becomes really usable, like we ran into this interesting
challenge with specifying what HTTP versions to allow...

## HTTP versions

Once upon the time, asking for which HTTP version to use in an upcoming
transfer was easy. It was HTTP/1.1 or HTTP/1.0. It was a time of sweetness and
innocence and maybe for a short while there we could almost trick ourselves
into believing that HTTP is simple. This would pass.

When HTTP/2 came, we added a new option "try HTTP/2 but fallback to an older
if you can't use it".

It was good for a while until we realized that upgrading to HTTP/2 on a plain
HTTP transfer is often troublesome, so we added a fourth option that makes
curl only try HTTP/2 if TLS is used. We call it "2TLS". Over time, that turned
out to be what libcurl does by default so that HTTPS connections will try to
negotiate and use HTTP/2. (In addition, we create a "prior knowledge HTTP/2"
option for the cases where you don't actually upgrade, you know it is HTTP/2.
Not used that much.)

In parallel with this development, curl also supports what is commonly known
as HTTP/0.9. It is the style of HTTP responses that were used back in the day
before 1.0 was created. It is a header-less content-only response. Because
this has no headers, basically anything that is not >= HTTP/1 looks like
HTTP/0.9. I mean any other kind of random TCP server. For this reason, we
added another independent option at some point: "HTTP/0.9 allowed", as a
safety precaution so that users will not get tricked when pointing their URL
to random endpoints on the Internet. If 0.9 is not allowed, such a response
thus instead generates an error.

A few years later HTTP/3 came and now we added a way to ask for HTTP/3, which
in reality means try HTTP/3 and HTTP/2+1 in parallel and go with the one that
works first. For conservative reasons and because this is not really the
official way to upgrade to HTTP/3, we have not switched to this by default. We
have stuck with 2TLS.

But in case you REALLY want just HTTP/3 and nothing older, we introduced the
3ONLY option that does not allow any fallback to older versions.

Twenty-five years of contiguous development can do this to a project. As I
trust you have figured out by now, is that the situation is not that simple
anymore. And the story is not over yet.

curl supports alt-svc headers [3], in which a server tells a client where it
can also be accessed, using which HTTP versions etc. The alt-svc responses are
parsed and curl stores an alt-svc cache for subsequent operations. If the
alt-svc cache says a server can speak for example HTTP/3, curl could use it
for that server. But what if the HTTP version option still says 2TLS ? Should
curl then be allowed to upgrade or not?

This week when we started poking on HTTPS RR support, which is yet another way
to get input about suggested HTTP versions we can try, the question popped up
again. Since curl defaults to 2TLS for HTTPS:// connections, should we still
acknowledge the HTTPS RR's suggestion of using HTTP/3 or not?

All this is now stewing in our brains and there is a likely change in the
pipeline. Somehow. Maybe a MIN/MAX setup. Maybe a bitmask style option. Maybe
something else. However we solve it, it feels like we need to come up with a
way that makes it easier for a user, an application, to know and to control
what HTTP version(s) that might be used.

## TLS

(in case you're unfamiliar with the topic, this concerns the Apple TLS
framework called "Secure Transport". Yes it is an unfortunate name.)

My blog post about deprecating Secure Transport [4] this summer took effect
and on the libcurl mailing list there is a follow-up discussion [5] from users
who believe they MUST have Secure Transport support. From the discussion so
far it seems that the primary reason for their need is the number two I
mentioned in my blog post: it gives them easy and convenient use of the iOS
certificate store instead of having to manage a separate one.

My hope is that we can turn this discussion into a development effort and make
sure that whatever TLS backend they switch to, we make sure that one can
access the Apple devices' native certificate stores. I would not be against
slowing down or postponing the deprecation a little if that helps, but I'm not
willing to give up those plans completely. At least not for just this reason.

## Coverity

This week I upgraded my Coverity install so that I could scan curl code using
their latest scanner. The 2024.6.1 version January 2025 update.

This lead to Coverity reporting twelve newly found defects. Apparently one
particular area they have improved in this version is in how it finds and
spots suspected integer overflow operations. In general that seems like a good
idea and I appreciate getting some extra "eyes" on math operations that risk
overflows, but static code analyzers always suffer from having a too narrow
view on the code so it of course warns about "possible overflows" in several
places that can never actually happen because of external circumstances that
the tool does not spot.

Still, at least three of the new remarks were valid flaws that while hard or
almost impossible to trigger, we fixed.

## snprintf horrors

When addressing one of the issues, in our internal printf code I had to update
our printf test cases, and in particular I added tests for a few outputs of
very big floating point numbers. This is relevant because our own printf code
does not actually deal with float outputs itself but hands that over to the
underlying "real" snprintf (or in the worst case even sprintf) implementation.

After I fixed the libcurl printf bug (which could lead to junk output in
extreme edge cases depending on the exact width and precision used for the
float specifier), a single test case still failed in a single CI job. A mingw
build using an older gcc version.

Having a single Windows build failing is of course super annoying because
Windows is such a peculiar platform and I never develop on it myself natively
so it easily becomes a trial and error race on the CI machines. But okay,
fast-forward through a period of hair-tearing and cursing into the void:

Turns out older Windows libc actually has a broken snprintf() implementation
that does not null terminate the output buffer if it reaches the maximum
length provided. Mind-boggling I think since while this turns out to be
documented[7] surely this is not commonly known and if you don't take the
necessary precaution portable applications risk something terrible to trigger
when trying to use a buffer that potentially might lack the null terminator.

## Hackerone

We received our 501st Hackerone report this week and it is being triaged. This
means discussed, dissected, researched and debated. Might be considered a real
security problem. Or not. Some issues are really hard to decide and takes
quite a lot of thinking and back-and-forth before we cane make up our minds
and this is certainly one of those.

Either way, it will be made public later. If not deemed a security problem,
sooner than if deemed to be one. Stay tuned.

## Coming up

- curl from started to end on Tuesday [1]
- curl factory duties all week. Like all weeks

## Links

[1] = https://daniel.haxx.se/blog/2025/01/16/presentation-curl-from-start-to-end/
[2] = https://www.rfc-editor.org/rfc/rfc9460.html
[3] = https://www.rfc-editor.org/rfc/rfc7838.html
[4] = https://daniel.haxx.se/blog/2025/01/14/secure-transport-support-in-curl-is-on-its-way-out/
[5] = https://curl.se/mail/lib-2025-01/0052.html
[6] = https://c-ares.org/
[7] = https://learn.microsoft.com/en-us/cpp/c-runtime-library/reference/snprintf-snprintf-snprintf-l-snwprintf-snwprintf-l?view=msvc-170#remarks

-- 

  / daniel.haxx.se