[Daniel's week] June 28, 2024
Daniel Stenberg
daniel at haxx.se
Fri Jun 28 16:26:32 CEST 2024
Hello friends!
In Sweden we have had a lovely summer week these past few days. It has however
not kept me away from the keyboard too much...
## haproxy curl use
On IETF's QUIC mailing list in the thread "Why isn't QUIC growing?", Willy
Tarreau posted [5] the curl version distribution they see in haproxy.org's
website logs.
The six curl versions in the top, the ones with more than 1% of the curl
requests to that domain, are:
1: curl/7.81.0 79.64%
2: curl/7.61.1 5.88%
3: curl/7.68.0 5.47%
4: curl/7.29.0 2.00%
5: curl/8.5.0 1.31%
6: curl/7.74.0 1.01%
Maybe number 4 there is the one to cause an elevated eyebrow or two. 7.29.0
was released in February 2013.
## feature freeze
curl has entered feature freeze for the pending next release. We now only
merge bugfixes until ten days after we ship 8.9.0.
I also decided to move the pending release to one week later (July 24) and I
told the curl-library list about it [15]. To not interfere with my personal
summer vacation planning. The extended release cycle this time will then be
compensated with a shorter one next time.
## user survey PRs
In a moment of clarity, I went back and studied the feedback provided in the
2024 user survey to extract the best "nuggets". It resulted in several new
items added to the TODO list but also a set of new pull requests.
This triggered these four new PRs, all waiting for the feature window to open:
* Add --show-headers as a new alias for --include [1]. Because the former name
does not really tell what it is for. If we do this, the former name will be
deprecated but still working for the foreseeable future.
* Provide a way for --dump-header to write the response headers to stderr [2].
Several users mentioned this desire and it is such a tiny change.
* Add --skip-existing [3]. This new option allows a user to skip a transfer if
the destination file already exists in the file system. Independent of size
or modification date. If the file is present, the transfer is considered
complete.
* --help [command line option] [4]. Makes curl output the manpage documentation
in text format for the given command line option. This one I immediately
appreciated myself when starting to try it out. Since curl already has the
entire manpage internalized for the -M option, providing this function is
"just" a matter of extracting the correct piece for that larger (usually
compressed) chunk of text.
One immediate effect of trying out this help option is that it makes the
stand-alone description of a single option more important. It makes the "see
also" more important etc. It ignited the flame that made me hunt down and
fix a number of different smaller "paper cut" problems in our manpage
generator script managen this week.
## redirect to any port
A fun side-story from my adventures above: several people in the survey
mentioned how they would like --resolve to work with any port number and not
have to specify it in the argument to this option. A fine and reasonable
request I thought and wrote up a first implementation that I also updated and
polished after first feedback round from Stefan Eissing.
Until someone pointed out to me on Mastodon that why do I add all this code
when we can already accomplish this feature with the --connect-to option?
Gah! Indeed, a wildcard catch-all for all hostnames to any port number
redirected to instead connect to a specific IP address can be done with this
option. Now, internally they are not doing exactly the same things, but for a
user who wants this ability I don't think the internal differences are
relevant.
So I could instead discard my branch new pull request and instead I made sure
to polish the documentation for these options to make it a little clearer that
this is possible. Avoiding new code by adding clearer documentation is a good
thing!
## graceful shutdowns
I merged Stefan's final PRs for graceful shutdowns which he has worked on the
last couple of weeks. He wrote up a blog post explainer [6], but the gist is
that by closing down TCP and TLS more carefully, curl reduces the waiting TCP
stacks have to do before ports can be reused for new connections. At least it
should reduce the problems, even if not completely removing them for all
cases.
This is in particular something that will help applications that do many
transfers where many or most of them are closed.
Stefan did a follow-up post [7] that elaborates more on the TCP benefits.
(This job was paid for by a support customer.)
## QNX
I have continued to make progress on doing QNX curl builds. I can now build
what I believe is fairly decent curl release packages for QNX directly from
curl's git master. There are details left to work out in terms of exact
package contents and making sure these packages are done for all the relevant
SDK versions etc but I hope we can soon offer "real" QNX downloads from the
curl site much in the same way we offer Windows downloads [16] already.
## author of the month
May 2024 was the first month since August 2015 in which I did not author the
largest amounts of commits in curl [8]. Viktor Szakats beat me this time. I
think this is nothing but good. Steve Holme used to beat me several months in
the 2014/2015 time frame.
## LTS
The topic of offering one or more long term branches or releases of curl comes
up every now and then and this week I decided to blog about it [9]. There are
no current plans to start any either, but I am not closing the door for us
possibly doing it in a future. It depends on the demand.
## SPNEGO
A customer driven case sent me deep down into the land of SPNEGO
authentication [10], kerberos and Negotiate HTTP headers. This is certainly a
neighborhood I rarely visit and I am no expert in this (weird) authentication
method and associated complexities.
Assuming that we find an business agreement (which seems likely), I expect to
find myself having to dig deeper in this slightly smelly backyard of HTTP
authentication going forward.
## backends
It was brought to my attention that the quiche QUIC library now supports using
the quictls TLS library in addition to BoringSSL. I say "now", because I
became aware now, but they actually merged this support a few months back.
Still, this made me go back and update my HTTP/3 backend picture and when I
did that, I also discovered that the general "libcurl backends" picture was
also slightly out of date.
With both backend images refreshed, I put the new ones into the backends
chapter of everything curl [11].
## faster QUIC
ngtcp2 lead developer Tatsuhiro brought a patch to curl [14] that in a single
blow made HTTP/3 transfers on my machine almost 40% faster (using ngtcp2)!
Admittedly, this is from an already low speed to just slightly less bad.
HTTP/2 still transfers more than twice as fast and HTTP/1 is more than six
times faster on my machine. In this CPU bound case where both the server and
the client are local.
This particular improvement is limited to the ngtcp2 and quiche backends only
(due to how they do UDP transfers), making the relative "poorness" of OpenSSL
QUIC even worse.
## curl on Windows
Windows 10 and 11 now ship curl 8.7.1 since mid June at some point, and I
updated the information on the site [12] about it.
I think there are primarily two things to mention about this upgrade:
* they build with zlib now, so --compressed works. This will now make
copy-as-curl from browsers work better. Like I mentioned five years ago
[13].
* we merged some x509asn1 parsing regressions in 8.7.1 that users now might
run into :-( They have subsequently been fixed, but I don't know how long it
takes our friends in Redmond to react to this.
## failed on top
When we make a pull request on GitHub to propose a change to merge into curl,
they typically trigger a large amount of CI jobs. Right now there are more
than 160 of them. When a PR is submitted, you can see the CI jobs on the page
for the PR but the list showing the jobs is tiny and only shows six entries at
the time. When you are interested in checking the result for a single specific
job, scrolling through that list to find what you are looking for is
mind-numbingly annoying. The order of the CI jobs is not sorted and there can
pop in new entries at any given time, at a seemingly random place in the list.
I brought this feedback (complaint) to GitHub and as a direct result, the curl
project is now opted-in into a feature they call "failed CI jobs at the top".
With this enabled, now all failed CI jobs appear first in the list. They are
by far the most common ones I want to check so this alone is super useful. It
then lists the "not completed yet" jobs and it puts the completed successful
ones at the bottom.
While this still makes it fairly annoying to find a particular job when you
know its name, it is a significant step forward. Only hours after having
gotten this enabled I could feel how my life quality went up several notches.
## Coming up
- QNX packaging work
- SPNEGO deep dive
## Links
[1] = https://github.com/curl/curl/pull/13987
[2] = https://github.com/curl/curl/pull/13992
[3] = https://github.com/curl/curl/pull/13993
[4] = https://github.com/curl/curl/pull/13997
[5] = https://mailarchive.ietf.org/arch/msg/quic/81Wa98MtpGt-idJTWClxQXVury8/
[6] = https://github.com/icing/blog/blob/main/curl-all-ends-well.md
[7] = https://github.com/icing/blog/blob/main/every-end-is-a-beginning.md
[8] = https://curl.se/gitstats/authors.html#author_of_the_month
[9] = https://daniel.haxx.se/blog/2024/06/27/long-term-curl-versions/
[10] = https://en.wikipedia.org/wiki/SPNEGO
[11] = https://everything.curl.dev/internals/backends.html
[12] = https://curl.se/windows/microsoft.html
[13] = https://bugzilla.mozilla.org/show_bug.cgi?id=1605540
[14] = https://github.com/curl/curl/pull/14012
[15] = https://curl.se/mail/lib-2024-06/0062.html
[16] = https://curl.se/windows/
--
/ daniel.haxx.se
More information about the daniel
mailing list