[Daniel's week] July 5, 2024

[Daniel Stenberg]

Hello.

The final week before my vacation was another full week...

## language

I happened to take a glance on something on the curl source code earlier this 
when it occurred to me that we used language there in ways we do not otherwise 
allow in curl documentation: words, contractions etc. (for example, we have 
agreed to write "username" everywhere in documentation, but some code still 
talked about "user name").

It made me run my cleanup script on the entire source code and then to make 
sure this is correct and that we will not reintroduce any of this in the 
future I expanded the "badwords" CI job to verify. I had to "tone down" the 
source code check a little though: for example, in the documentation we ban 
the use of the word "will" (to force the language to use present tense) but it 
is used literally everywhere in source code comments that I could not muster 
to change those. I also don't think it is such bad word in comments.

I then extended the badwords CI job for the website to make sure that we stick
to the same language better there as well.

## website

Having fiddled a little with the site I decided I should try to maybe improve
the huge blob of features that we listed for curl on the front page [1]. It
was always mostly a way to brag but also maybe to make sure that search
engines would know to list our site if someone would search for such terms.

I decided to make the list of features a little more structured and put into
categories, so I made a table with categories on the left and listed a number
of features curl can do within those categories on the right. Doing this, I
came to think of several features not previously mentioned there so I added
them as well. I think it turned out rather good and merge it.

As usual: I am not really a web developer, you just have to live with this
design until someone else makes it better.

While poking on the website front page, I also decided to tweak the right-side
boxes a little to make them more distinctly separate from the main content,
use the same font size, add some images and reduce the number of words used in
them.

There is now a small white curl logo symbol in the top-left corner that is
visible on every page on the site. Unless the display is narrow.

The page on the site that lists all available libcurl functions [2] was always
manually edited, which several times in the past has made it lack a few
functions. Now I wanted to add the short descriptions for every function so I
wrote a script to generate the list directly the from the docs source code. It
made me realize that the short descriptions for the functions needed polish to
be better (clearer) - and consistent. So I fixed that as well.

Having improved that list of libcurl functions, I clicked around a little on
the libcurl docs on the site and I noticed that several of the pages lacked
the "related" box on the top right side that we otherwise try to have on every
page. So I added lots of new cross-links for several libcurl pages. Should
make it easier for users to find related documentation.

## wcurl

Samuel Henrique announced [6] that the Debian project shipped a new tool 
called wcurl [3] included in the curl package when they shipped their curl 
8.8.0-2 package this week.

wcurl is intended to mimic wget when used with only a URL: simply download the 
URL into a local file.

The wcurl announcement got a lot of attention. I saw many people express joy 
saying they have longed for something like this, but of course also several 
people asking why this is "necessary" when wget already exists and the third 
group of people who instantly figured out things that wcurl currently cannot 
do, that curl or wget can.

There was instantly a number of requests that the curl project should adopt 
this tool [4] to more easily allow more distros and user to get their hands on 
it. There are no real objections against such a move, but we are also not in a 
hurry so we can let it first try its wings and see how it fares where it is 
now [5].

It quickly appeared elsewhere too; like in MacPorts [16], Alpine [18] and Arch
[17].

To be continued.

## HTTP/3 on Debian

The very same Samuel Henrique also announced [7] that curl 8.8.0 on Debian
unstable now supports HTTP/3. They accomplish this primarily by building with
GnuTLS instead of OpenSSL [8] together with ngtcp2 and nghttp3.

This is likely to cause some friction and perhaps some new subtle bugs among
users as this package has been using OpenSSL for such a long time. I am
curious to see how it will fare. I hope readers of this jump on this bandwagon
to try it out and will tell the team if there are any remaining issues to fix.

Early curl 8.8.0-GnuTLS users on Debian seem to already have run into some
x509 bugs. Fortunately, so far they have only hit bugs we already fixed
upstream lined up for getting shipped in 8.9.0. I believe the Debian team
might backport the fixes for the bugs that bite users.

## Faster HTTP/3

Tatsuhiro Tsujikawa provided a fix [19] for improving curl's HTTP/3 transfer
speeds. Two out of the four backends got a huge bump in performance by
enabling the use of UDP GRO. In a basic measurement of mine, I saw HTTP/3
transfers using ngtcp2+quictls run roughly 39% faster on localhost (CPU
bound).

Even after this improvement, HTTP/3 is still *significantly* slower than
HTTP/2 and HTTP/1 for the same workloads on the same hardware but this is a
good boost in the right direction - we just need to keep more of them coming.
Ironically, this made OpenSSL-QUIC relatively even worse than before since the
nature of their API makes it so that we can't speed it up it with GRO
ourselves but they need to work it out and enable it themselves, if it is
possible.

If someone would build ngtcp2 with different TLS backends and compare relative
HTTP/3 performance, I think it could make for an interesting read.

## categories

The curl --help output uses "categories" as the key mechanism to split all
options into sub sections. Every command line option is specified to be part
of one or more categories and curl -h can list the categories and it can list
all the options within such a category.

This system was introduced in July 2020, shipped first in curl 7.73.0.

This week I decided to review the categories and how options are put into
them. I found that there was a lot of room for improvements.

In a new PR [9] this week, I did a huge overhaul and cleanup of the
categories: I removed a few, I added a few new and I modified the category
descriptions.

Further, I added and moved categories for a whole range of options as I think
some of them had ended up slightly incorrect - mostly I believe because we
(I?) have not paid enough attention to these details.

Presumably this change will not be much noticed, but should ideally just feel
better when used and provide information better.

## QNX

This has been in the works for quite some time but I am happy to finally be
able to announce start we now offer official curl builds for QNX [10].

This is a commercial cooperation with Blackberry who is the company behind the
QNX operating.

The plan is to ship curl for QNX packages in sync with the normal releases to
the benefit of all users of that operating system.

All source code needed for these builds are upstream already and all the build 
scripts and configs necessary for building these releases are provided on 
GitHub [11] - full transparency.

>From my point of view this is a perfect collboration that is a big win for 
both sides. We get a customer paying for us to maintain curl for their target. 
We do it better, faster and probably cheaper than they can manage themselves. 
They get frequent and updated (secure!) curl releases with full support and 
everything based on pristine upstream source code.

Ideally, other operating system manufacturers will realize how awesome this 
setup is as well.

## more LTS

Last week I blogged about how curl does not provide LTS release or maintain
any long-term branches [12] and I mentioned in my email as well.

This week I want to especially highlight and quote a comment I received on the
blog, that I think is so good and valuable that I have selected to include it
here for you my friends.

This is written by Willy Tarreau, known for being the founder and lead
developer of haxproxy ]13]. He makes rather strong case for offering and
maintaining some sort of long-term support branches. Certainly food for
thought.

Please let me share my experience of LTS on HAProxy. What we’ve found over
time is that it takes a little bit time but far less than just chasing and
fixing bugs (which is needed anyway). Doing maintenance releases essentially
requires to backport existing fixes, which is not very long. Also the farther
the LTS branch is from mainline, the least likely it is that you’ll need to
backport a fix. We maintain our LTS versions for 5 years after release and the
non-LTS ones one year, so we effectively have ~6-7 versions at a time. Most
often the oldest ones barely see more than a fix per week in the last year
because almost nobody’s using them anymore, and we know that their users value
extreme stability so let’s not backport risky fixes for bugs that affect
nobody.

We’re also fairly open to backport suggestions from the users (best way to
deliver ultra-stable versions after all), provided these don’t come with a
risk. We tend to accept to backport tiny useful features to the latest LTS
branch when that makes users’ lives easier, but never beyond. A new feature
*is* one reason to force an upgrade. This works very very well. Usually in 6
months after an LTS release is emitted, many of the users have switched to it
(based on bug reports, which can be skewed by the increased number of bugs in
any new version), then maybe 30% for the previous one, 10-15% for the one
before, and the rest being within noise. For distros it’s quite convenient,
because we know they just pick select patches, and these 5 years of
maintenance help them get patches close to their version.

What I noticed overall is that there is a sort of moral contract with users
that increases both trust and feature delivery: by being extremely careful not
to break old LTS releases, we definitely have a part of the user base, the one
most sensitive to bugs, that is perfectly secured by running slow-moving
releases. These ones almost never face a regression, and if one ever happens
due to a problematic fix, we have no problem instantly emitting another
version with only this issue fixed. This allows us to be a bit more aggressive
on recent versions. The latest stable that is not LTS can move a little bit
and receive a few occasional backports for harmless popular features. This way
the share of the population which values features more than stability tests
new features early and shares interesting feedback that allows to improve
these features before they’re widely adopted. And in parallel sensitive users
almost never face a breakage. This means that the pressure on the development
team caused by bug reports usually is extremely low: bugs being worked on
generally do not affect users in a critical way. It even happens that we
develop new debugging features to get more info on a bug, and users generally
are happy to participate to the bug resolution because it’s not critical for
them either.

Before we had this LTS cycle, I was regularly forced to integrate small
features into the “current” branch along with fixes, and users were extremely
reluctant to upgrade, even to fix a bug, due to the risk that a change would
cause them some trouble. This fear has totally disappeared. We even indicate
on a page the list of bugs affecting each and every version, that we use to
suggest users to upgrade when they report an issue.

Overall I find that this process works extremely well. The only difficulty
with old LTS versions can be the tooling. For example when you build a 5-year
old version with your shiny new laptop that comes with a shiny new compiler
and you discover that it’s no longer supported by that old version, it’s quite
annoying. But that doesn’t happen often, and we all have a collection of
native and cross-compilers anyway when working on portable projects.

In our case we alternate non-LTS and LTS (2 versions total per year). Odd
versions are not LTS while even ones are LTS, so that users know upfront which
one will have what status. And it’s reminded in the version output, with an
indication of the date of end of support. This has also helped make older
versions disappear completely.

I can only encourage you to go into that direction. The beginning can be more
difficult than the long term due to the change of habits, and most users
initially using the same LTS version with different expectations. But after a
year, the fog will dissipate and you’ll start to see different groups on
different versions, then this will become cool.

Hoping this can help you in your decision.

/ Willy

## pycon 2023 keynote

My keynote presentation "you can do it" from Pycon 2023 [14] was finally made
available on YouTube - completely without python contents. I have got a lot of
positive response on this one, and keen observers are correct in spotting that
it is almost identical to the talk I later did at FOSDEM 2024; then called
"you too could have made curl" [15].

## Coming up

  - I'm taking a three week vacation starting now. I will not be completely
    without access to email and keyboards but I will certainly be slower to
    respond than otherwise. I intend to send at out short weekly emails even
    during my vacation.

## Links

[1] = https://curl.se/
[2] = https://curl.se/libcurl/c/allfuncs.html
[3] = https://daniel.haxx.se/blog/2024/07/03/wcurl-is-here/
[4] = https://curl.se/mail/lib-2024-07/0000.html
[5] = https://github.com/Debian/wcurl
[6] = https://samueloph.dev/blog/announcing-wcurl-a-curl-wrapper-to-download-files/
[7] = https://samueloph.dev/blog/debian-curl-now-supports-http3/
[8] = https://daniel.haxx.se/blog/2024/06/10/http-3-in-curl-mid-2024/
[9] = https://github.com/curl/curl/pull/14101
[10] = https://daniel.haxx.se/blog/2024/07/05/curl-for-qnx/
[11] = https://github.com/curl/curl-for-qnx
[12] = https://daniel.haxx.se/blog/2024/06/27/long-term-curl-versions/
[13] = https://www.haproxy.org/
[14] = https://youtu.be/p-8of5cpspo?si=gWcQvTAhkCI4tgAU
[15] = https://daniel.haxx.se/blog/2024/02/06/fosdem-2024-you-too-could-have-made-curl/
[16] = https://ports.macports.org/port/wcurl/
[17] = https://aur.archlinux.org/packages/wcurl-git
[18] = https://pkgs.alpinelinux.org/package/edge/testing/armv7/wcurl
[19] = https://github.com/curl/curl/pull/14012


-- 

  / daniel.haxx.se