[Daniel's week] March 22, 2024

Fri Mar 22 23:55:08 CET 2024

Hello!

I had a packed week. Here's some of what I did:

## everything curl transition

As I wrote about last week, gitbook.com has made me disappointed recently as
it has repeatedly failed to update content on everything.curl.dev [1] long
after we pushed updates to git. I decided to act on it.

The entire book is written in markdown and switching over to generating the
entire thing with a different tool turned out to be a fairly smooth ride. I
went with mdBook, which also works with the `SUMMARY.md` file in verbatim
form. Generating an entirely new version with the new tool only took me a
little reading of the docs to figure out how to config things appropriately.

I put up my first test shot as a subdirectory on https://curl.se and I had to
scratch my head for a while to figure out why it rendered different there than
it did when I loaded the local file into my browser. The key turned out to be
`Content-Security-Policy` which effectively disabled the JavaScript of the
book. Once I figured out how to remove that header from the book parts of the
site, it worked excellently.

We use the everything.curl.dev name all over for the book, and the CNAME in
DNS has been pointing to the hosting gitbook.com offers. We would need to
change that nonetheless and host something there ourselves. That host could
either be a set of clever redirects to curl.se/[subdir] or we could simply put
the book as generated by me there. I decided to go with the later. I set up a
new webserver, put the generated book contents on it and I made sure
everything.curl.dev gets managed by the general curl CDN config. It offloads
traffic from our origin server and brings the site closer to users.

I figured out how to tell mdBook to "fold" the sections in the left sidebar
similar to how gitbook does it. There are just too many chapters for it to be
manageable to have them all expanded.

When the new render of the book appeared on the site, the tiny and somewhat
subtle changes in the rendering process became obvious. In the previous
version, we could link to "[something]/REAMDE.md" in markdown files, to have
them work in both the book and when browsing the sources in GitHub, but the
new version generates broken links for these. I had to update content
accordingly.

The other important difference turned out to be that mdBook generates lots of
pages/links with `.html` extensions for cases where the previous version did
them without extension. This made a lot of old links trigger 404 responses
until I figured out I could solve this with a bit of Apache config magic
applied .. which checks if there is an access attempt to a file that does not
exist, but there is no `.html` used and then checks if there is a file present
using that same name but with an `.html` extension and then redirects the user
to that! It sounds complicated but seems to work...

Finally, I have setup a cronjob that detects updates to the git repository and
then updates the book site accordingly. The default caching in the CDN site is
60 minutes, so some changes will not appear instantly. I figure this is good
enough for a start. I can always improve details later on.

So in the end, the book site should now look like and work pretty much like it
used to. Currently the site serves about 4GB of content daily.

Thank you Fastly for sponsoring our hosting.

## curl 26

I wrote up a short blog post about curl turning 26 years old this week [2].
Next week there is another kind of anniversary but I won't spoil it by telling
you ahead of time what it is!

## curl up 2024

We have decided to run curl up 2024 [3] in Stockholm and we invite every curl
maintainer and contributor to join us over the weekend May 4-5. I have not
booked the exact venue yet, but I am about to and I will update the
information page as soon as I have.

We are also very keen in getting proposal for agenda items. Talks you want to
do, subjects you want to listen to and topics you want to discuss. And
similar.

The event is free to attend but registration is mandatory.

## distro meeting

The curl distro meeting 2024 [4] happened on March 21. We where around
twenty-five attendees in the meeting and we went over a whole range of
subjects and discussions. I have yet to write up short reported about it and
present to the world, and act on some of the ideas and proposals that were
made during the event.

Thank you everyone who attended. Working together I think we can make curl
better for everyone.

## libcurl option manpages

For command line options the curl manpage specifies exactly which protocols
the options work. For libcurl options that information is usually provided,
but only in plain English but also sometimes not.

This makes it impossible to for example properly generate a list with libcurl
options that are relevant for HTTP. I decided to fix this. With my recent PR
[5] merged, every libcurl man page now lists specifically which protocols that
the option is for. I also made the build fail if the information is missing
for any option, making sure we will maintain this going forward.

This protocol information is not yet really used anywhere, but I intend to use
it to render the protocol information in a consistent way in generated
manpages going forward, and probably offer generated convenient "options for
[PROTOCOL]" lists on the website etc.

In the similar spirit, for all the manpages that now specify that they are for
TLS, there is also a new pending mandatory meta-data field that they must
specify: for which TLS backends the option works [6]. With this information I
can generate a table on the website for all TLS related libcurl options and
detail exactly which TLS backends that support which option. Here again, this
triggers a build failure if the information is missing.

Someone asked me for that information (what options does TLS backend XYZ
support?) the other day and I then realized that it was next to impossible for
us to answer easily and answering it would involve a lot of manual reading or
doing error-prone grepping in docs and code. This is coming to the website now
[10].

But okay, in both the protocol case and the TLS backend case we add this
information in documentation files and there is a risk that they are not in
perfect sync with the code, but I figure with a joint effort we will correct
any remaining mistakes over time.

## curl --help

After it was pointed out that the `curl --help all` output could be enhanced a
bit, I took it upon myself to polish the look. The help texts should be brief
and concise to make the output look decent even in 80 column terminals [7].

But also, by making it adapt properly to wider terminals [8] it can be made to
look even better.

## webinar

Next week I will explain how to get started with libcurl [9] to whoever wants
to listen. It is a beginners tour into the libcurl API basics, how the general
design works and how to write your first Internet tranferring application.

## Coming up

- write up the curl distro report
- curl 8.7.0 release on Wednesday
- webinar on Thursday [9]

## Links

[1] = https://everything.curl.dev/
[2] = https://daniel.haxx.se/blog/2024/03/20/curl-turns-26-today/
[3] = https://github.com/curl/curl-up/wiki/2024
[4] = https://github.com/curl/curl/wiki/curl-distro-discussion-2024/
[5] = https://github.com/curl/curl/pull/13163
[6] = https://github.com/curl/curl/pull/13168
[7] = https://github.com/curl/curl/pull/13169
[8] = https://github.com/curl/curl/pull/13171
[9] = https://daniel.haxx.se/blog/2024/03/18/getting-started-with-libcurl/
[10] = https://curl.se/libcurl/c/tls-options.html

See you next week!

-- 

  / daniel.haxx.se