[Daniel's week] December 22, 2023

[Daniel Stenberg]

Hello friends,

I kept myself occupied another week.

## everything curl

I must say that the new PDF version of Everything curl [3] that I fixed last
week turned out to be a nice way for me to read through parts of the book
again and when doing that I found lots of tiny mistakes and places that needed
updates. Which subsequently made me merge two dozen commits or something in
the beginning of the week.

## rtos

The deal with a company doing a commercial RTOS has gone through. I'm still
awaiting on some practical details to be arranged properly before my work can
begin for real. I'll spill the beans properly once I do. Presumably in the
first weeks of next year. "The deal" means that I will work on making curl run
smoothly on their OS and provide binary packages for download on curl.se to
users of that OS.

## printf

In the libcurl code we use a lot of printf-style calls, and all of them use
our internal printf implementation. I have preciously sped up a lot of
functions by *avoiding* using printf and instead rather use other functions
but this week I decided to take a closer look under this hood. Lots of libcurl
code still uses and will remain using printf in various forms. The printf
implementation is mostly code that originates from an implementation I and
Bjorn Reese wrote before I started on curl. Old is not strong enough to
describe it. Lots (most?) of the lines in that source code file were older
than twenty years according to git blame.

The internal implementation was originally written to make sure we can use
snprintf() on all platforms, since once upon the time it was not universally
available. It then became a way for us to make sure out printf uses work
exactly the same everywhere. I also had the (bad) idea to add the printf
functions to the public libcurl API so applications can use them. I consider
it a bad idea partly because it means we are stuck having to support a full
fledged printf even if we don't use all the features internally and removing
some of them could certainly be beneficial for performance. Another reason it
was a bad idea is that it is not transfer related and libcurl should not
provide APIs for non-transfer activities.

I started by writing a small application that uses a lot of snprintf() calls
with many %s, %d, %c etc. Basically lots of the most commonly used
%-specifiers with most common version (for libcurl code) of printf calls. I
then measured how fast it ran when using current libcurl as well as how it
compared to the glibc version of snprintf. It turned out that libcurl was
mostly on part with glibc, speed-wise.

I then switched to a debug-build version of libcurl and ran the same code
again and used that as the base metric. That test code did 1.4 million printf
calls at an average of 821 nanoseconds per snprintf (in a best of five runs).

I tested various tweaks and polished away things I thought could have an
impact, and I reran the speed test between every tweak. If the tweak seemed to
have an impact, even if very small, I kept it and moved on to the next little
thing. This way, over a few hours of poking and iterating over lots of smaller
edits, I had squeezed the time down to 675 nanoseconds with no functional
difference. Not too shabby!

Quite happy with this, I added a few new test cases to the test suite and
boom, now I realized that for a particular use case there was a rather ugly
discrepancy between what the curl printf code did and how glibc (and probably
POSIX) behaves. I pondered on ways to fix this problem that involved how to do
when a %-qualifier is used but with no legal printf character following. Like
%K or %r.

It was one of those challenges that was hard to let go and it kept my brain
spinning for a night and the morning after I knew what I had to do: I need to
do a rather drastic rewrite of the code so that it would more clearly 1) parse
the format string 2) generate a list of input arguments and their types 3)
generate a list of output segments and their formatting options and 4) a
function that combines and generates the output based on the previous steps.

It sounds simple, but the existing code did not do the stages separated this
distinctly which made it hard to fix that bug.

Luckily we have a lot of printf test cases in the curl test suite and since
curl uses printf a lot internally, I could basically hammer on the code,
re-run the test suite, fix the problems and then try again. Over and over
until all the test cases would report success.

When all the test cases ran fine again, I held my breath and reran the
performance test again. Would the refactor perhaps ruin the improvements I
previously achieved? Nope. The new version clocked in at 650 nanoseconds and
after some further polish I timed it at 619.

I then spent some additional time at editing the code for style, renamed some
functions and variables etc to make the result "nice code". If I may say so,
the new version of the printf code is a much easier read. This is code that is
not hard to understand.

Of course I still fear that there is some tiny detail I got wrong and that we
have missed to add a test case for, but the code [2] is merged now and no
static code analyzer or fuzzer has yet reported any problem with it. Maybe,
just maybe, the ducks were mostly in a row this time!

## h3 audit

Trail of Bits have worked on this security audit of the HTTP/3 components in
curl and on the Thursday we got an update and progress report from them. They
have not found any alarming problems which certainly is good, but they did
spot several areas with room for improvements. I will mention one of them
below.

The audit continues a little bit more and then they will work on a report that
will be presented next year, and if they find any security problems those
problems will of course have to be handled correctly and responsibly before
any such details can go public.

I will of course tell you all about these thins as soon as I can.

## fuzzing

Trail of Bits pointed out that curl's fuzzing coverage on OSS-Fuzz shrunk
dramatically A YEAR AGO and nobody has noticed during all this time and this
is a real concern since who knows what possible flaws we have missed due to
this.

It turns out the coverage drop was because of a build flaw that made it run
without TLS enabled. Once we fixed this problem, the coverage rate increased
back up to similar levels as before [1] and luckily enough it seems it could
not find any new problems even though it had not been doing its full job for
an extended period of time.

Needless to say, we are now pondering what we should do to avoid this
happening again. What support oss-fuzz can help us to notice this or what kind
of automation we need to implement ourselves. Phew, that was bad!

## keyboard

Just yesterday I started using a new mechanical keyboard. I am not going to
talk much about it just yet since I think it needs a week or three of intense
use before I can tell my honest opinion of how it works and what I think about
it. I will detail everything in a blog post later.

## Coming up

- Christmas. I am going to be slower than normal. Just relax.

## Links

[1] = https://introspector.oss-fuzz.com/project-profile?project=curl
[2] = https://github.com/curl/curl/pull/12563
[3] = https://everything.curl.dev/


-- 

  / daniel.haxx.se