From daniel at haxx.se Sat Feb 4 09:24:16 2023 From: daniel at haxx.se (Daniel Stenberg) Date: Sat, 4 Feb 2023 09:24:16 +0100 (CET) Subject: [Daniel's week] February 4, 2023 Message-ID: <7276qns9-r48q-3sn8-9qqs-954ooq58315@unkk.fr> # Daniel's weekly report ## Vacation I managed to not respond to a single curl mail nor touch any GitHub issues in my absence! My two refreshing weeks off were excellent and now I am back again and moving forward at full speed. ## New dev machine I eventually received all components for my new dev machine and this week me and my brother Bj?rn put the thing together. Our first snag was to learn that we did the the power cable from the PSU to the motherboard wrongly so at the first power on attempt not a single led turn bright and no fan started to move. After some quick online research and Bj?rn testing the PSU to make sure it actually works, we corrected the cable and we could see that the power setup worked. Something else did not work though. A few leds turned on but nothing showed on screen. One of those red leds indicated that the problem was the CPU and after more online research we realized this disheartening fact: the CPU is too new for the bios version on the motherboard. In order to flash an new BIOS version on the motherboard, we need a working CPU because this particular model does not provide a feature that would allow us to upgrade it without a CPU... Then, as the hero he is, Bj?rn brought the entire box within him back to his house and a day later at his house he ripped his own machine to pieces, extracted his (slightly older but still fairly new) Intel CPU, inserted the older CPU on my motherboard, upgraded the BIOS version to the latest version available, then removed his own CPU again and put it back into his own machine and finally he inserted the *intended* 13th gen Intel core-i7 CPU into the motherboard and the machine would finally work. There would be one more snag: the motherboard does not like a fully stocked 64GB of 3600MHz memories inserted as 4 16GB modules. When all modules are inserted, it apparently only wants to run them at the most at 3100MHz. We work around this issue now by leaving it with 32GB installed. A minor setback, but I am convinced I can survive quite a few years into the future with "only" 32 GB of ram. Bj?rn still has the new machine in his house so I have not yet started installing or using it. I will post all details in a blog post once things are in better shape. ## Performance By a coincident I fell over the URL decoder function in curl and realized that it called `strtoul()` to convert hexadecimal numbers and it just occurred to me that it is probably not very fast. My subsequent fix [1] ended up running my test program 2.8 times faster with the help of some table lookups! Having improved the decoder, I took a look at the encoder as well and wow, by replacing the method [2] of how it outputs `%HH` codes, my test case ended up 7.8 times faster than when I started. In the mean time, Stefan also fell over inefficiencies in the HTTP/2 multiplex handling [7] and made curl transfer two streams over the same connection where the streams were very evenly distributed over the connection *much* faster. Between two and three times increased transfer speeds! We also concluded that there is more to do in that area and that we should probably loop back to this later this year and see what more we can do to enhance transfer performance for multiplexed transfers as we have some good ideas. ## URL parser speed I got into a conversion with Yagiz Nizipli this week, the author of the ada URL parser [3] that is being proposed as a replacement for the URL parser used by node.js. They run speed tests and benchmarks how fast that parses is and that includes some comparisons with the curl parser. Not too surprisingly (to me), their parser outperforms the curl one with some two-digit percents or so. Ada parses WHATWG URLs and is not compliant with curl's parser and the parsers are not on 100% feature parity, but I still found it interesting and it certainly shows that more can be done to the curl parser to make it faster. Of course we can also debate about what exactly faster means and for what URLs we should optimize the parser etc. Still I found a few minor tweaks that I landed this week. Probably too small to actually be noticeable though. ## FOSDEM The first FOSDEM I attended was 2010. Since then, I have visited every physical version of this awesome conference and I'm thrilled to do it again this weekend. This is also why my weekly report goes out a little early this week: I'm taking off mid-day Friday to Brussels and then I will hang out with friends, talk and live Open Source for an entire weekend, drink beer, eat waffles and hopefully hand out a lot of curl stickers. ## GitHub Accelerator I am a member of brand new the GitHub Accelerator program [4], and this week we had our first meeting in what seems like an impossible task in selecting the first 20 "fellows". I won't give away any details or numbers now, but man there are many really awesome people, projects and applications in there. ## GitHub Star On the subject of GitHub, I was also this week confirmed a "GitHub star"[5] in 2023 again. For me, one of the best perks that comes with this honor is that I get a great channel to the teams behind the service that I use so much. I can tell them my opinions and views and I get early peeks on what they're working on. ## Blog posts - curl's use of many CI services[6] ## Coming up - start preparing the Feb 15 release - write security advisories for the pending security fixes ## Links [1] = https://github.com/curl/curl/pull/10376 [2] = https://github.com/curl/curl/pull/10377 [3] = https://github.com/ada-url/ada [4] = https://accelerator.github.com/ [5] = https://stars.github.com/ [6] = https://daniel.haxx.se/blog/2023/02/01/curls-use-of-many-ci-services/ [7] = https://github.com/icing/blog/blob/main/curl-2023-02-03.md -- / daniel.haxx.se From daniel at haxx.se Fri Feb 10 15:08:32 2023 From: daniel at haxx.se (Daniel Stenberg) Date: Fri, 10 Feb 2023 15:08:32 +0100 (CET) Subject: [Daniel's week] February 10th, 2023 Message-ID: <57o67rqp-29po-5512-s596-o599qn98q15@unkk.fr> Hi friends! Another week passed. ## FOSDEM The conference was back in its former glory and the level of attendance seemed to be similar to that of the years before covid. The chaos, the excessive number of tracks, the incredibly occasional niche talks, the beer, the crowds, the lines to the food trucks, the waffles and of course the many friends. This year I made a point of stocking up curl stickers in the wolfSSL booth and I hung out there for a bit too which allowed me to meet with and talk to a lot of people. Friends, users and yeah, I dare to use the word: fans. I was asked to participate in selfies with people numerous times. I am certainly more than happy to do that, even if I sometimes don't really know how to handle the celebrity vibes. I even did autographs and a few signed stickers - even if I cannot recommend the signing stickers thing because they are just too glossy and hard to get any pen to write decently on. I attended some good talks and learned new things, and I also got to taste a lot of good Belgian beers and talk with many friends. This year I did not do any talks myself, which was sort of relaxing and laid back. I think I will try to cook something up to talk about next year. Because I will most certainly go back. I handed out over one THOUSAND curl stickers during the weekend and now I'm all out. ## GitHub Badge GitHub ran a social event on the FOSDEM Saturday. Me and a bunch of friends attended, and while there I was one of the lucky ones who won a fancy e-ink badge in the lottery [3]. ## NASA By the time I had to leave FOSDEM to head over to the airport for my trip back home, Dr Steve Crawford of NASA entered the stage at the biggest room at FOSDEM and did a closing keynote. He talked about NASA's use of Open Source, and on slide 28 he showed a screenshot of a tweet of mine from 2021 when I was complaining about NASA repeatedly sending emails to me asking about curl details [2]. Lots of friends highlighted this for me and wow, it would have been a blast to have been in the room when that occurred. ## Websockets I landed more WebSocket fixes this week. It not only makes the code and API better, it also shows that we have a few users actually trying it out and provide feedback to us. It has made me start to believe for real that we will be able to push the WebSocket API forward and maybe remove the experimental tag from it later this year. ## New dev machine After last week's adventures with my new PC, I have moved it over to my house. I booted it and I have started the slow work of setting it up the way I want it and moving over data from the previous work horse. I will write a separate blog post will all the details later, including some benchmarks to compare performance on the old vs the new. I think I'll do the big switch immediately after I have done the release next week. ## Roadmap I have asked around for input. Around this time of the year I try to put down a few things I think could be worth working on a little extra to see happen during the year. Of course paying customers will have slightly more to say to guide me, but in I primarily want curl to keep up with the times and go where the Internet leads us. I will do a separate webinar on the topic later (on March 23rd), but I am rather confident I will end up at least including these five areas: http3, WebSocket, HTTPS records, ECH and hyper. Which incidentally has some overlap with what my roadmap plan said last year... ## Release prep I am busy writing details in the release blog post while also merging the final bugfixes that are coming in. 7.88.0 will ship in the morning my time on Wednesday February the 15th. There will of course be a live-streamed video presentation on that day, as the tradition goes. On that day we will also publish three security advisories, which of course all are fixed in that release. ## Birthday prep The 7.88.0 release is meant to become the final curl version 7 release. We are then looking forward to curl's 25th birthday and the 8.0.0 release, both which will happen on the same day: Match 20th, 2023. This also means that we will not open the feature window between these two releases, with the hope that this should help us make 8.0.0 a well polished release. We've created a celebration thread [1] over in the curl discussions on GitHub if you want to send your best wishes, curl memories or just want to read what others have to say. ## Coming up - curl release on Wednesday - restock on curl stickers - switch dev machines and get some performance numbers [1] = https://github.com/curl/curl/discussions/10465 [2] = https://daniel.haxx.se/blog/2023/02/07/closing-the-nasa-loop/ [3] = https://daniel.haxx.se/blog/2023/02/06/a-badger-bagde-for-bagder/ -- / daniel.haxx.se From daniel at haxx.se Fri Feb 17 17:20:50 2023 From: daniel at haxx.se (Daniel Stenberg) Date: Fri, 17 Feb 2023 17:20:50 +0100 (CET) Subject: [Daniel's week] February 17, 2023 Message-ID: <7s8sq049-s4r7-68sq-754n-9pn4793qsp10@unkk.fr> Hello! ## release Things looked fine the days before the release and we shipped the supposed last ever release of curl version 7 on the Wednesday, together with announcing three security advisories. I managed to complete my video presentation with that good feeling still hanging on, but then everything took a different direction. A data corruption bug was reported over on IRC and an intense work session started. The symptoms: using latest libcurl with arch Linux' pacman installation program, with a config allowing it to download files in parallel would trigger it. It turned out difficult to trigger under any other scenarios and that made it initially hard to wrap our minds around. Stefan Eissing and Harry Sintonen worked fiercely through the day and well into the night, and the following morning we had to pull-requests that together had the issue fixed. Additional details include that it requires HTTP/2 and multiplexed transfers with small enough frames to get multiple frames for the same transfer into the libcurl buffer, with more than one transfer in progress. This turned out almost impossible to trigger with Apache httpd but not with nginx. Of course it was also somewhat timing sensitive. Due to the seriousness of the bug we quickly decided we should do a patch release soon to help people avoid this. ## another release We decided we do the patch release on Monday Feb 20. A second final version seven release if you will. curl 7.88.1 it will be. In this release we will of course fix the data corruption bug, but as there are also other bugs reported and fixed we will merge more bugfixes. That's also why we give it a few more days, as it allows us to collect more things to address while we're doing a release anyway. So, I'm back at preparing a release again, pretty much like I did last week. ## Birthday prep I will arrange some kind of online "birthday party" on March 20 to celebrate curl's 25th birthday and the 8.0.0 release which will happen on the same day. The celebration is probably going to be an open Zoom meeting for anyone to join and hang out at, at which I might do a curl presentation going through the main events in the curl project over the twenty-five years it has existed - as I have been the lead developer of the things during its entire lifetime. I will provide details, links and more at a later point. Stay tuned. The celebration will happen in my early evening time, maybe 17:00 UTC or something. What makes it complicated for US peeps is that this date is within the period AFTER the US has switched to daylight saving time, but BEFORE Europe has. I have ordered myself a 25 year old single malt whiskey for the event. ## Sourcegraph Half a year ago (!) I was a guest on the Sourcegraph podcast and this week the episode eventually went public [1]. Lots of talk about how I got started with computers and my journey from Commodore 64, via Amiga into unix and networking programming etc. And of course lots of curl. ## Stickers I mentioned previously how I completely drained my entire set of stickers while at FOSDEM. This week I bounced some new sticker ideas with friends on Mastodon before I ordered a new badge of curl stickers. I mostly ordered good old trusted designs but I added two "limited edition" ones that came out as a result of the brainstorm: one "can you curl it?" and one "yes we curl". I am sure I won't be able to resist to post images of them when they arrive here. ## Elektroniktidningen I previously made a magazine version in Swedish out of my blog post "IDN is crazy" [3] for Elektroniktidningen that appeared in print a few weeks ago. This article is now available online [2]. Perfect for practicing your Swedish! ## Blog posts - curl 7.88.0 seven stops here https://daniel.haxx.se/blog/2023/02/15/curl-7-88-0-seven-stops-here/ ## Coming up - another release - switch dev machines and get some performance numbers ## Links [1] = https://youtu.be/ZLtqHFxEDm8 [2] = https://etn.se/index.php/69777 [3] = https://daniel.haxx.se/blog/2022/12/14/idn-is-crazy/ -- / daniel.haxx.se From daniel at haxx.se Fri Feb 24 16:34:30 2023 From: daniel at haxx.se (Daniel Stenberg) Date: Fri, 24 Feb 2023 16:34:30 +0100 (CET) Subject: [Daniel's week] February 24, 2023 Message-ID: <46n17n5-6915-3o4-6on8-4qr0p7qo8r59@unkk.fr> Hi friends, ## release Doing a release on a Monday is perhaps not the ideal day because it forced me to spend more time on the weekend to make sure all my ducks were in a row. The final Windows related bugfix was merged early Monday morning just before I packaged and uploaded the new release. Also not ideal, but I really wanted the bug fixed in this release! This time the release went smoother and now several days after we have seen a few bugreports on the new release but none of the serious kind we saw for the previous release. I think we can conclude that there will be no more version seven releases and instead we are now running ahead towards the version eight release on March 20. ## new machine Once the release was out I could finally do the dev machine switch and since Monday the new machine is now my new primary development box. The transition turned out really smooth and now several days later I have still not ran into any into major snag so overall just a very convenient step. This new beast builds curl almost 5 times faster than my old PC. ## security We have received a few security reports this week about potential security problems from two reporters. Left for me is now to assess if they truly are security problems and if so, how serious. It is hard and difficult work. ## live-coding I revived my live-coding this week and did two three-hour sessions [1] in which I worked on bugs, merged other people's pull-requests and submitted numerous pull-requests of my own. Fairly productive sessions actually, and they also turned out to be fairly well viewed as over a thousand unique visitors viewed at least a part of the Thursday version! Live-coding is difficult as I need to talk, work on the code, make sure the right stuff is visible on screen and think at the same time. The chat and people's questions and feedback make it all really fun and worthwhile. ## reducing sscanf Recently I've turned unhappy with the frequent use of sscanf() in the curl code and I have started to slowly rewrite functions to transition away from those calls. sscanf() itself is not really bad, but the function call is slow, many times the parser accepts a little bit more than maybe intended and sometimes the pattern and what it does is really hard to understand. I just have this gut feeling that we migth have some hidden bug somewhere in our uses of ssscanf() just because of this. ## lists.haxx.se Someone pointed out to be that some of the mailing list pages and forms for mailman that appear on the `lists.haxx.se` site were using http:// URLs. Apparently they could make browsers show warnings in some situations, even if use HSTS on the HTTPS version of the site so very few (if any) requests would actually ever get sent to the unencrypted version of the site. Anyway, after some digging into the details I figure out how to update the entire site and all individual mailing lists to use HTTPS URLs. I have a local Mailman 2 instance built from source that I use for this. It even uses a local Python 2 build, since Python 2 has been dropped from Debian and Mailman 2 does not work with later Python versions and the successor Mailman 3 is such a completely different thing and monster to install and upgrade to that I gave up trying. ## stickers I explained before how I ran out of curl stickers at FOSDEM 2023 and I ordered a new set that arrived at my house this week. A big box of stickers. ## podcast We recorded the initial pilot episode of our foss related podcast today February 24. There are just a few more details left to get done and before we should be able to go public. We will then be most eager to get feedback and suggestions on where to go with this, guests to invite and topics to discuss. Here: https://github.com/fossified/podcast ## Blog posts - 7.88.1 the second final one https://daniel.haxx.se/blog/2023/02/20/7-88-1-the-second-final-one/ - My 2023 dev machine https://daniel.haxx.se/blog/2023/02/20/my-2023-dev-machine/ - Restocked on stickers https://daniel.haxx.se/blog/2023/02/23/restocked-on-stickers ## Coming up - fix bugs, merge pull-requests ## Links [1] = https://www.youtube.com/playlist?list=PLpXAyWkDQy43xYWkRVDvxyPxhLeHDxdmg -- / daniel.haxx.se