From daniel at haxx.se Fri Mar 7 18:29:12 2025 From: daniel at haxx.se (Daniel Stenberg) Date: Fri, 7 Mar 2025 18:29:12 +0100 (CET) Subject: [Daniel's week] March 7, 2025 Message-ID: <8752o482-11q1-3n3n-8rq2-s1277ps4no8p@unkk.fr> Hello friends! ## Email format I'm back to my previous weekly email formatting again, edited in emacs. I was not entirely happy with the other style and I had readers tell me that it was a worse style than before. It is a gentle balance between the ease of writing and how it looks to readers of it. I suppose I will try out variations going forward as well. ## my cookie spec problem I could not resist but objected in a cookie discussion on the httpbis mailing list in which I brought up my old complaint about the existing cookie spec RFC 6265 and its coming successor still known as 6265bis. When someone then asked me about said objection I figured instead of trying to explain myself in a short reply I would clarify myself properly. So I instead blogged[1] some words about it and now there's a URL I can refer others to if this comes up again. ## podcast I joined the podcast Security Weekly [2] as a guest this week and we talked curl security. CVE, CVSS, not rewriting it in rust and what we do to try avoiding letting the world burn. A fun 35 minutes chat. Recorded only the day before it went public. ## upload flags curl's 268th command line option became --upload-flags [3]. Introduced in the pending curl 8.13.0 release. It is a new option to set which flags to set for an IMAP resource when such a one is uploaded to a server. answered, deleted, draft, flagged, and seen are supported and with this we fix a minor feature gap people have been pointing out several times in the past. This is of course powered by a new libcurl option for this [4]. ## strtok We have banned the use of the strtok() function from curl code for a long time, but we also offer a way to do exceptions from this and we have used such exceptions in the curl tool code for just about as long as the ban has been in place. It has always been unfortunate, but not terribly. It has been deemed okay since the code using it is not multi-threaded and it got the job done. This week I went over the code again and rewrote the remaining functions still using strtok() to instead use something else. In most cases also by avoiding actually writing data to the buffer being parsed so that we can a) avoid an extra malloc/free cycle and b) use more const char pointers. This reduces memory use and is generally a nicer thing to do. Now there are no strtok() calls left in production code. Yay. This also made me ultimately able to remove the unconditional strdup() we previously did for every command line option provided to curl. Admittedly not a major memory waste or anything, but still nice to be able to stop doing completely in vain. ## roadmap 108 unique viewers watched the curl roadmap 2025 [5] stream on twitch and over a dozen people participated on Zoom. Just some word about features and things I want and hope to see happen in curl over the coming year. ## UAF We received this interesting UAF report [6] on HackerOne. It correctly identified a Use After Free bug and one that we could reproduce. After a lot of deliberating and thinking however, we deemed that we could not find any reasonable security angle, vulnerability or possible exploit of it, so we have decided to treat it as "normal bug" [7]. This process, making these decisions, is the truly hard part of curl security. I'm keen on making sure that we do them with full transparency and openness so that everyone who wants to can check this out, scrutinize our work and object if there are reasons to. ## fuzzer The reported behind the UAF problem above, Philippe Antoine, also this week extended the curl-fuzzer [8] a bit, and thanks to this unveiled at least two smaller memory leaks. This triggered me to build the curl fuzzer locally again, which I hadn't done in a while and then by extension I found a bunch of minor issues to fix and those have now been merged into that git repository and should make building the fuzzer a little bit smoother going forward. I still need to apply a local patch to make it build on Debian Unstable, and have now provided a separate documentation update for doing this [15]. The curl-fuzzer is what oss-fuzz [9] uses to fuzz curl around the clock in a never-ending fashion. Extending the fuzzer (to do more fuzzing of more libcurl options) helps us keep libcurl solid. We have not had any gits in oss-fuzz for many months now. Maybe this will shake things up a little? ## sub-10 issues As of the morning of March 7 2025, we have less than ten open issues in the curl project on GitHub [10]. Of course thanks to a lot of hard work by awesome contributors. ## location-mode I think I mentioned before that I merged support into libcurl for new ways to do HTTP redirect following that involves a custom method [13]. This is both to allow better redirect-follows in general, but it has also been triggered a little extra by the work on introducing QUERY [14] as a new HTTP method. If, or when, QUERY becomes popular I want curl to be able to use it properly and then it needs to deal with redirects better. So, with the libcurl part merged we need to offer a way for the curl tool to use these new powers. I wrote up a pull-request [12] this week for this purpose, but I deliberately decided to not merge it in this feature window but instead wait for possible more feedback and thinking. I can then merge this new tool feature in the next release cycle. The idea is to provide a --location-mode option. One reason to do it this way, is that users can then for example set a default choice in their .curlrc and then the -L and -X combos will automatically work as decided. Your feedback on this option or other ways to improve this, is welcome! ## feature freeze There are still four PRs marked feature-window that were not merged this cycle because they need attention, changes or both. We still count nine changes listed in the RELEASE-NOTES and a whopping 169 bugfixes merged already, so I think we are doing pretty well. This feature freeze is also the first one with an associated reelase candidate according to the new policy [11]. ## Coming up - curl feature window closes tomorrow March 8 - curl 8.13.0 rc1 ships tomorrow March 8 ## Links [1] = https://daniel.haxx.se/blog/2025/03/01/my-cookie-spec-problem/ [2] = https://youtu.be/0UavY_kKKic?si=M-CiZBzblnwqihlP [3] = https://curl.se/docs/manpage.html#--upload-flags [4] = https://curl.se/libcurl/c/CURLOPT_UPLOAD_FLAGS.html [5] = https://youtu.be/0vVTNm2q2vg?si=duBS2iF8TPgrDYHf [6] = https://hackerone.com/reports/3022041 [7] = https://github.com/curl/curl/pull/16588 [8] = https://github.com/curl/curl-fuzzer [9] = https://google.github.io/oss-fuzz/ [10] = https://github.com/curl/curl/issues [11] = https://daniel.haxx.se/blog/2025/02/28/adding-curl-release-candidates/ [12] = https://github.com/curl/curl/pull/16543 [13] = https://curl.se/libcurl/c/CURLOPT_FOLLOWLOCATION.html [14] = https://www.ietf.org/archive/id/draft-ietf-httpbis-safe-method-w-body-08.html [15] = https://github.com/curl/curl-fuzzer/blob/master/DEBIAN.md -- / daniel.haxx.se From daniel at haxx.se Fri Mar 14 16:26:06 2025 From: daniel at haxx.se (Daniel Stenberg) Date: Fri, 14 Mar 2025 16:26:06 +0100 (CET) Subject: [Daniel's week] March 14, 2025 Message-ID: <9086o627-1o23-0op4-orrq-42p211p61789@unkk.fr> Hello friends, We are again celebrating Friday afternoon here in the European time zone and here are some words about my adventures this week. Enjoy! ## rc1 The first curl release candidate [1] under this new policy happened on Saturday, on the same day curl entered feature freeze. Almost a week later, I think we can conclude that so far this has been an astounding success. We have gotten reports about several regressions that we have fixed in curl, and one of the reports even identified a problem in one of the applications using curl - since Debian is awesomely also running the tests for all reverse-dependencies: the packages that depend on curl. Every identified regression so far has resulted in extended test cases together with the fixed code in this never-ending attempt to try to make sure that we at least should not be able to introduce the exact same regression in a future. ## 0 One of the silly regressions happened because I modified the number parsing done in the URL parser. It parses numbers when it parses IPv4 numerical addresses, you know like "192.168.0.1" etc. There are however many more weird ways to write numerical IPv4 addresses [9]; one being as a single 32-bit number. This is sometimes used as a shortcut when people use plain "0" to equal 0.0.0.0 which normally reaches localhost. Anyway, I then learned that I had no test cases for "0" and I could quickly fix this problem and I added like 30 new tests for the parser. The mistake I did was about parsing numbers that are 0-prefixed, as they are specified as octal but the code (the regression) misbehaved if it was just a single 0 and nothing more. So, when extending the tests and rerunning the code many times to make sure there were no other slip-ups in this area, I also tested what IP addresses my ping tool would accept and compared with what curl did. When doing this testing, I spotted that if I invoke "ping 018.1.1.1" it actually, to my greatest surprise, sends a ping to 18.1.1.1. Wait, what? The first number there is a *broken* octal number (since they can only contain the digits 0-7). This address should NOT be considered a valid IP address, it should be considered a hostname and since I have no such host name it should return a failure! It took me far longer to realize what had happened than what I like to admit. The address *is* treated as a hostname, and it just so happens that my local (wifi router's) DNS server actually "helpfully" resolves what "almost" resembles an IP address into a corresponding IP address! And it seems to ignore leading zeroes and treat the four numbers as decimal no matter what: $ dig +short 018.1.1.1 @192.168.0.1 18.1.1.1 $ dig +short 099.01.1.0010 @192.168.0.1 99.1.1.10 Sigh. ## stats We are at 29 days since the previous release and we count 232 bugfixes merged so far - by 53 contributors, out of which 33 wrote commits. We did 92 commits over the last seven days. We received three vulnerability reports on HackerOne this week. One totally silly and two reports that identified bugs. The bugs were deemed not security problems as they were both located in experimental code (in the HTTPS RR parser and in the rustls backend). ## thanks I received what must be considered one of the best kinds of emails you can get. The kids are alright apparently: Hi Daniel, I've written a web fuzzer in c using your library libcurl. I want to thank you for writing a this easy to use, fast and good library. I'm 11 years old and I don't have much experience but your library made it possible. ## matrix We're slowly working on setting up a Matrix bridge to the curl IRC channel [2] in an attempt to bring more curl chit-chatters together. I hope to be able to announce all the details about it soon, of course including specifics on how to use it. ## 30K followers I spotted that I actually have surpassed 30,000 followers [3] on Mastodon this week. I am a happy user there and I must say that the fediverse and Mastodon works excellently for me as a place to read news, talk tech and post random nonsense. With friends and like-minded people. ## 19K commits I reached 19,000 commits in the curl git repository this week [4]. Another totally meaningless number of course, but I happen to like numbers. ## fuzzing We received a great PR in the curl fuzzer repository this week that extended the code with a number of new "code points", which has then resulted in the fuzzer having found some new code paths it previously had not been taking. This generated a few new interesting issues for us to work on. Nothing terribly alarming or anything, but still a set of tiny glitches pointed out to us and now we are a tad bit better after having fixed them. ## mvp In October 2024 I bragged about how I was invited, or accepted, into the Microsoft MVP program [5]. I even got a glass plaque fine-looking award thing that I now have here on a shelf. Recently I responded to a survey within the MVP program. Mostly because I couldn't help myself and I figured they could use a share of my honesty. I told them straight out that there was absolutely zero things in the program that interest me: no Open Source, no Internet technologies and nothing about protocols - and with no chance to meet any interesting people or dev teams involved in things I interface. I had low expectations already from the beginning, but it was even worse: it's just a non-stop stream of Microsoft technologies, Microsoft products and of course a huge amount of AI. And I don't care about any of that. This week I received the email that explains to me in plain and simple terms that they are now "retiring" me from the program. curl runs in literally *billions* of Windows machines and Windows remains an important operating system for curl users, but being in the MVP program seems to offer no benefits whatsoever for my current or future curl endeavors. ## curl up sponsorship Did you know that YOU can sponsor this year's curl up [6]? curl up is the annual curl developers meetup, this year taking place in May in Prague. You can have your company name associated with the event while at the same time funding critical Internet infrastructure and one of the world's most widely used software components. ## 100YC I call one of my planned sessions at curl-up 2025 for "100-year-curl". I believe there is an actual chance the project could live a hundred years. We already passed the first quarter of that. Foreseeing the future is of course impossible. There might be things to do to in and around the project to increase the chances of this going well. And there might be things to avoid. I think it could be interesting to see what people think. The 100YC problem. ## bandwidth I checked yesterday, and over the last 30 days the curl website has served 72 TB data, which is another 10 TB more than what I blogged about just a few weeks ago [7]. It is an incredible amount of data, and no I still cannot explain it. ## not TLS 1.3 I submitted a first PR [8] for the planned upcoming removal of TLS backends without TLS 1.3 from curl: BearSSL and Secure Transport. It is a pretty nice negative 5,000 lines patch. I keep mentioning this planned removal and I submit this PR early, all as an attempt to make sure that everyone that needs to know about this gets this information. The removal is planned to happen after May 2025. Some users might need to adapt somewhat when they switch over to use another TLS solution. ## HTTP/1 response In a cleanup raid I did on the curl source code recently [10] I converted a lot of ISSPACE() to ISBLANK(). This, to make sure the code does not match more characters than what is actually intended and meant. The ISSPACE() macro matches space, tab as well as the newline bytes (ASCII 10 and 13) while ISBLANK() only matches space and tab. It is important to make sure the code only ever accepts exactly the bytes it is meant to match, as otherwise there is a bug in there waiting to happen. In one of these replaces, I changed the check of the byte following the status code in a HTTP/1 response. The first line of a HTTP/1 response is meant to look like: "HTTP-version SP status-code SP [ reason-phrase ]" [11] - where SP is a single space. Previously the code used ISSPACE() on the byte after the status code, now it does ISBLANK(). This seemingly subtle change has already caused problems in at least two projects that send HTTP responses to be consumed by curl. The original curl code *intended* to check for that space, but because how it used the wrong, too liberal, function it also accepted newline characters and thus the SP could be skipped if there was no reason-phrase. Whereas now the code insists on a space (or a tab) to be present for the response to be considered valid. I fear we have only seen the tip of the iceberg for this. We might have to backpedal this protocol strictness. ## vacation I am checking out, leaving the keyboard and will be away from work next week. If you need anything curl, file issues like normally and there will be lots of good people around to help out. ## Coming up - curl 8.13.0-rc2 ships on Monday. I will arrange it from away - on Thursday March 20, curl turns twenty-seven years old - no email next Friday, as I will be on vacation ## Links [1] = https://curl.se/rc/ [2] = https://curl.se/docs/irc.html [3] = https://mastodon.social/@bagder [4] = https://daniel.haxx.se/blog/2025/03/14/19000-curl-commits/ [5] = https://daniel.haxx.se/blog/2024/10/02/im-a-professional/ [6] = https://github.com/curl/curl-up/wiki/2025 [7] = https://daniel.haxx.se/blog/2025/02/22/curl-website-traffic-feb-2025/ [8] = https://github.com/curl/curl/pull/16677 [9] = https://daniel.haxx.se/blog/2021/04/19/curl-those-funny-ipv4-addresses/ [10] = https://github.com/curl/curl/pull/16520 [11] = https://www.rfc-editor.org/rfc/rfc9112.html#name-status-line -- / daniel.haxx.se