libuv Integration Has Started...an I/O Renaissance

So I put together a build with libuv, and...

...It's Gone Extremely Well!

:flight_departure:

As a first step, I decided I'd try using libuv for all of our filesystem calls. They're based on the POSIX filesystem APIs, so with a not-impossible amount of work I was able to change our POSIX code over to libuv calls.

Once I did so, I could throw out the Windows-specific code...so all of %posix-windows.c could be deleted!

Of course, now we have to link in libuv. But there's a lot of good news on that front:

  • I was able to integrate libuv into our build without using any special build system, it was easy enough.

  • libuv has care and concern for weird platforms...more than we do! So all the FreeBSD/OpenBSD/NetBSD/BSDi are covered, Haiku, etc. are covered.

  • libuv is pure C and it builds with TCC...so the Rebol-built-with-TCC-and-libuv can still build itself and bootstrap!

  • I haven't done a formal study of the exact size impact but it's in the 100s of ks not the 10s of megabytes range; for what it offers it's light and it seems they care.

While I Was Unifying The Code, I Made... TESTS!

There were basically no tests of the filesystem. Features like /SEEK came late in the game, and there were lots of bugs and design holes. No one knew how buggy it was because I think no one really used anything besides READ and WRITE of entire files!

Getting the semantics for ports hammered out is a tall order, and beyond the scope of these first steps with libuv integration. But to get things on the right foot, I've started with some tests!

https://github.com/metaeducation/ren-c/blob/master/tests/file/file-port.test.reb

I'll put in my usual tearful plea for others to try kicking the tires here... but, well. People are busy I guess. :cry:

Technique-wise, there are just some really cool things even right there in those tests. There's a fuzz tester which creates an adaptation FUZZWRITE that will mimic what the write is being asked to do to the file to a buffer.

So One File Was Deleted...What Else Do We Get?

Deleting %posix.windows.c was actually a bigger deal than it sounds. Even though I'd whittled it way down from its Device-Model hairiness, it was still a small amount of bad code. And I'm sort of glossing over the kind-of-Herculean level of pain tolerance it takes to dot the i's, cross the t's, and write the tests for the new code to replace the old buggy stuff.

But it's just the beginning. We're about to get a lot.

Many libuv functions are able to take a pointer to a function to call when an operation is done. If you pass in nullptr for that callback, then the operation runs synchronously.

For my first task with the filesystem, I just passed in NULL to all the file-reading and file-writing routines for the callback. So we are doing the same blocking I/O as always.

Being able to do asynchronous file I/O isn't a priority right now. But if we need it, it's there.

The real benefit will be having vetted asynchronous network I/O. It will take the place of buggy garbage we had, that was nigh undebuggable. We should be able to write working timers and other interesting things.

Modify With Confidence: I/O Edition

I had said that a goal of Ren-C was that if there was something we thought up, the limit to doing it would be the limit of being able to articulate the design... not having murky code.

I'd claimed that point had been reached, and the new goal was just to "elevate the art" of the language.

But that conveniently overlooked the fact that the device layer and I/O were all still horrible murk. I'd kind of blocked it out of my mind, since I'd thrown it over the wall...and been able to make a WebAssembly build without it. It "didn't count".

Now the Windows/Mac/Linux/Haiku/etc. builds are back in the game, and The Design Handcuffs Are Off

I'd actually said at one point that if we wanted Network I/O to be any good we should just take Node.js's code for interfacing with V8. I didn't realize they'd actually made it a goal for that to be reusable and factored it out as a C library. I thought we'd have to rip out some C++ code and do the work ourselves. But libuv is that work already done, and we seem in good company as far as the Amish-oriented goals go!

Anyway, change is coming. So for starters...if you have ideas or complaints about files...now would be the time to speak up! The real turning point will be on improved networking, so stay tuned.

7 Likes

Sounds fantastic. Well done!

1 Like

Bravo! @hostilefork is a legend!
Is Ren-C still a rebol if virtually every part of it has been replaced?

2 Likes