Expanding Opportunity (by Dropping PORT!s from the pure C Build)

I was closing and resolving all the rebol-issues tickets related to PARSE...considering them "resolved" if they are solved by UPARSE. Quite a lot of them were! :partying_face:

Among the few tickets that weren't resolved was an old one in which BrianH talks about adding an /IGNORE flag to PARSE:

Rebol Issue #1080

But to me... this sounds like a generalized feature being narrowly articulated as a PARSE wish.

What he's asking for is there to be a kind of iterator over a series which does not show some of its content. The closest concept that Rebol has potentially had for this would be a PORT!.

Here's some random imaginary pseudocode:

>> input: [1 2 3 4 5 6 7 8]

>> port: make-filtering-port input :even?
== #[port! [...]]

>> read/part port 2
== [2 4]

>> read/part port 2
== [6 8]

Ideally you could do this kind of filtering on a file on disk, a series in memory, a network stream, etc. Even if the file were gigabytes in size you wouldn't wind up creating a huge amount of data just to iterate over the even numbers in it.

This Kind of Mechanic Has Prior Art

(Well, at least when I assume you're not asking for the impossible. e.g. the data you're filtering is read only. And you're not expecting to be able to have some automatic way of inserting things in the middle of a disk file and sliding the contents up...writing back over only the even values. Sane requests here only, please.)

In the past I've often brought up Boost.ASIO, the C++ Asynchronous I/O library. It's for wiring up streams and layering them...dealing with buffering and disconnection exceptions and multi-platform eventing. The whole nine yards.

The filtering example I give above is actually one of the textbook examples of something you can do in Boost by building on streaming iterators:

struct is_even {
    bool operator()( int x ) const { return x % 2 == 0; }

std::vector<int> input;
input += 1,2,3,4,5,6,7,8,9;

    input | filtered(is_even()),
    std::ostream_iterator<int>(std::cout, ","));

That gives the output 2,4,6,8,

But it goes much further than that. You really can stack a streaming decompressor on top of a network port, and then stick a TCP processor and a TLS layer and an HTTP layer on top of that: https://dens.website/tutorials/cpp-asio/ssl-tls

There's a library for WebSockets in Boost.ASIO, and I've already talked about how I think it's important for the desktop executable to be able to speak to the browser automation APIs.

Plan: Strip PORT! Out of The Pure C Build

There's no reason that the pure C build can't read files into memory in one fell swoop, and be able to bootstrap itself. But I can say without a doubt that with the PORT! implementation behind the scenes today: "There's no gold in them thar hills."

So why not leave the pure C build with the basics of reading and writing files in their entirety... very simple and without any need for a PORT! model. Then start from scratch on ports embracing existing work as much as possible, and building on proven and tested code?

We'll keep the CALL facility written in pure C for now. That way the basic executables can always reach out to curl or wget or whatever, if they want to trigger something like a network request.

Starting From Scratch Can Advance Rapidly

It should be clear from UPARSE that when I start out tackling a problem anew with the stored up knowledge and experience from seeing the previous problems, the results can be good.

I've not really worked much on the port model, because there was no model there. But in keeping the code adjusted to stay running and be more convenient over time, a ton of design points on the core have been stressed. So as bad as much of the code is, it's been a proving ground for various pieces of the system.

But now...it's time. I think that basically wiping out pretty much all the PORT! code and starting again can go nowhere but up!

1 Like

Looking at the C++ ASIO in more detail for non-network applications (e.g. streaming files) it doesn't offer as much as I thought in that area. The streaming is all byte-level so you have to build higher level services based on eventing if you want to come up with streams of objects. So it really doesn't map well to Haskell libraries like Conduit or Pipes the way I might have imagined. :frowning:

It may be that the path of least resistance library--which provides more of a generic platform abstraction layer--is likely libuv:

Design overview — libuv documentation


It's used by Node.js and Julia and Rust, and in a number of standalone servers and clients. It seems to be pure C, and may even compile with tcc. We'll see.

Either way, my point remains the same...

It's time to kill off the R3-Alpha networking and file system code. There's nothing there to fix.

  • We won't wind libUV in with the core--provably so--because the WebAssembly build won't use it.

  • It will also be provable because we'll allow making desktop builds without it...they just won't have PORT!s. They will have READ-FILE, WRITE-FILE, and CALL.

I'm trying to get the Sea of Words and Roaring Bitsets stuff tied up, but I'm actually looking forward to deleting the R3-Alpha "Device" layer. The time I've spent with it has really been about switching it to do more of its work in terms of Rebol values and APIs...so hopefully that will pay off in terms of having hardened things to be able to interface nicely with libuv.