Ren-C 2018 Roadmap and Retrospective

Though there is the Trello to track progress...

...since a New Year is upon us, I thought I'd reflect on what we've accomplished and what we plan to do.

The Past

The Ren-C effort started perhaps a month or so prior to the Pull Request #1: Coherence One, on Jun 16, 2015. Having invested a bit into modeling what a C++ API for Red might look like to use, I found it easier to test that API linked to Rebol (since it was C, and given that many things in Red were incomplete).

Not wanting the work on understanding I'd done to go to waste, I wanted to un-fork the divergence between Atronix and earl's "community" build. It seemed to make sense to couple that with using compiler warnings to maximum advantage, getting the codebase on solid ground with Valgrind and Address Sanitizer...and then not letting it slip back again.

So it's been going on for 2 1/2 years. If you look at GitHub's commit graphs, it hasn't been exactly continuous work since the start date. But it hasn't gone away. In that time there have been many accomplishments, to name a few:

  • @Brett's work on getting the native specs into the C code, along with accessing arguments by name instead of number (with no degradation in performance by doing so) made a night vs. day difference in how the code looks.

  • @szeng's rewrite of how extensions work and move of the build system to be completely Rebol-based has pointed toward how things should fit together, though we need to do everything we can to reign in growing complexity (there and elsewhere)

  • The console's move to userspace...including for processing command line parameters...was another night vs. day difference. @draegtun's work--one of the few places we actually have documentation :slight_smile: was an inspiration to continue pushing on it.

  • Thanks to the initial work of earl for adding 64-bit support and Travis building 32 and 64 bit variants of Linux and Windows, plus that of @giuliolunati on Android and @szeng on Mac, we have an aggressive build matrix on Travis that works for us every day.

  • @gchiu has been amazingly resourceful in stepping into earl's big shoes...linking those builds up to a download page and organizing other community resources (like this forum, of course)...staying present and keeping the chat lively.

  • @rgchris, @MarkEye, @kealist and everyone else have been there to talk out the big or small technical questions through the days, as we mull over what this language can and can't become.

  • @johnk managed to save institutional knowledge from CureCode with a massive migration to GitHub, which is especially helpful as the rift with Red means no support keeping Rebol bugs alive would be expected.

  • BlackATTR, IngoHohmann, JacobGood1, GeekyI and everyone all coming to participate on chat and the forum help keep me interested and perhaps, even, a little bit too, pekr, and even iArnold. :slight_smile:

  • rebolek and everyone on the Red team have kept me irritated enough to need to finish my vision to prove them all wrong. :stuck_out_tongue: (Oh fine, put yourself in the previous bullet point if you feel like it.)

Hopefully I haven't left anyone off. But of course we all know who the real hero is!

Just kidding. :slight_smile: Yet one thing that I must realistically confront is that if I stopped working on the system, the odds aren't too good at this juncture that anyone else would step in to push it forward. And nothing would make Red happier than if I gave up.

So it's worth mentioning I've certainly had moments of doubt if I wanted to continue. Life is short, I have a lot of interests, and this seems to have grabbed a lot more of my time than I would have ever imagined.

My stance--as many people know--is that Rebol is a language that is both very deep, and deeply flawed as a tool for generalized software development. We see moments where it does things no other tool does, and other moments where it fails...and there's not exactly a clear map of the space of application for the language. But it seems every time I've thought of sending it to my digital dustbin, some new innovation pops up to give me faith that it's something worth doing: definitional returns, specific binding, specialize/hijack/chain/adapt/enclose, void semantics, enfix, FRAME!s, redo, elide/comment... the list of design accomplishments is getting quite long.

Now with Red's recent doings--it would seem responsibility for delivering the vision of the original Rebol has fallen even more squarely into our hands. Everything I've seen so far in the reaction to the crypto announcement indicates that regardless of if they raise the desired money, they're going to be attracting what I'd call "the wrong element". A different kind of get-rich obfuscating crypto crowd, who wouldn't be truly moved by Carl's initial vision statement (or even be moved by Nenad's own "What is Red" presentation, which I actually quite liked)

The Future

As some of you know, I've traveled extensively the last couple of years. It's hard for me to envision life a year from now--in more ways than one--it seems almost unfathomable that as the clock ticks 2019 I'd be writing something along the lines of "hmmm, we still haven't changed the %.r files in the repo to be %.reb". Groan.

I'm in this because I've said I want to build the Minecraft of programming. Something intuitive, composable, and visceral. More intrinsically flawed than pure functional programming--but accessible to a wider audience with similarly pleasing ergonomics (perhaps even to the point of feeling toy-like). But get people hooked to where they're having so much fun they'll forget that the particular task they're doing could be done more rigorously--or even more easily--in some other language.

Ren Garden showed a glimpse of what this could be, and Qt is still very much how I would do things cross-platform, if it were just me. Yet I would like to remind people that it was really written in only about a month...and at a very early time in my understanding of Rebol's implementation, while the API was being hammered out. I'll also mention that while it's cool...I would nearly guarantee that undertaking such an effort so it ran through a browser interface would be a more strategic choice. Even if it used something horrible like Electron. Groan again.

In any case, I have mixed feelings of investing too heavily in Ren Garden. Yes: I do still want us to be able to have automatic Travis drops for Windows/Mac/Linux. But it's probably less useful as a tool in its own right than it is a showcase of how to embed Rebol in a C++ application. It has been a great test for that, and because of its general ambition helped me to avoid some design mistakes in the API

The technical things I want to get hammered out in the next several months are:

  • UTF-8 Everywhere: This isn't so much that I feel pressured to solve the problem of emoji in code, as needing to solve the API-level behavior of strings in Rebol. It's quite a gargantuan task, affecting the entire system (when we say "everywhere", we mean "everywhere"). It is unbelievably difficult--probably the single most difficult change to date, rivaled only by specific binding. If not for the debug build, its noisy alerts, and the ability to choose to build the code as C++ I would call it impossible...for all practical purposes. But not only is it crucial, I think it could make a good marketing tool for the language being built on foundations that are future-forward...which would appeal to the UTF-8 Everywhere Manifesto crowd.

  • libRebol: I want it to be in working order, with full variadic support with C string runs, and built as a target in Travis. Yes, I mean rebDo("print [", rebInteger(x), "+ (2 *", rebEval(negate_function), "3)]", END);...forming blocks and groups across spans, going even further than RenCpp did but with pure ANSI C. Once libRebol can be brought in, that should make the Ren Garden build "more or less trivial" (to the extent configuring Travis is trivial)

  • Derived Binding: freeing us from the situation where 1000 instances of an object with 100 methods means making 1000 * 100 deep copies of function bodies. That's the kind of serious flaw that you would think it would be flashing in big bright letters on the description of OBJECT!, because people will run into this. Atronix did, and it was a very big deal. I don't have the solution completely worked out, but enough of it seems to work that I'm confident. (UPDATE: committed February 3rd, and no bugs reported...yet.)

  • Bring Rebmu Up To Date: Since I talk about wanting programming in Rebol to feel like you're playing a fun game, there is one gaming community that actually plays with their code: Code Golfers. Not only do I think it's probably the most intriguing golfing language people have invented yet--and a good marketing tool--I think it could be the language used to win a lot, if sufficient care is put into it.

  • Userspace Stepwise-Debugging and Tracing: The userspace console is already--in my opinion--pushing a lot of boundaries for how a console works. My recent ENCLOSE/HIJACK example shows a glimpse of how this is going to go even wilder, in the form of a very-hackable debugger, where most all of its logic is written in Rebol. To say there are a lot of open questions is an understatement, but I feel optimistic.

I can't easily see too far past these "near-term" goals (they're actually all huge things). And I don't know how much anyone else can help me with those...or other deep things that I don't feel design confidence about--like "arbitrary" virtual binding, user-defined types, modules, multiple dispatch, port design...

But we'll need project growth to take the continuation-of-Rebol mantel away from Red, where it was previously hoped to be (especially hoped by me, because I didn't want to be in the position of making a product). So that means we'll have to start publishing. And should look a lot like Red's download page. Maybe start writing some cool articles. Or pay a company to make a press release we write about ourselves. :stuck_out_tongue:

I also think selling a language is quite difficult, so any ideas people have for products that can sneak-attack people is a good idea. Whether that's something like Graham's lighting router, or an in-browser PARSE tool that looks something like RegExr, I don't know.

In any case, Happy New Year to everyone. Thank you for your help so far, thank you for putting up with me. I hope something can come out of all this collective effort that makes people happy--even if it's just a few of us!


Note: it only lets you @ 10 people per post, so workaround I'll quote the relevant portion here:

  • @BlackATTR, @IngoHohmann, @JacobGood1, @GeekyI and everyone all coming to participate on chat and the forum help keep me interested and perhaps, even, a little bit too, @pekr, and even @iArnold . :slight_smile:

  • @rebolek and everyone on the Red team have kept me irritated enough to need to finish my vision to prove them all wrong. :stuck_out_tongue: (Oh fine, put yourself in the previous bullet point if you feel like it.)

So, the reason for Ren Garden is just to test libRebol? And not to give talks on Amish programming?

I hope it can be improved to the point of being the casual console of choice for most users...moreso than a traditional r3.exe. It should be good enough to load and execute most programs. It should make a good "Rebmu Workbench" if it has a bit of graphical debugging. I'll see what I can do.

It's nice to have a lot of options based on what needs to be done. @szeng has expressed desire to jettison much of the existing R3/View code and use Qt plus their custom zoomable control. Someone who wants to put a Rebol-powered app on Android, like yourself, might be served by making the UI in Qt with its designer tools and Qt Quick, but then much of the program logic in Rebol and bundle it into the APK.

I don't know exactly what the future holds for Rebol and Qt/C++ integration, but it's good to have a test case which happens to do something useful for us.

Very well summarized. Also adding or just and getting an elevator pitch on that site. Introducing articles getting people on various platforms on speed in using REN-C and after that in helping the development.
A VID like addition will also attract a lot of attention and is a must for the GUI minded youngsters out there (like myself :wink: ).
Completeness of the 'product' will help 'our' cause a lot in the acceptance of the Rebol language as a serious alternative over the products used in businesses over the world these days.

Just a reminder that Ren-C is not a name I really want to stick.

The hope--however vague it may be--is still that this is Rebol. If Carl has another agenda for what he thinks Rebol4 should be, or whatever, then that's not going to work. But I'd like to be able to make a case for why Ren-C should be the evaluator behind rebol.exe, and what I once called Ren-C would be "libRebolCore"...while what used to be called "The RL_API" would be called "libRebol".

But I don't want to make that pitch until several criteria are met:

  • I want near parity in performance with R3-Alpha. We've got a big back of tricks now, and when a codebase is really under control you don't have to guess about whether an optimization is "safe" or know it's safe. It is my belief that despite a tremendous number of new features of the evaluator, changes have fundamentally been balanced in such a way that it will be competitive--notably faster at some things, even. There has to be some reasonable metrics taken and any creeping performance regressions have to be fixed.

  • I want near parity in size with R3-Alpha, when the same set of features are included. We should remember things like that R3-Alpha had no HTTPS or related crypto. For a truly fair comparison we might modularize it so that things like the function composition tools are in an extension as well, and have a rough idea of what they cost.

  • I'd like the source to be fewer characters, as well. Comments need to be reviewed and brought down to essentials that convey the same information. Any comments that are actually GitHub issues in disguise need to be moved out and referred to by their issue number. Overly paranoid asserts that have never triggered--or are redundantly being checked by callers and callees--need to be pared down.

  • The build system needs to be reigned in. To get there we'll need to rethink this all-Rebol build system so that it uses an order of magnitude less has to get much more declarative.

These are the sorts of things that if they are not done, there might be a perception that in some ways--despite all its advances--Ren-C has taken some steps back. I want no steps back.

Unlike other work, this is stuff that other people can--actually--help with.