More History Mining: Rebol 3.0 Front Line Blog

Domains Redbol
1

I did a quick full skim-through of Carl's Rebol Blog, which split off into a second blog to discuss technical issues in R3-Alpha.

The second blog kicked off on April 6, 2006:

http://www.rebol.net/r3blogs/0001.html - "Welcome to the REBOL 3.0 Front Line..."

It got up to 352 entries, tapping out on February 20, 2011 (which was prior to the open-sourcing):

http://www.rebol.net/r3blogs/0352.html - "Relative speeds from compiler optimizations"

Again, I'm Skimming It For Anything Useful

However there is a lot more here to react to. This could take days!

But I do think it's worth it, in part just to kind of help put together a big picture of where this project was and where it is now.

  • Introduction of TYPESET! - Oh... Rebol2 didn't have them? :thinking: Well, neither does Ren-C... typesets are replaced by type constraints.

  • MAKE Function Used To Be Variadic - I don't think I'm a fan of MAKE being variadic, but I have questioned the idea that there even need to be a MAKE FUNCTION!, vs just having FUNC be an arity-2 native to start with. This actually may be more prescient than I imagined, as there is no function datatype any longer... functions are just antiform FRAME!s.

  • To COPY or not COPY - I've sometimes critiqued Carl for seeming to have not confronted Rebol's actual core flaws head on. But here he does...mentioning new users getting bitten by mutability -and- also the problems from mutable binding on a block being passed in to two different MAKE OBJECT! calls. I've been satisfied with Ren-C's answer to the first issue by the CONST implementation--which has held up super well for years now. And how object creation works is undergoing a renaissance as I type this...no mutable binding (or even virtual blanket binding) required.

    • It turns out there was a period in R3-Alpha where it was deep copying object specs and people eventually complained. He says he was waiting to see if anyone noticed. I do think it goes to show how little truly sophisticated Rebol code there was in practice, for people to not be able to tell sooner.

  • Hot Errors Removed - He says "These hot errors turned out to be overkill, and the benefit of error locality was offset by the difficulty of handling error values in general." But in effect this is another decision where Ren-C went more in a Rebol2 direction, introducing a whole new antiform error design.

  • No Out Of Range Errors With FIRST, SECOND, etc. - Because an early development of Ren-C was to make FIRST, SECOND, etc. specializations of PICK... I'd always thought of them as being related. Apparently at one time they were not. In any case, the issue of range errors has been something that's crossed my mind... and I think I'd rather you have to say try fourth in order to tolerate out of range errors. But the unification of PICK mechanics and TUPLE! selection, e.g. block.3 is the same as pick block 3 throws a wrench in it. This is still an open question.

  • The 64-bit question - Wow, this is still not decided. Ren-C still uses 32-bit indices (but stores them as int_fast32_t etc. so they'll use 64 bits of space if the registers make it faster). It hasn't been an issue at the forefront. I'm more interested in distinguishing in the code distinct datatypes for 1-based "Index" vs. 0-based "Offset" and making sure those are distinct types that are harder to mix up.

  • JIT Binding - I've never heard of this and don't believe I saw any code for it. On the surface it sounds like a relative of virtual binding, but it mutates the bindings in the block permanently and just sounds like something strange that never saw the light of day.

  • Debug Hook - Here he talks about being able to essentially break into the console at the moment of a FAIL and inspect the stack. One thing about historical Rebol is that because error trapping was frequent (e.g. ATTEMPT etc.) and being used as a programming style, you'd be getting a lot of false hits. Ren-C's error handling model is such that if you handle an error that is generally done by reacting to a RAISE before it's actually promoted to a FAIL, so you would not be getting broken into the debugger constantly in the casual operation of your code. We do need this. I believe in my heart that I am being attentive to the needs of a debugger, but there some more giant issues to knock down first.

  • CLOSURE functions - nope, all functions are closure-semantics now...without the crazy overhead of deep copying and binding the body of a function on every call!

  • ALIAS - Old ALIAS was removed long ago, because it was crazy. I've written about reviving this idea more reasonably for modern times.

  • HASH! - I covered this in the other blog response, but the comments here might be useful. He talks about it in a later post as well. And then again, offers some conclusions

  • Getting More Information From FOR-EACH - Asking for not just the item but the series position is one idea. But coming up with ways to ask for more information in enumerations has been explored a number of places. I've wanted to be able to enumerate and know things like "is this the last element" or "is this the first element" and be able to check that easily. @hiiamboris has a fancy FOR-EACH that if you use a refinement in the spec it assumes you want the index at that position, etc. These are interesting ideas, so definitely want to look into them.

  • OBJECT! Field Type Constraints - I've wondered about this. It's definitely the case that languages which don't support type information wind up having another language layered on top of it that does (e.g. TypeScript). But this was never in R3-Alpha, and hasn't been a priority. I note he also mentions this kind of type constraint in module specs.

  • ! and != - I was never a fan of adding ! but we do support != My biggest problem is that != and == are a pair in C, but not a pair in Rebol: != is paired with = for lax equality, and !== is paired with == for strict equality. That makes it more confusing than anything, and I've meant to resolve this but it's another thing that's slipped.

  • Considering Setters - I've mentioned in the past that Ren-C might have an answer to this, where you could use something like obj.member. to get at the "real" value bypassing the functions that assign them. There's a big bag of tricks in Ren-C to implement these kinds of features, but seems there's always something more foundational to attack. R3-Alpha had no code for it, though Red has ON-CHANGE* to react to assignments.

  • Bye Bye System Object? - The idea of making changes to Rebol to support multitasking the way it was being done was fundamentally flawed. Needless to say the system object is still there (though the "system context" is now SYSTEM.UTILITIES, or SYS.UTIL for short.)

  • Should All Functions Return Values - Returning an UNSET! was a "value", just an ornery one historically, e.g. x: print "hello" was an error in Rebol2 and it remained one in R3-Alpha. Though it has been debated, in Ren-C PRINT returns NOTHING (the moral equivalent of UNSET!) and is legal to assign to a variable...it just unsets it. The only thing that you can't assign is an empty block antiform (a multi-return parameter pack), because when empty it is considered to be holding no values. This is called NIHIL and it is rarely returned.

  • Leaky Functions - Not knowing this discussion was taking place in May 2006, I made a similar argument here in September 2019. But there's an additional wrinkle that Ren-C has unique customizable per-function RETURN definitions. So it's pretty much a slam dunk to make the change, and introduce LAMBDA as both returning its body result and not having a RETURN.

  • BIND Expands Contexts - Carl says "A few of you will ask: why does this matter? The main reason is memory (something we are optimizing on R3.0). Why force allocation for "free" variables (module variables) that are never used?" Everything is different in Ren-C, though it does deal with something called "Attachment Binding" where module variables actually are not allocated until they are used.

  • Current Module - I actually thought he was speaking about there being something available in a module similar to a SELF or THIS, which seemed like a pretty good idea. But instead he's talking about running DO on something and it expanding the module you called it from. Ren-C doesn't do this, DO runs its code isolated and can only give you back its final result. You have to use IMPORT if you want to bring in new definitions to the module.

  • UNICODE Support - It is interesting to see the considerations here, where it seems being able to load old LATIN-1 scripts was a concern. No consideration of UTF-8 Everywhere is given for storage, but that's what Ren-C does. He's wary of allowing UTF-8 in scripts themselves: "Another possibility would be to allow UTF-8 encoding within strings in the source code. The advantage is that you will be able to view the strings in the appropriate editor. The disadvantage is that the script would contain a range of odd looking characters." I have thought that being able to limit what Unicode is permitted as a kind of security measure might be important, but there are higher priority issues.

  • "Scant" Evaluation - FWIW, I do not really believe in the premise of scant evaluation. I think if you do something like construct/only [foo: true bar: 'f] then you get FOO as the WORD! true and not the logic, and you deal with it with tests like == 'true. I guess I have to see more examples, but I've only seen this in module headers and I prefer the you-get-what-you-see version.

  • At a Class Crossroads - I just brought this post up recently.

  • Overwriting System Functions - He mentions the general problem that since modules import their own copy of system functions under a word in their own variable space, they won't see changes made to the variable in the system. He talks about making some kind of "search and replace re-set" that would find all those imported stubs and fix them up. I've been thinking in pursuing this through allowing variables to alias other variables, so what you import starts out by default as an indirection and sees changes in the original. But sSo far, if you want to hook a system function, in Ren-C, you have the option of HIJACK-ing the identity of the function value itself

  • Free Variables in Modules - I dislike these but the codebase has not been retrofit to not use them. Only LIB and SYS.UTIL are currently enforced to not be able to create them. But I think you shouldn't be able to by default anywhere (e.g. "strict mode" in JavaScript).

  • Source Reflection Returning Unbound Copies - Again speaking somewhat from a security standpoint, Carl points out that if a module gives you a function you can get at its source code, and if you can get at its source code you can get at its bindings, and if you can get the bindings then you can reach the data. I think being able to get the source code itself would be problematic in this adversarial situation--with or without the bindings. It would be nice if some day people go over the language and make it secure, but I think we're well into the phase where making it good and interesting is far more important.

  • Should a Module Be Loadable More Than Once - e.g. if you are using different versions as dependencies of other different modules. Hmmm.

  • Do We Dare Add ++ and -- - Ren-C addresses this with ME and MY. e.g. counter: me + 1, counter: me * 10. It fetches the value of the SET-WORD! on the left and substitutes that for ME. It's a lot nicer, and saves ++ and -- for weird symbolic purposes.

  • Deprecating CONTEXT - He's opposed to it because CONTEXT creates an OBJECT!, but there's no such thing as a CONTEXT! datatype. I'm opposed to it based on it being a noun. I'm thinking WRAP might be best applied to what this does.

  • How Best to Blockify? - He proposes BLOCKIZE for what I call ENBLOCK (e.g. envelope/enclose in a block). BLOCKIFY is something where if what you have in your hand isn't already a block, it will wrap it it one. :man_shrugging:

  • DIR? - I came to be of the belief that all directories must end in a slash, and we enforce this systemically. Hence DIR? is just a question of whether a FILE! ends in a slash or not.

  • PORT! Redesign Objectives - This probably belongs in the What is a PORT! discussion. More to add there:

  • Lexical Exceptions in LOAD - I'm against finding a way to load e.g. 1st. Ren-C gives you 1.st if you want it. This isn't a good investment of energy.

  • IO Devices in R3 - And all of the Device Model is happily gone. We use libUV now, and it's a lot better while still being standard C, cross-platform with many more features, and maintained by other people.

  • None Propagation - Ren-C has VOID-in-NULL-out and it's working splendidly.

  • Explicit Evaluation Terminator - After a failed attempt to convince people that vertical bar | would be BAR! and serve this purpose, we have COMMA!, and it has turned out to be a lot better.

  • R3-Alpha Released to Select Developers - This was June 2007. The Beta was expected on July 15, 2007, then delayed (?!) Well there never was a Beta.

  • Changes in MAKE FUNCTION! - Related to this, I just wrote a sort of new Dialected MAKE Manifesto

  • WITH and IN - I do admit to liking the do in context [...] construct. The IN construct is completely virtual binding now. BIND is still around but uses have been dropping as more and more of the codebase is able to work with virtual binds on top of largely unbound blocks of code, "viewed" several different ways without touching the binding bits.

  • UNSET! As An Argument - R3-Alpha got fairly permissive, e.g. allowing comparisons of UNSET! to other values. I've brought up that I think it may be a mistake to allow things like comparisons to take NOTHING, and offered justifications for that.

  • Modules Ripping Carl Apart - "I have to admit that so far modules have been, well, too much like taming a tiger. And, so far, the tiger has been winning... often, ripping me to pieces. The tiger is the main reason R3 is running late. Yes, blame the tiger. I've got many scars."

  • Admission that Unicode Was Harder Than Thought - " So, Unicode is the focus of our current development, and it must be clearly stated, this is a non-trivial project. Our goal is to have it ready for initial testing by the end of the month. I'll admit that we under-estimated the magnitude of the Unicode project..."

  • R3 Alpha Public Release - This was in January 2008. My first encounter with the Rebol language was approximately April 2008, and so I didn't use Rebol2 at all... believing from the community that Rebol3 was the hot new thing. I didn't realize it had only become public that recently, because everyone else who was involved and talking about it was on the inside track I guess. I don't know if this timing makes me fortunate or unfortunate, but it certainly explains why I never got attached to a Rebol GUI.

  • ISSUE! as a WORD! datatype - I disagree with this decision (as many did), and in Ren-C it is an immutable textual type..used as the implementation for characters as well (characters are just single-character ISSUE!s). I've proposed renaming these to TOKEN!.

  • BITSET!s - BITSET! is terribly inefficient and buggy, if you try to do things like XOR a negated bitset with another none of it works right. If your bitset is sparse (e.g. representing a Unicode character set, as one might do) then it will quickly consume megabytes. I did some work with Roaring Bitsets and linked them up as the implementation for bitsets, which worked well but I didn't feel like maintaining the dependency given higher priorities. But that branch is still around.

  • The ZERO? Question - I hadn't thought of this. POSITIVE? rejects non-numbers, NEGATIVE? as well, but ZERO? lets you test series as a simple synonym for = 0 but then also considers 0:00 to be ZERO? Ugh. I like TRASH? (~) as a placeholder better than zero for many reasons, so the single-character-zero-filler argument doesn't apply. I think ZERO? should apply to numbers only and error on other types.

  • STRING! is not BINARY! - Because R3-Alpha used variable sized encoding for strings inside the system, it couldn't offer you a usefully invariant binary view of strings. Ren-C can do it because it uses UTF-8 Everywhere, and if you're looking at a BINARY! alias of a TEXT! you can make modifications only if that atomic modification leaves the result as valid UTF-8. You can also alias WORD!s as TEXT! or as BINARY!, but it will be an immutable view. The benefits inside the system of sticking to one string encoding are innumerable and well worth all the work.

  • Script Error - Hadn't really thought about it but he has a point here. Something to consider in an ERROR! redesign, if such a design ever happens.

  • Pruning Down READ and WRITE - Despite earlier assurances that READ and WRITE give BINARY! and it's LOAD that does the fancy work, this shows some of the confusion slipping in about what READ is...and that's what provoked my own confusion at things like READ/STRING and READ/LINES and READ/AS. Red has these refinements too. It's all quite confusing, but Ren-C has the advantage of letting you do as text! read %some-file.txt which gives you a mutable alias over the UTF-8 data with no loss of efficiency. I haven't killed off READ/STRING or READ/LINES but now I feel I really should... that's LOAD's business to have some convenient syntax for. (Dialected? load %file.r or load [<text-lines> %file.txt])

  • VID Expression Optimization - So this is the first time I've felt there was something to comment on in a GUI post, since it's dialect philosophy. I started a thread on it

  • MAP! Indexing - Again, this is relevant to the determinism I've been discussing, and it's interesting to see there was questioning about it.

  • Multicontext Variable Lookup - Like it or not, this is what we have with Virtual Binding.

  • Find on OBJECT! - He mentions using IN. But with the binding logic of virtual binding, IN has to return the original item unbound if it wasn't found in the context... so it doesn't give a good answer. Ren-C uses HAS for this (null if a word not present, vs. the unbound word) and disallows FIND on objects. Use HAS.

  • RFC: func-local, funclo, funco, funo, funx - Wow, I didn't know the name considerations were so bad...before FUNCTION was chosen. Ren-C has virtually-bound LET and has dismissed with the bad idea of locals-gathering functions that gather all SET-WORD! as variables. That is not coherent... SET-WORD!s are used in places where they should not create variables (object keys, other dialects). So this idea is dead and will only be resurrected as a toy for code golf competitions or similar. And FUNC and FUNCTION are slated to be synonyms, as all abbreviations are intended to be.

  • Data Conversions that MAKE Sense - This definitely needs to be thought about, but also to eliminate needless duplication between TO and MAKE. I've written about the differences and the possible rules.

  • PARSE Project - While a lot of this has gone offline, I did preserve the remarks from the wiki on a Trello board (that seriously needs updating, now!). Sigh. Hurry up AI, I need you to edit all this for me.

  • Angry Comments About A30 Not-Quite-Public Release - Here it's January 2009. Rebol 3.0 was certainly positioned in a promise-making and not-delivering way, and Red followed that pattern. Part of what keeps Ren-C from being completely exhausting is not doing that.

  • Inklings of REWORD - It seems the REWORD we have today started with this discussion, right on the heels of the previous feedback also in January 2009. The design, authorship, and testing of these things simply add up to so much time... it should have been obvious to anyone reading this at the time this would never be finished. (I wasn't reading it, I didn't get involved until Rebol became open source in 2012...)

  • PICKing Negatives and Zero - Have to admit I haven't thought about this at all in years. Ren-C reverted to the Rebol2 behavior. In practice, I don't think it comes up enough for people to care.

  • Weird ASSERT/TYPE Refinement - I didn't like it and didn't see the point, so I killed /TYPE. Ren-C lets you do ensure object! spec and ensure [~null~ tuple!] spec.version ... you can do your own FOR-EACH enumeration over a block of type/value pairs if you have enough of them to matter.

  • Isolated Namespaces In Modules - Turning on isolation was horribly bloated, but Ren-C has solved it efficiently and makes it the default for all modules.

    • May I just say that the code for all of the module stuff--especially isolation--is convoluted and buggy with equally buggy usermode portions, and it literally made me give up trying to use my 2018 bootstrap executable for the FENCE! conversion. I had to go gut the module system to make enough simplifications that it would work at all, and am still dealing with the issues of deploying new cross platform binaries that will serve for at least another year or two.

    • If you're wondering "why not just use a modern Ren-C for bootstrap" the answer is that in most areas unrelated to modules--e.g. places where I worked on hardening it--I can rely on it more than the dark corners of modern executables. Bootstrap is rather demanding. Also right now has various in flux instabilities and bad performance.

  • DECODE and ENCODE - I never really got the split in responsibilities between ports and codecs. We're told that if you want to do streaming hash calculations in chunks on large files to use a PORT! to do it. But then DECODE and ENCODE aren't light wrappers over a chunking port but rather some very naive functions. Trivial design that was used as an excuse to make some more weird monolithic C code... there's no architecture, it's just a function table.

  • Read Only Strings, Blocks, and Objects - So apparently PROTECT came along in 2019. The interesting thing about Rebol being a C codebase with limited type checking is that it was scout's honor for all the code in the system to remember to check the protection bits before making a modification. Ren-C can build as C++, and enforces the checking of mutability bits before changes with the type system...so if anyone doesn't use a check routine to transition from const to mutable in the C, then when building with C++ that will be caught. I'm quite proud of that...and it has been working without bugs enforcing CONST as well as PROTECT (and evaluator holds, etc.)

  • PROTECT/HIDE - While I thought this was kind of cool when I first saw an example, it turns out that this is another features that was added on a whim without a systemic review of the implications. But for this case I haven't come up with a way to really make it much other than the scout's honor I describe from R3-Alpha's basic protection bit. I've not killed the feature because hidden bits are used to various effects in things like specializing fields out of frames, but I'm really not sure if it makes sense in other cases.

1 Like
2

(CONTINUED)

I HIT THE DISCOURSE POST LENGTH LIMIT!!!

  • Goodbye to FUNCTORs - I don't know what these were from the description. If it was one line of code, he should have put it in the blog. :pouting_cat:

  • Notes about SECURE - I kept this around much longer than I should have, but ultimately deleted it.

  • REDUCE/into and COMPOSE/into - My opinion on these is well known. Die, /INTO, die.

  • Minimal Readline - Whenever I go back to tinker with R3-Alpha I remember just how bad the console was. Modern Ren-C has what I think is a very good custom multi-line console... and it shares a huge portion of that code between the browser interface and the terminal. It isn't built with libReadline or anything like that, but even the C portions largely delegate to snippets of code bridged to Rebol to do much of the work.

  • Evaluation Quotas - This really shows how much Rebol wants to be an operating system and not a language. How many people when they run a Rebol process care about setting a quota on the number of interpreter evaluation cycles that run? It's not a metric anyone would know or care about. Start the process in an OS-sandbox that quotas on time or CPU use or memory use. If the OS lacks such features it's probably not used by professionals and who is going to care about this?

  • Relative filenames: Relative to what? - This recognizes the different needs. Ren-C has made some progress here and also in some places by using different datatypes. e.g. load <foo/bar.r> and load %foo/bar.r are different. There are oddities you encounter if you try to use the system/script/path-relative form in a function which may be called by another module...because you'll load relative to their path not the path of the one where the source is written. It's like you need another form like load 'this-module/<foo/bar.r>, where it has some variable it can latch onto to know what module you want to be relative to.

  • Paths for Blocks - We've been in a sort of stalemate over this, which I've written up in BLOCK! and Object Parity In Path Picking. I still think it should go by SET-WORD!.

  • REPLACE replace - The reluctance to just use PARSE with CHANGE here seems short-sighted. It's more performance focus when there are bigger problems.

  • OBJECTS: To COPY or not to COPY (member series) - Some of these questions feel kind of absurdist, and I definitely feel like some time clocked with serious projects in C++ or Haskell or whatever would help "un-ask" the question. If you don't have some methodized code coordinated with the kind of cloning intention you have, you can't know what the intent is. If you don't know the intent, the best you can do is as little as possible.

(There's a bunch of PARSE-related entries I'm just going to skip because UPARSE is fully understood and the most well-tested Rebol dialect ever made.)

  • Another EXPORT Method - I like this form of exportation very much, e.g. export emit: funct ["Emit HTML...]. I don't remember if it existed in R3-Alpha, but regardless Ren-C's implementation is based on entirely different code.

  • UNSET! is not First Class - This was used as a reference when trying to reason about NULL, in a time before the whole families of "can't put it in a block" antiforms arose. I notice the disclaimer on that outdated post is out of date. Sigh.

  • Pairs as floating point? - Not even Red cared about this, I removed the feature.

  • So, Do You Want RESET - Rebol trying to be an operating system, again. I've mentioned how if you build to WASI and run your web code there, you get the fast tear down and startup for free. Pitching this kind of feature, this late in the game, seems preposterous.

  • About /LOCAL Words - Ren-C of course dismissed with the idea of not using a different part of speech to switch into locals in the function spec dialect. There's nothing wrong with [a b <local> c d] instead of [a b /local c d], is there? Well of course it was rejected due to the fact that /LOCAL is a refinement and hence was based on an interned word, while each <local> tag took up space as a string. What I didn't know was the sense of urgency that was felt about callers being able to slip values into the locals from a security standpoint: "We need to address this issue immediately." My impression was that there was no urgency...as it was never fixed.

  • Amiga PowerPC Version Released - So here we can just sort of admit that this is a hobby project.

1 Like
3

Well, That Was...Long

It didn't take days to go through. But like, half a day.

I didn't get involved in any of this until after the open-sourcing, at which point he gave up and there weren't really any more design posts. Previously I didn't feel any real motivation to read the blog--being as long as it was. I just read entries that were linked from issues or posted in discussions as I became more familiar with the topics being discussed.

So this has been interesting to me because I knew all the features (well the ones that weren't deleted before the open source release, at least). But I didn't have context for in what order they were done. Finding out the order certainly is telling about why things stalled out the way they did.

Moreover, I know that without being willing to at least dual-build your C codebase as C++... that trying to graft something like changing Latin1 interpretations of strings into unicode--and being sure all your callsites are getting it right--will be a hope-draining and torturous endeavor. Without its C++ build, there could be no Ren-C... and certainly no UTF-8 Everywhere!

I'm sure it was very hard for Carl to admit defeat after all of this. Rebol 3 just didn't come together. It was plagued by the vicious module system, Unicode support, and of course...trying to make an all new GUI at the same time. In the end, I think the main persistent advancements were the new features designed and tested in PARSE.

But there was another advantage R3-Alpha gave: It generated a lot of usermode code, demonstrating various wacky attempts at how to implement mezzanines. These bits of code were truly on the edge of any Rebol that had ever been written...and had to stay working in order to boot the system under any change. So for me, it set up an uncountable number of puzzles to dig through when tweaking any feature. And as I tweaked, I used the C++ language to my advantage...still following the rule that it can build as C.

Plus there were at least a lot of tests. Not necessarily great ones, but many that arose from bugs showing ways a lot of different people might try to use the language. Still more puzzles to meditate on, and be forced to rethink in order to keep them working under interpreter enhancements.

So Ren-C really only exists because R3-Alpha was a codebase that had gone insanely out of control in its aspirations, leaving piles of puzzles in its wake. I'm proud of quite a lot of what I've done in my years of working on it. Hopefully others with a stake in Rebol--including Carl--will sometime get a chance to appreciate some of it too.

2 Likes