EVAL + EVALUATE and REEVAL + REEVALUATE

[hostilefork]

Internally to the Ren-C code, the term "Eval_Xxx()" has come to imply one step of evaluation, while "Do_Xxx" will take in something like a block and always execute it to the end. Hence a DO is a sequence of N "steps" of EVAL.

I mentioned in "Re-imagining DO/NEXT" why changing the interface was needed, so that DO/NEXT became a new routine called EVALUATE:

 >> var: null

 >> block: [1 + 2 10 + 20]

 >> block: evaluate/set block 'var
 == [10 + 20]

 >> var
 == 3

 >> block: evaluate/result block 'var
 == []  ; Notice it did not go directly to `null` when there was a result

 >> var
 == 30

 >> block: evaluate/result block 'var
 ; null  ; null is only signaled when the evaluation had no result

 >> var
 == 30  ; ... in which case the variable you passed in is undisturbed

It has a subtle behavior of leaving a lingering "empty" step before returning null and leaving the variable as-is. This turns out to be foundational. It allows invisibles to remain pending and not run "too soon" (imagine if the last step was actually [comment "hi"] and not [], e.g. not being at the end of the block isn't the only way you can wind up in a situation where there's not going to be a value coming from the next evaluation...so the steps need to be consistent).

For this to work, it really does require you to pass a variable in. Even if there was a clean implementation of multiple return values, you don't want to overwrite a previous result on a last potentially-vaporizing step. (The mechanics of how this is done inside the evaluator are actually rather slick, but beyond the scope of this post. Userspace you have to do it by passing in a variable that is potentially left as-is.)

Naming Collision: EVAL

When EVALUATE came on the scene, there was already a native called EVAL which did something useful...also, exactly one step:

 >> x: null

 >> eval (lit x:) 1 + 2
 == 3

 >> x
 == 3

Hence this EVAL is a very tricky variadic that takes a first normal (hence evaluated) argument, and then re-evaluates it as if the expression had been there all along. You'd have quite a hard time writing such a thing yourself (!) but what it does is use some slippery code to re-feed the evaluator with a previous output cell.

However, I'm not very comfortable with having eval be anything other than a shorter name for evaluate. So what if we made those two synonyms, and called this one reevaluate with the shorthand reeval?

I think that cleans up some confusion, and would bring the userspace terminology more in sync with the C implementation.

[hostilefork]

It has a subtle behavior of leaving a lingering "empty" step before returning null and leaving the variable as-is. This turns out to be foundational . It allows invisibles to remain pending and not run "too soon" (imagine if the last step was actually [comment "hi"] and not [] , e.g. not being at the end of the block isn't the only way you can wind up in a situation where there's not going to be a value coming from the next evaluation...so the steps need to be consistent).

For this to work, it really does require you to pass a variable in. Even if there was a clean implementation of multiple return values, you don't want to overwrite a previous result on a last potentially-vaporizing step.

This situation has been giving me a headache. The problem is:

• There is by definition no single return result value that can indicate an invisible evaluation occurred. ANY-VALUE! is a legal evaluative product, and since it is an evaluation NULL is also a legal evaluative product. This means we can't use the "null means nothing" trick that applies many other places.

• If EVALUATE processes invisibles and throws away their results at the tail of evaluations, it may run them too soon. Consider a branch like case [true [print "True"] elide print "Elided"], you don't want to see the output of the elided. Hence you cannot merge invisible evaluations with the left side...they have to wait.

• If EVALUATE processes invisibles and throws away their results at the head of evaluations, it may run them too late. Consider a currently topical case of print ["a" _ comment "hi" _ "b"] wanting to treat literal blank! as a space, but evaluative blank! as nulls. The first blank can be recognized literally, but then if the comment "hi" runs and pulls in the ensuing _ with it automatically, then the second blank will seem evaluative when it was meant literally.

This doesn't need to be a problem inside of the evaluator. Discerning the situations isn't a problem, because there's a cell bit (CELL_FLAG_MARKED_STALE) to track when output cells are tainted by invisibility. The clever optimization is that they're never actually overwritten...it's a nifty stunt.

But exposing this bit to usermode would seem to need a third variable to the output of EVALUATE

block: ["a" _ comment "hi" _ "b"]

>> [block value was-invisible]: evaluate block
== [_ comment "hi" _ "b"]
>> value
== "a"
>> was-invisible
== #[false]

>> [block value was-invisible]: evaluate block
== [comment "hi" _ "b"]
>> value
== _
>> was-invisible
== #[false]

>> [block value was-invisible]: evaluate block
== [_ "b"]  ; now we get a chance to see _ literally
>> value
== #[void]  ; or null ...?  Void catches if you don't check `was-invisible`
>> was-invisible
== #[true]

That's starting to feel super wordy and all the conditionals of handling starts looking like a mess. But from a technical standpoint, it would would give enough information to clients so they could successfully navigate invisible evaluations in their dialect...if they wanted to support it. (If they didn't process the was-invisible status, they'd likely error on the VOID!)

Is there better? I think so...

What if a Quoted Return Position Signaled Invisible Values?

Quoting is a cheap place to put a bit, that also fiddles the datatype in a checkable way. So let's look at the impact:

block: ["a" _ comment "hi" _ "b"]

>> [block value]: evaluate block
== [_ comment "hi" _ "b"]  ; not quoted, result is visible
>> value
== "a"

>> [block value]: evaluate block
== [comment "hi" _ "b"]  ; not quoted, result is visible
>> value
== _

>> [block value]: evaluate block
== '[_ "b"]  ; a-ha, quoted so the void result is actually invisible...
>> value
== #[void]  ; all invisibles are voids, but not all voids are invisible

What's neat about this is that you have a condition you can test to decide if you want to discard a result that comes back on the main return result:

 ; This is how your dialect can just skip over invisibles
 until [not quoted? [block value]: evaluate block]

That won't be right for every application--for instance the PRINT case I'm talking about that wants to discern evaluated BLANK! from not evaluated blank. That requires being able to say "okay, the last step was invisible so ignore it, but look ahead at what we've got in front of us literally".

So that's a solution to what I used to call the "SYNC-INVISIBLES problem". A dialect author who wants to mesh evaluation with their own secret keywords or processing needs to know what the evaluator knows, and I think this is nailing it down.