Pivotal Design Question: Is Evaluator State Just A Block?

Usage Philosophy
1

I ran into a hitch with LET and EVALUATE in single-stepping.

It raises a pretty big question about how much we want to tie the hands of the evaluator in favor of "simplicity".

The Problem

On the surface a LET statement might seem impossible for step by step evaluation:

>> block: evaluate [let x: 10 print ["X is" x]]
== [print ["X is" x]]

That LET statement declared a variable, but where did it go? It only lives until the block is over. It would seem that the PRINT is out of luck.

But... I could make it work since a BLOCK! can carry along virtual binding state. So, at each step you just get a little more state added on. The [print ["X is x]] is different from the block you'd have gotten from saying SKIP 3 on the full block, due to this binding.

But what if you reposition the block?

>> block: head block
== [let x: 10 print ["X is" x]]

Now you have a block that has X defined in its bindings, and if you step through it you'll define it again.

Can This Be Solved?

I think the cleanest and clearest way to solve it is to rethink EVALUATE so that it operates on a FRAME!...not a BLOCK!. This would match the internal model better.

Today, we have to tear down a frame and build up a new one each time you do a step. This would say that you'd be keeping it alive.

You'd be limited in terms of being able to look back over past values you had already evaluated. That limitation would keep you from rewinding... if you wanted to go back and do things over, you'd have to do that by working with your original block that started the whole process.

What you'd be able to do in terms of looking ahead would be more like what a variadic is able to do today.

The Big Philosophical Question

I guess the big philosophical question is not necessarily so much about LET itself, but should we rule out the existence of things like LET in general.

In other words: is it imperative to step the evaluator across a block and not accrue any state particular to that evaluation?

I've been kind of looking for a convergence between things like the evaluator and PARSE, and so asking what the restrictions on the evaluator are may be asking what the restrictions are on anything that tries to leverage the FRAME!-based processing of blocks...for tracking positions, giving errors, etc. If we require them all to be amnesiacs after every step, this would make it hard to write things like the COLLECT/KEEP feature in parse with the rollback feature...because it would have to record its state in some external thing.

@rgchris --^ please see that and think about it. If EVALUATE returned a FRAME! representing the block and not a new position of a block, what kind of disruption would it be? Do you see the accrual of state in the evaluator to be something that should be ruled out--thus killing off LET or anything like LET--to be worth it to have the feature of a memoryless evaluator?

My leaning on this is to say that we would be crippling the language by ruling out LET-like things in the design.

Right now, I have this test code working. Note it's three evaluation steps, because the LET is actually invisible (1 unit lookahead to see x and add the binding, then leaves x: 10 to run normally)

x: <in-user-context>
output: '~unset~
block: evaluate evaluate evaluate [let x: 10 output: x]
did all [
    block = []
    output = 10
    x = <in-user-context>
]

And when you look at some of the other designs of how this is plugging together, I don't think we should turn back. It's simply too hard to build abstractions on top of FUNC if the bodies cannot dynamically declare new variables, and I think forcing everyone in those situations to deal with USE is ergonomically just too awkward to feel like the language is living up to its promises.

I don't want to give up on virtual binding and LET when it has come this far. It may be broken, but its brokenness is already a better kind of broken than what was there before...and there's no proof yet that it can't be made better.

Rebol's M.O. has been throwing imaginative but I want the code to look like THIS at a data structure and see how far that can go...without proof that it can or should work well. Every now and again I think I should have the right to throw my own bad idea that looks good in there. And maybe some poor sucker in the future can figure out the limits of how it can be made to seem like it works more. :slight_smile:

2 Likes
2

I don't feel I understand FRAME! enough to appreciate its applicability. It would be preferable that the position returned from EVALUATE be the same series as the input. Presumably you'd have the same issues if you were evaluating a string:

text: evaluate evaluate evaluate "let x: 10 output: x" => ""

I apologise if I'm not reading this correctly, but I can see the similarities in my use of STATE in my Parse Machine—the actual currency of which the operative series becomes just one component. In which case it wouldn't be the worst thing if the return was FRAME so long as you can still somehow access the new position of the original series.

What happens currently if you do [let x: 10 let x: 20 probe x] ?

3

You can experiment with the current state in the ReplPad.

What happens is that says X is 20. Mechanically what's going on is that you are creating two variable "patches", and the most recently added one is the highest priority.

Between the two X's, any X would latch onto the state prior to the second LET.

‌>> code: []
== []

>> do [let x: 10, append code [x + 1], let x: 20, probe x]
== 20

>> do code
== 11

My broader point is "being able to do this kind of thing is more important than the evaluator being enforcedly simple in design to prohibit it".

It may mean that those who wish to use a "hooked" evaluator have to bend a little more to its interface, but I think that if evaluator hacking gets more nuanced that's all right, if it means giving better user experience on the whole.

4

So this actually exposes another point: should all evaluative steps be forced as a model into being steps on the original series at all.

For instance, let's imagine I write a macro:

macro: twothings [x] [
   return [print "Hi!" 1 +]
]

The concept behind MACRO is to splice code into the execution stream. But how can you single step across that?

  • We can argue such features are not worth their complexity and forbid them. Such splicing is disallowed, because a step of execution no longer correspond to the original series.

  • Or, we can allow them but only consider behaviors which force through the point of making "a step" match an input series position (e.g. run all the MACRO expansions in a step...including anything that adds partial material like the 1 +, has to go ahead and finish running until it gets to something that is an actual position in the input series again).

  • Or we can say that a FRAME!...once running...is operating on a virtualized series that you must process through the evaluator's point of view.

The evaluator restricts lookahead to help protect you from a "mirage". Instructions downstream can have different meanings based on context...if you don't know about COMMENT you might not realize that the thing you looked ahead and saw isn't going to be part of the evaluation at all. Things like MACRO are also part of that.

(Another reason why you can only do one unit of lookahead has to do with being able to run code as a C variadic...a va_list cannot be traversed backwards. So the frame only holds one item at a time. This has become less relevant as performance has taken a backseat...frequently variadics are just converted into blocks anyway.)

We should dig up some various "DO/NEXT"-style usages to better understand what people are wanting to accomplish with the ability, and what the expectations are. What has it been taken for granted traditionally that worked (that may have not, strictly, "worked") and what kinds of features must be forbidden or limited somehow to keep those expectations going...or if things should be done another way.