Simplifying Refinement Promotion

There are two modes of partial specialization which Ren-C has supported.

One form would be like if APPEND takes something like :DUP and fixes it to a value:

>> append2: specialize append/ [dup: 2]

>> append2 [a b c] <d>
== [a b c <d> <d>]

Then there's a trickier kind of specialization, which is to ask for the parameter but not specify it... thus just increasing the arity:

>> appenddup: append:dup/

>> appenddup [a b c] <d> 4
== [a b c <d> <d> <d> <d>]

The second form is pretty unusual in the language world. There doesn't seem to be much prior art in "the conversion of optional parameters to required parameters" (at least I don't know any, and the AIs I asked don't know any).

Since it's pretty different from what what people think of as partial specialization, let's call it "refinement promotion".

Refinement Promotion Is Tricky

Something that has been non-negotiable in the design is the straightforward array of parameters and locals that specify a function interface.

Refinements are defined in some order in that array. But you are not required to use them in that order.

So consider a definition like:

 /foo: func [a [integer!] :b [integer!] :c [integer!]] [...]

Someone might do refinement promotion of this as foo:c:b/ - this makes it seem to the caller a function originally written with spec:

 [a [integer!] c [integer!] b [integer!]]

The techniques so far have tried to mimic the way that refinements work. So get $foo:c:b would produce a function that was accompanied by a [c b] array, that would get pushed when running the promoted function. The elements in the array would not just have the words, but those words would have binding information to say what index those words were found at in the parameter array.

But since the underlying array is left the same, this means every time you want to know something like "what's the first unspecialized normal argument" you have to mimic the refinement gathering process. It convoluted the process quite a lot, and really went against the idea of the implementation being "simple".

Q: Is This Really Required To Support? (A: Yes)

This may not seem like a super-common need. But if you're implementing a dialect that wants to support calling functions with refinements, it's pretty important.

Let's say you're implementing something like the feature in UPARSE that lets you call functions:

>> data: copy ""

>> parse ["a" "b"] [some [/append:dup (data) text! (2)]]

>> data
== "aabb"

Basically, if (get $/append:dup) can come back with a function that you can query for its parameters and get answers just like it was any other function, then support for refinements comes basically for free.

Should The VarList Just Be Rewritten?

If you look at what a modern SPECIALIZE followed by AUGMENT can do, they can hide parameters...and then add back parameters with the same name. Which parameters are visible depend on the "phase" of the frame.

So why couldn't refinement promotion be done just by making a new function interface that removes the argument as a refinement, and adds it back as a regular argument... then has a dispatch phase that moves the argument data to its old position for the subsequent phases?

It's not particularly "cheap" to do that, space-wise. You'd need a new VarList* and a new Phase*, and the Phase would have to remember the new and old positions to do the rewrite. But it would make parameter enumeration blunt and simple, because you'd really just be enumerating the parameters in order.

There'd be some cases where the position of the refinement would allow it to just be naturally rewritten to be a regular argument, and that could be optimized for.

What About When You Have Lots of <local>?

This is kind of the dark side of the simple FRAME! model, which is that if you use it to create a lot of local variables, then operations like SPECIALIZE and AUGMENT which do VarList manipulation have to make copies of everything for the new VarList...including a bunch of locals that aren't changing at all in each new form.

/foo: func [x [text!] y [tag!] <local> a b c d e f g h i j k l m n o p] [
    ... 19 frame cells (includes RETURN) ...
]

/bar: augment foo/ [z [integer!]]  ; z is last item in new 20 item frame

Refinement promotion would become another one of these situations that would do seemingly unnecessary duplication.

It would be possible in cases like this to create smaller frames and then proxy the results into larger ones, essentially simulating what a user might do to manually call FOO from a new function BAR which had a frame with 3 elements.

/bar: lambda [x [text!] y [tag!] z [integer!]] [
    foo x y  ; imagine doing this, but with faster internal mechanics
]

Some calculation could be done where the size of the frame justified it. I have a feeling that the frame would have to be reasonably large before a technique like this would be beneficial.

What About Refinements At Head?

Well, in that case, you would have to build a new VarList* that extracted the arguments, and then proxy them into position for the new interface.

At least one wouldn't be worried about the "bloated copies of locals" situation.

So...Use The Auxiliary Array Simulating Refinements?

The code for simulating refinements when asking simple questions like "what's the first unspecialized normal arg" is unappealingly complex. :nauseated_face:

Making it further unappealing is that when you have this array of refinements "off to the side" but still allow people to fill in slots in frames to specialize out arguments, you end up needing to have "reconciliation"...because those frame slots that are referenced by this out-of-band array are no longer part of the refinement promotion.

>> f: make frame! append:dup:part/  ; has auxiliary [dup part]

>> f.dup: 3  ; what cleans up [dup part] to just [part] ?

I've talked about not knowing about what "moment" to do these kinds of fixes, and I'm increasingly looking for ways to avoid there being any such moment. If the physical experience of the frame was that DUP and PART were ordinary parameters and not refinements, then it "just works".

The "Dumb" Mechanical Answer Is Likely Best

I sometimes forget just how much I take for granted in Ren-C, regarding the ability to compose functions together.

The "inefficient" idea of making a new parameter list and then proxying the arguments into position would be more efficient than having to create and evaluate an interpreted function that had to manually copy the parameters.

There's a huge tax created by having to compose an off-to-the-side parameter reordering list in with the frame variables, and that tax is paid by any code that wants to interpret the list. It's just too big a tax to pay.

It pains me a bit to delete it, because it was hard to write and seemed clever at the time. But techniques have advanced...and while the auxiliary list may have seemed somewhat optimal for storage, it's no longer the right choice.

1 Like

There May (?) Be More Than One Kind of Promotion

Currently there is some code which lets you "build a frame by example":

>> make frame! [match integer!]
== #[frame! [
    test: &[integer!]
    value: ~
]]

Even though MATCH takes two arguments, the evaluator went as far as it could and then stopped, leaving extra arguments unspecified.

But... what about refinements that take arguments?

Historically what you would get with refinement promotion would move the DUP to the head of the parameter list and make it not a refinement... leaving :PART and :LINE as refinements:

>> make frame! [append:dup [a b c] [d e]]
== #[frame! [
    series: [a b c]
    value: [d e]
    dup: ~  ; now has an associated PARAMETER! that's not a refinement
    part: ~  ; still a refinement
    line: ~  ; still a refinement
]]

But if you're just looking at this as a frame without comparing the parameter list, it doesn't seem different from how it would look if you hadn't specified the :DUP at all.

In order to not lose information about this specific instantiation, we need what might be called "narrowing refinement promotion":

>> make frame! [append:dup [a b c] [d e]]
== #[frame! [
    series: [a b c]
    value: [d e]
    dup: ~
]]

This means any refinements you didn't specify are gone from the interface. Because you're trying to encode the intent of an invocation.

"The user has spoken: They want APPEND:DUP"

What If This Was The Only Refinement Promotion Type?

From the earliest days of FRAME!, I've been assuming that this is what should happen when you MAKE FRAME! on APPEND:

>> make frame! append/
== #[frame! [
    series: ~
    value: ~
    part: ~
    dup: ~
    line: ~
]]

But in an parallel universe, what if I had assumed instead:

>> make frame! append/
== #[frame! [
    series: ~
    value: ~
]]

>> make frame! append:part:dup/
== #[frame! [
    series: ~
    value: ~
    part: ~
    dup: ~
]]

>> make frame! append:dup:part/
== #[frame! [
    series: ~
    value: ~
    dup: ~
    part: ~
]]

The Latter Is The Only Use for Refinement Promotion (So Far)

e.g. I've never actually wanted to make a refinement that takes an argument become non-optional, unless the intent was specifically to model an invocation (that wouldn't be using any other optional parameters).

I do know I still want to be able to get a copy of a frame I can fill in all the fields as I wish. But if I'm doing that, I'm not passing a refinement chain. I either fix parameters completely...or not at all.

Anyway, MAKE FRAME! should be consistent in terms of how it answers. If it's narrowing when given a variadic example (which it must be), seems it should be narrowing when given a refinement chain.

MAKE can legally perform evaluations:

>> make frame! [append reverse [a b c]]
== #[frame! [
    series: [c b a]  ; had to evaluate to get that
    value: ~
]]

>> make frame! append/
== #[frame! [  ; should be consistently narrowing
    series: ~
    value: ~
]]

COPY FRAME! doesn't set PARAMETER! fields to ~ antiforms... and it shouldn't. And TO FRAME! of a FRAME! has to be a synonym for COPY.

So... uh, name that operator:

>> XXX append/
== #[frame! [
    series: ~
    value: ~
    part: ~
    dup: ~
    line: ~
]]

This has been a bit of an eye-opener. :eyes: I'm going to have to think about it.

1 Like

So I'm thinking about the guts of the system, for things like:

 >> if (try second ["a" "b" "c"]) (print/)
 b

What happens in the guts of IF when you use an action as a branch, is that it wants to simulate a call with one argument... but not raise an error if it's arity-0.

Let's imagine instead of PRINT you had something with a refinement, followed by a normal argument:

demo: func [:refine [integer!] arg [text!]] [probe arg, probe refine]

So we'd expect this behavior:

>> if (try second ["a" "b" "c"]) (demo/)
"b"
~null~  ; anti

Okay, but... when did the narrowing happen?

It didn't happen when you said demo/. If you use terminal slash to fetch an antiform frame, the recipient expects all of it.

There's some moment in the branch processing which said "I'm ready to invoke, make me a narrowed frame".

And the frame it got needed to look like:

#[frame! 
    arg: ~
]]

That frame is what's examined and filled by the branch processing. If instead you had passed demo:refine/ it would look like:

#[frame! 
    arg: ~
    refine: ~
]]

Assuming refine is not "endable" ("unspecifyable"), then filling the first argument but not the second would result in an error when you tried to invoke it.

So "GET" Does Not Narrow

Let's say I were to write append:line/ and pass that somewhere.

That function has :PART and :DUP still available.

So you've "made" a frame from a CHAIN!. But you didn't make a narrowed frame, and you didn't replace the PARAMETER!s with nothing.

This is the classic sense of "GET". Hence there's a difference between:

  • get $foo:bar (a.k.a. foo:bar/)

  • make frame! foo:bar/ (a.k.a. make frame! get $foo:bar)

The former will promote BAR to an ordinary argument, but leave all the other optional parameters alone.

The latter will promote BAR to an ordinary argument, but null out all the other optional parameters as if they'd been specialized away.

Perhaps Not Profound, But...

This explains a bit about how it is that a function like IF is able to tell what the first unspecialized argument is.

The moment it "made a frame", any optional arguments were nulled out.

I'm coming to think that the operator is actually:

>> XXX append/
== #[frame! [
    series: ~null~
    value: ~null~
    part: ~null~
    dup: ~null~
    line: ~null~
]]

If I'm intuiting things correctly, there is no moment at the system in which frames are manufactured with nothing (antiform ~) in refinement slots, unless those slots are the result of narrowing.

Hence the typechecking concept of defaulting nothing for refinements to null is unnecessary/incorrect.

Natural consequence is:

>> frame: XXX either/
== #[frame! [
    condition: ~null~
    okay-branch: ~null~
    null-branch: ~null~
]]

>> frame.okay-branch [print "truthy"]
>> frame.null-branch [print "falsey"]

>> eval frame
falsey

This seems a bit unfortunate when we could have caught the failure to assign the condition, but, I think it's the only reasonable answer.

I guess the next question is if more functions should disallow null. Should INTEGER? accept null, or should you have to say (integer? maybe null) so you're actually testing VOID? Starting to look that way, isn't it...

Turning my head sideways and looking at the range of uses, I'm wondering if there really is just one MAKE operation for FRAME!, it is just sensitive to whether an argument is mandatory or optional.

>> make frame! append/
== #[frame! [
    series: ~
    value: ~
    part: ~null~
    dup: ~null~
    line: ~null~
]]

>> make frame! append:dup/
== #[frame! [
    series: ~
    value: ~
    dup: ~
    part: ~null~
    line: ~null~
]]

Hence you can still glean from the frame whether a refinement was used or not. But you're not prohibited from overriding it if you like.

I'm having a little bit of deja-vu about this, like maybe this is a design point that was passed through at some time. But I don't actually know if it has been. This ties in with "frame lensing" in a sense that if a lens gets applied those refinements will actually vanish... but since COPY FRAME! and MAKE FRAME! have diverged in terms of their intents, you wouldn't be using MAKE FRAME! if your intent is to create a derived function.