Should append null fail, append BLANK! add nothing unless /ONLY?

[hostilefork]

The Rebol design concept that used to be discussed under the label "none propagation" has become a system of what I think are pretty good rules:

• Blank input gives Null output...for non-mutating operations
• Mutating operations do not accept BLANK! targets as input
• Null input gives an error for operations that aren't specifically about null handling
• TRY can be used to "defuse" nulls into blanks

The reason this has merit is because it gives you a "hot potato" in the form of NULL, without going so far as to error in the moment. You get pinpointed error locality where a problem might be, and have an easy mitigation--if you want it--to use TRY to turn the null into a BLANK! and pass through all other results. And there are a host of null-driven operations to let you pick up the slack, e.g. first block else [code].

(I've suggested that if you ever actually wind up having to put a null state in a variable, test for null later with NULL? and act on it, that you probably could have done it better...reacting to the null without ever using it as a stored state. But it's your choice, I guess.)

So this is being applied across the board. first _ is null...just like first [] is.

Consequences for APPEND/INSERT/CHANGE...

These rules affect a current feature of things like APPEND that I doubt many people have tried:

 >> append copy [a b c] null
 == [a b c]

>> append copy [a b c] if false ['d]
== [a b c]

When it was introduced, I thought it made sense. Since NULL is not a value you can put in a block, you don't get confused about what that means. Then since BLANK! is a legitimate value that can appear in a block, you could append it:

 >> append copy [a b c] _
 == [a b c _]

But quietly accepting NULL loses the hot-potato benefit. If you say append foo third block and there is no third element in the block, doesn't that seem like a good time to complain? Rebol code can pretty quickly get hard to read and debug, and wouldn't it be nice if the person who wrote that line could articulate that there may not be a third thing in the block by saying append foo try third block?

This suggests making the default behavior of BLANK! to be to add nothing, and you need an /ONLY to override that. It may sound like a missed opportunity for null to exploit its out-of-band status. But realistically speaking, I don't think purposefully adding blanks is all that common--certainly a lot less common than adding a block as-is, and that requires /ONLY!

No one probably cared (...yet...)

Like I say--most people probably were unaware of this, and so I don't know there's a lot of append block if condition [...] out there today. We might need APPEND/OPT to be null tolerant, though the only reason to use it would be with /ONLY since otherwise you could say append block try if condition [...].

[hostilefork]

So the answer, after delving and thinking on this issue, has come up as NO.

NULLs will continue to append or compose without error.

>> data: copy [a b c]
>> append data null
== [a b c]

>> compose [a (if false [<omitted>]) b]
== [a b]

But if you have a BLANK! in your hand, that is a "normal" value that will be COMPOSE'd in, or APPEND'd, etc.:

>> data: copy [a b c]
>> append data _
== [a b c _]

>> compose [a (_) b]
== [a _ b]

It's your job to OPT something that might be blank, if you want blanks to be turned into NULLs and evaporate.

Sound a bit awkward, given how many routines return BLANK!s when they fail? Well, no worries...

NULLs are taking over a lot of former BLANK! roles

A failed ANY or ALL, a FIND that didn't find anything, a MATCH that didn't take... these will all be returning NULL, and not blank. This is the new return value protocol, where BLANK! is no longer used as a return value meaning "no match". And it will be tolerable because NULLs will be conditionally false.

This gives BLANK!s a renewed purpose, but a purpose for which they should be thought of as "things".

It's a good feeling to pin down this previously oft-speculated-upon question, since it really was inelegant to have to say compose [a (opt any [...]) b], etc. But we need wonder no longer...you will be able to omit the OPT because ANY will now return NULL!

[hostilefork]

NULLs will continue to append or compose without error.

This now has a fairly strong philosophical defense in NULL, first-class values, and safety.

The way I've started to feel comfortable with this is saying that NULL is what you use when you're talking about a "what". You can use a null to opt-out of that. But you can't use a null to opt-out of a "where". Only BLANK! can do that.

But if you have a BLANK! in your hand, that is a “normal” value that will be COMPOSE’d in, or APPEND’d, etc.:

This is a really tough one to think about.

As the rules have shaped up, I often feel this should probably change to requiring an APPEND/ONLY or COMPOSE/ONLY, otherwise blanks should act like nulls. The premise is this:

It's likely better to embrace blank's "nothingness-without-triggering-errors-on-access" as its primary facet, with its ability to be represented in a block as secondary.

It's hard to see a clear answer purely from a nuts-and-bolts point of view. But it just seems like it's more natural. Consider:

  foo: try case [...]
  ...
  compose [... (opt foo) ...]

There's arguably a nice aspect to the annotation that it's optional, and you might say the random blanks popping up to tell you that you left off an OPT is a feature and not a bug.

But in practice I'm not feeling that. I didn't invoke the TRY because I wanted to create a thing. I put the TRY there because I wanted to say that my tolerance of nothingness was intentional, and I expect this variable to be defined...but conditionally false.

If I were nervous and felt I needed some kind of additional signaling not to use it in an optional sense, I'd have left off the TRY...so foo would have been unset. Why should I have to repeat myself by way of an OPT annotation when I use the variable in this casual circumstance? Doesn't it kind of undermine the convenience of having blank in the first place?

I've made the argument that wanting to append literal blanks is probably less common than wanting to append literal blocks, and that takes an /ONLY. You can append a literal blank as easily as append block [_] if you want to, so it's not like there aren't good options for the cases when you need it.

What's tough about it is that it is a slippery slope. This makes one think "oh, well then should refinement arguments go back to being blank when they're not supplied?" And some of these things that sound good then turn around and bite you because conflating values with nulls loses the rigor of null and compromises the features it enabled. :-/

It's definitely a serious consideration to change this. So please speak up with any observations...

[hostilefork]

"It’s likely better to embrace blank’s “nothingness-without-triggering-errors-on-access” as its primary facet, with its ability to be represented in a block as secondary."

To put myself on the other side of this statement: both aspects are going to come into play, whether you like it or not. It's not going to be all one way or the other. You're going to sometimes be thinking of a blank as a "thing", and sometimes as a "nothing".

Let's look away from APPEND and COMPOSE for a moment, and think about maps. Today these are different:

my-map/key: _ // sets the key to a blank value
my-map/key: null // removes the key

Here you don't have the option of an /ONLY. So if blanks were treated as nothings, you'd lose the ability to put them into maps with this mechanism. But as long as they are distinct, you have the choice to say either:

my-map/key: thing-that-might-be-blank
my-map/key: opt thing-that-might-be-blank

Going back to APPEND and COMPOSE, the /ONLY is already controlling one aspect: splicing or not. But...

• ...what if you want to splice if it's a block, and treat it like a literal blank if it isn't?
• ...what if you want to not splice if it's a block, but treat it as nothing if it's a blank?

When in Doubt, Favor the Mechanically Consistent Behavior

Sometimes things seem annoying when you look at them as individual cases, but you realize their necessity if the big picture is taken into account.

I think needing OPT at the callsite is just one of these necessary evils--that's not even that evil.

For historical perspective: When I initially complained about NONE!s not "vaporizing" in things like COMPOSE, BrianH was very adamant that NONE!s were things. He believed a lot of Rebol's idiomatic behavior involved having empty placeholders in slots of regular record-like structures. So he was generally resistant to using them to opt-out when blocks were being produced.

So I guess this just kind of informs what the system's bias is: Blanks are values first and foremost, and nothings second.

[hostilefork]

Blanks are values first and foremost, and nothings second.

I just had a weird idea on this that I'd not had before: make words and paths that look up to blank yield null by default, and need a GET-WORD! to say they're actually BLANK!.

>> x: _
== _

>> x
; null

>> :x
== _

>> get 'x
== _

>> y: null
; null

>> y
** Error: y is not set

>> :y
; null

So blanks themselves are inert and have no evaluator behavior. And if a blank comes back from a function call you get it. But plain words and plain paths that look up to blank variables evaluate to null. While plain words and plain paths that look up to null variables error, and plain words and plain paths that look up to ACTION!s run them.

Zany, I know. But what this means is that while blanks can come back from a function or evaluation, you wouldn't get them out of an ordinary variable unless you were kind of trying.

You might think of it as an enhancement of "blank-in-null-out", just bringing the feature to ordinary variable references.

I actually thought of this while thinking about mapping blanks to spaces. I wondered if this could be done e.g.

>> unspaced ["some" "stuff" _ "other" "stuff"]
== "somestuff otherstuff"

But this ran up against a desire to be able to say:

>> var: try third ["stuff" "morestuff"]
== _  ; e.g. no third item, TRY blankified null

>> unspaced ["a" var "b"]  ; look ma, no `:var`
== "ab"  ; and no error...but blank var gave null

Plain WORD! var couldn't serve both roles; it would be either a synonym for null that didn't cause an error on regular references, or a synonym for space.

I've also been noticing that I think the version of REDUCE that a GET-BLOCK! runs needs to be able to vaporize nulls and not error, otherwise you couldn't write:

append b :[if true [<append>] if false [<skip>]]

Not being able to handle nulls to drop data entirely would be a shame for this very cool solution to REPEND.

[hostilefork]

Zany, I know. But what this means is that while blanks can come back from a function or evaluation, you wouldn't get them out of an ordinary variable unless you were kind of trying.

This is seeming extremely promising, for how it shifts the responsibility away from constructs like APPEND or COMPOSE and onto the caller to say what they mean. Because you're not getting blanks unless they are really intended. I have a cautious optimism of how this looks.

BUT it messes with the idea that BLANK! is the equivalent in Redbol to NONE!. In Rebol2 you can say:

rebol2> x: none
rebol2> y: x
rebol2> none? y
== #[true]

Yet under these rules, that becomes x: blank, so then y: x will act as y: null. Meaning none? y will fail on an unset variable.

While we can change many things in Redbol emulation, the concept is that it still uses the same evaluator. So the only way to get around this and still use blank for none would be if assigning nulls to variables set them to blank instead of unsetting them. Then, routines that are under the emulation's control would have to be smart about handling both null and blank... (e.g. none? would accept either null or blank and return true).

The (uncomfortable) consequence would mean that x: null would not be a synonym for unset 'x:

 >> x: null
 ; null (but it *actually* stored a BLANK!)

 >> :x
 == _

 >> null? x  ; won't error, regenerates the null
 == #[true]

 >> unset 'x
 ; null (or should unset's return value be the old value?)

 >> :x
 ; null

 >> null? x
 ** Error: x is unset

One casualty of this is error handling. Suddenly:

 >> x: third ["Hello" "World"]
 ; null (but stored a blank)

 >> append [a b c] x
 == [a b c]  ; hrmph, no error anymore :-(

Some might ask, is there any good reason why the above code should act any differently from if you said append [a b c] third ["Hello" "World"] directly?

Red and R3-Alpha don't think so:

>> x: third ["Hello" "World"]
== none

>> append [a b c] x
== [a b c #[none]]   ; though it renders as [a b c none]

>> append [a b c] third ["Hello" "World"]
== [a b c #[none]]

It may seem a reasonable argument, but what makes me queasy is not just about error handling. The null state really is distinct...and being able to not conflate it with a value that's there is critical. Assignment is foundational, and if assignment itself throws away the null/blank distinction then something powerful was lost. So my beef is actually with this:

>> append [a b c] second ["Hello" _]
== [a b c _]

>> append [a b c] third ["Hello" _]
== [a b c]

>> s: second ["Hello" _]
== _

>> t: third ["Hello" _]
; null

>> append [a b c] :s
== [a b c _]

>> append [a b c] :t
== [a b c _]  ; ...ack!

That's the bad mojo. Even a motivated individual wanting to know the difference...who is using a GET-WORD! to know...can't tell. Then, add onto that the loss of the error handling in the plain WORD! case.

I'm aware that my radical proposal about "blank-to-null-decay" is all about conflating blank and null somewhat. But because it doesn't happen with a GET-WORD!, I'm saying that it becomes the option of the user of the variable to decide if they care, in the context they are using it. The variable is still holding the actual state that it captured at the time of the assignment, you just have the choice to decide if that matters at the point of retrieval. A similar decision is made when you distinguish append from :append; the word looks up to a function, but did you want the function invoked or did you want its ACTION! value? You choose. If you could only run the function but were unable to fetch its value, that would be a problem!

Hence I'm pretty sure I'm not cool with x: null doing anything other than unsetting x...as a distinct state from x: _ or x: second [a _ b].

---

So in the event that BLANK! takes this path of "null decay"...Redbol needs another solution for its NONE! than reusing BLANK!, because it just won't have the same evaluative properties. So what's the workaround? Perhaps a special NONE! legacy type that is falsey, but does not translate to a null when fetched through WORD! or PATH! access? It could reuse the same cell as BLANK!, but just have a flag set to say "don't decay".

Or it could use something that simply didn't exist in Rebol2/etc., that is inert...like @none. That wouldn't conflate with anything Rebol2 uses the way something like #none might. But since it's not intrinsically falsey, anything that did tests would have to be hooked pretty deeply in their sense of logic testing. Some could be easy:

 true?: func [v] [
     either :v = @none [false] [lib/did :v]
 ]

 if: adapt 'lib/if [
     condition: true? :condition
 ]

Others would be more difficult, such as digging into the behavior of CASE. Though maybe not, using predicates, which could have similar overrides on the tests. Redbol's CASE could just be Ren-C CASE specialized with Redbol TRUE? as the predicate. Same for ANY, ALL, etc.

These things are certainly profound tests of the flexibility of Ren-C. And when we think about performance, the idea that TRUE? would be making so many appearances gives the kind of motivation of something where it's worth coding as a native in the Redbol extension, as optimized C against the internal API.

But the interoperability will face limits at some point. How could PARSE be hooked to treat @none as an empty rule that is simply skipped? Some compatibility points like this might just require making a non-decaying blank for NONE! that gets thrown in, and loading the Redbol extension enables that datatype to become visible.

I'd rather do something like that to bridge the gap, than not do a good idea for compatibility's sake!

[hostilefork]

Now having worked with Zany Proposal experimentally for about 24 hours, it clearly shows some coolness...though it has some mind-bending properties. (!)

But whether the idea is viable or not as conceived so far...it just is another way of looking at the answer to the title of this thread as being NO. APPEND and COMPOSE should remain purely mechanical. If they get a null, that means nothing. If they get a value that can legitimately appear in a block, then they should put that value in the block...and not editorialize, via refinements.

Whether it's via an old-school OPT or a creative new evaluative behavior that can bring about more brevity...it's up to the caller to pass the right value. Operations like APPEND and COMPOSE should not editorialize via refinements, /ONLY or otherwise. (And if all atomic additions are forced to use /ONLY as being proposed, it wouldn't even be an option to control it that way.)

---

The biggest concern I have about the idea of blanks decaying to null has nothing to do with TRY. Because if blanks are just reified nulls, then making it easier to reconstitute that null via evaluation seems like a pretty neat feature. I see several places where that neatness is indeed showing off well.

But my point about Redbol compatibility problems also show up in Ren-C scenarios where you are using BLANK! for a generic placeholder in the fashion that NONE! may have been used before. For instance, let's say you have:

data: [
    "Foo" 10 <alpha>
    "Bar" 20 _
    "Baz" 30 <beta>
]

If this is a kind of "CSV-style" tabular data, you might enumerate over it and try to extract the third column, doing some transformation:

map-each/only [x y z] data [
    case [
        z = <alpha> [<gamma>]
        z = <beta> [<delta>]
        default [z]
    ]
]

The author of this code wanted to get [<gamma> _ <delta>]. But under the proposed "zany" rule, they'd get [<gamma> $void$ <delta>]. Because the blank would decay to a null, and conditional branches voidify null results... to reserve null for the "no branch ran" case.

But this may suggest we're really talking about two different datatypes. One is a reified null, that itches to reconstitute null whenever it can. And one is an inert placeholder, in the spirit of classical NONE!

In R3-Alpha, you could use # for NONE!, which had a bit more heft to it than the light-looking BLANK!

data: [
    "Foo" 10 <alpha>
    "Bar" 20 #
    "Baz" 30 <beta>
]

This wasn't a popular choice, but it had some more visibility.

However, maybe with the non-decaying @word, @pa/th, @[bl o ck], and @(gr o up... there could be a place for a non-decaying...LIT-NULL!, @, which would be evaluatively inert, but also falsey?

data: [
    "Foo" 10 <alpha>
    "Bar" 20 @
    "Baz" 30 <beta>
]

Doesn't seem like a terrible idea. But by the arguments given, you probably wouldn't want wacky decaying BLANK! for spacing in your non-implicitly spaced prints. You might want to do processing on those blocks with similar mappings, and casual coders could get burned by the same issues that would affect those doing the CSV-style data.

But who knows...maybe it could just be a buyer-beware thing; if you are the kind of person who takes advantage of the feature you have to be careful.

---

(All of this is really just some experimental tinkering. I'm getting back into the rhythm of dealing with Rebol, so it's good to do something fun...but...what needs to happen is the Travis and other planning. So I'll probably put this down while I let the questions sink in a bit. Again, this was all just solidifying the answer to the topic of this thread, which I feel is settled now.)