Should append null fail, append BLANK! add nothing unless /ONLY?

[hostilefork]

The Rebol design concept that used to be discussed under the label "none propagation" has become a system of what I think are pretty good rules:

• Blank input gives Null output...for non-mutating operations
• Mutating operations do not accept BLANK! targets as input
• Null input gives an error for operations that aren't specifically about null handling
• TRY can be used to "defuse" nulls into blanks

The reason this has merit is because it gives you a "hot potato" in the form of NULL, without going so far as to error in the moment. You get pinpointed error locality where a problem might be, and have an easy mitigation--if you want it--to use TRY to turn the null into a BLANK! and pass through all other results. And there are a host of null-driven operations to let you pick up the slack, e.g. first block else [code].

(I've suggested that if you ever actually wind up having to put a null state in a variable, test for null later with NULL? and act on it, that you probably could have done it better...reacting to the null without ever using it as a stored state. But it's your choice, I guess.)

So this is being applied across the board. first _ is null...just like first [] is.

Consequences for APPEND/INSERT/CHANGE...

These rules affect a current feature of things like APPEND that I doubt many people have tried:

 >> append copy [a b c] null
 == [a b c]

>> append copy [a b c] if false ['d]
== [a b c]

When it was introduced, I thought it made sense. Since NULL is not a value you can put in a block, you don't get confused about what that means. Then since BLANK! is a legitimate value that can appear in a block, you could append it:

 >> append copy [a b c] _
 == [a b c _]

But quietly accepting NULL loses the hot-potato benefit. If you say append foo third block and there is no third element in the block, doesn't that seem like a good time to complain? Rebol code can pretty quickly get hard to read and debug, and wouldn't it be nice if the person who wrote that line could articulate that there may not be a third thing in the block by saying append foo try third block?

This suggests making the default behavior of BLANK! to be to add nothing, and you need an /ONLY to override that. It may sound like a missed opportunity for null to exploit its out-of-band status. But realistically speaking, I don't think purposefully adding blanks is all that common--certainly a lot less common than adding a block as-is, and that requires /ONLY!

No one probably cared (...yet...)

Like I say--most people probably were unaware of this, and so I don't know there's a lot of append block if condition [...] out there today. We might need APPEND/OPT to be null tolerant, though the only reason to use it would be with /ONLY since otherwise you could say append block try if condition [...].

[hostilefork]

So the answer, after delving and thinking on this issue, has come up as NO.

NULLs will continue to append or compose without error.

>> data: copy [a b c]
>> append data null
== [a b c]

>> compose [a (if false [<omitted>]) b]
== [a b]

But if you have a BLANK! in your hand, that is a "normal" value that will be COMPOSE'd in, or APPEND'd, etc.:

>> data: copy [a b c]
>> append data _
== [a b c _]

>> compose [a (_) b]
== [a _ b]

It's your job to OPT something that might be blank, if you want blanks to be turned into NULLs and evaporate.

Sound a bit awkward, given how many routines return BLANK!s when they fail? Well, no worries...

NULLs are taking over a lot of former BLANK! roles

A failed ANY or ALL, a FIND that didn't find anything, a MATCH that didn't take... these will all be returning NULL, and not blank. This is the new return value protocol, where BLANK! is no longer used as a return value meaning "no match". And it will be tolerable because NULLs will be conditionally false.

This gives BLANK!s a renewed purpose, but a purpose for which they should be thought of as "things".

It's a good feeling to pin down this previously oft-speculated-upon question, since it really was inelegant to have to say compose [a (opt any [...]) b], etc. But we need wonder no longer...you will be able to omit the OPT because ANY will now return NULL!

[hostilefork]

NULLs will continue to append or compose without error.

This now has a fairly strong philosophical defense in NULL, first-class values, and safety.

The way I've started to feel comfortable with this is saying that NULL is what you use when you're talking about a "what". You can use a null to opt-out of that. But you can't use a null to opt-out of a "where". Only BLANK! can do that.

But if you have a BLANK! in your hand, that is a “normal” value that will be COMPOSE’d in, or APPEND’d, etc.:

This is a really tough one to think about.

As the rules have shaped up, I often feel this should probably change to requiring an APPEND/ONLY or COMPOSE/ONLY, otherwise blanks should act like nulls. The premise is this:

It's likely better to embrace blank's "nothingness-without-triggering-errors-on-access" as its primary facet, with its ability to be represented in a block as secondary.

It's hard to see a clear answer purely from a nuts-and-bolts point of view. But it just seems like it's more natural. Consider:

  foo: try case [...]
  ...
  compose [... (opt foo) ...]

There's arguably a nice aspect to the annotation that it's optional, and you might say the random blanks popping up to tell you that you left off an OPT is a feature and not a bug.

But in practice I'm not feeling that. I didn't invoke the TRY because I wanted to create a thing. I put the TRY there because I wanted to say that my tolerance of nothingness was intentional, and I expect this variable to be defined...but conditionally false.

If I were nervous and felt I needed some kind of additional signaling not to use it in an optional sense, I'd have left off the TRY...so foo would have been unset. Why should I have to repeat myself by way of an OPT annotation when I use the variable in this casual circumstance? Doesn't it kind of undermine the convenience of having blank in the first place?

I've made the argument that wanting to append literal blanks is probably less common than wanting to append literal blocks, and that takes an /ONLY. You can append a literal blank as easily as append block [_] if you want to, so it's not like there aren't good options for the cases when you need it.

What's tough about it is that it is a slippery slope. This makes one think "oh, well then should refinement arguments go back to being blank when they're not supplied?" And some of these things that sound good then turn around and bite you because conflating values with nulls loses the rigor of null and compromises the features it enabled. :-/

It's definitely a serious consideration to change this. So please speak up with any observations...

[hostilefork]

"It’s likely better to embrace blank’s “nothingness-without-triggering-errors-on-access” as its primary facet, with its ability to be represented in a block as secondary."

To put myself on the other side of this statement: both aspects are going to come into play, whether you like it or not. It's not going to be all one way or the other. You're going to sometimes be thinking of a blank as a "thing", and sometimes as a "nothing".

Let's look away from APPEND and COMPOSE for a moment, and think about maps. Today these are different:

my-map/key: _ // sets the key to a blank value
my-map/key: null // removes the key

Here you don't have the option of an /ONLY. So if blanks were treated as nothings, you'd lose the ability to put them into maps with this mechanism. But as long as they are distinct, you have the choice to say either:

my-map/key: thing-that-might-be-blank
my-map/key: opt thing-that-might-be-blank

Going back to APPEND and COMPOSE, the /ONLY is already controlling one aspect: splicing or not. But...

• ...what if you want to splice if it's a block, and treat it like a literal blank if it isn't?
• ...what if you want to not splice if it's a block, but treat it as nothing if it's a blank?

When in Doubt, Favor the Mechanically Consistent Behavior

Sometimes things seem annoying when you look at them as individual cases, but you realize their necessity if the big picture is taken into account.

I think needing OPT at the callsite is just one of these necessary evils--that's not even that evil.

For historical perspective: When I initially complained about NONE!s not "vaporizing" in things like COMPOSE, BrianH was very adamant that NONE!s were things. He believed a lot of Rebol's idiomatic behavior involved having empty placeholders in slots of regular record-like structures. So he was generally resistant to using them to opt-out when blocks were being produced.

So I guess this just kind of informs what the system's bias is: Blanks are values first and foremost, and nothings second.

[hostilefork]

I started this thread talking about the two competing opportunities with NULL and APPEND.

• The opportunity to have an "escape" out of a value bearing slot to say "no, I actually don't want to append a thing".

  >> append copy [<a> <b> <c>] (if 1 = 1 '[<d> <e>])
  == [<a> <b> <c> <d> <e>]
  
  >> append copy [<a> <b> <c>] (if 1 = 2 '[<d> <e>])
  == [<a> <b> <c>]
  

• The opportunity to catch an error in intent when NULL is an accident.

  >> data: [<a> <b> <c>]
  >> block: [<d> <e>]
  
  >> append data third block
  ** Error: APPEND doesn't accept NULL as its value argument
  
  >> append data try third block
  == [<a> <b> <c>]
  

As evidenced by the back and forth above, I see both sides.

Null Behavior With The ^ Operator Shifts the Balance

There is an interesting property in the new ^ mechanisms that shows a way we could make blanks disappear and still be taking advantage of NULL's unique status.

Here is the property:

 >> absent: null

 >> ^absent
 == null

 >> try absent
 == _

 >> append [a b c] _
 == [a b c]

  >> append [a b c] absent
 ** Error: APPEND doesn't accept NULL as its value argument

  >> append [a b c] try absent
  == [a b c]

  >> append [a b c] ^absent
** Error: APPEND doesn't accept NULL as its value argument

  >> append [a b c] try ^absent
  == [a b c]

Don't see what's special about that yet? Think about how it would work with BLANK! instead.

  >> present: _

  >> append [a b c] present
  == [a b c]

  >> append [a b c] try present
  == [a b c]

  >> append [a b c] ^present
  == [a b c _]

  >> append [a b c] try ^present
  == [a b c _]

The magic of TRY here is that you can use the TRY on the outside of the literalizing operator and not wind up putting in blanks that get quoted.

So...Is This The Answer?

It's now a strong argument to say that APPEND and friends act with /ONLY semantics on QUOTED!, splice BLOCK!, do nothing on BLANK!, error on NULL.

>> append [a b c] "text"
== [a b c "text"]

>> append [a b c] <tag>
== [a b c <tag>]

>> append [a b c] [d e f]
== [a b c d e f]

>> append [a b c] _
== [a b c]

>> append [a b c] ^ _
== [a b c _]

>> append [a b c] [_]
== [a b c _]

>> append [a b c] null
** Error: APPEND doesn't accept NULL as its value argument

With quoting in reach as easily as ^var instead of var, or ^ some expression instead of some expression... it spins my view toward where we'd be instead wasting blank if we didn't take advantage of its unique status as being a thing you can put in a block and opt out of an operation with.

Then it will error unless you use QUOTE or ^:

>> append [a b c] 'd
** Error: Use QUOTE or ^ to literalize evaluative types with APPEND

>> append/only [a b c] 'd
== [a b c d]

>> append [a b c] ^ 'd
== [a b c d]

>> append [a b c] ^ ''d
== [a b c d]

>> append [a b c] [d]
== [a b c d]

>> append [a b c] ['d]
== [a b c 'd]

>> var: the 'd
>> append [a b c] ^var
== [a b c 'd]

One way of looking at this is that evaluative types are the ones that have the most to lose by conflating their quoted forms with their non-quoted forms on accident.

Another way of looking at it is that inert types exist in other languages as being able to be added to arrays, so making that simple gives parity on the common feature...while the domain of code building costs you an ^. Moreover, I think changing COMPOSE so that it leverages null and uses (( )) for splicing gives a powerful alternative for building up blocks of code.

It's certainly the best proposal so far. My confidence is fairly high it's about as good as this is going to get.

[hostilefork]

@hostilefork

But quietly accepting NULL loses the hot-potato benefit. If you say append foo third block and there is no third element in the block, doesn't that seem like a good time to complain? Rebol code can pretty quickly get hard to read and debug, and wouldn't it be nice if the person who wrote that line could articulate that there may not be a third thing in the block by saying append foo try third block?

Checkmate.

NULL is the answer. There is no "BLANK!-means-nothing" in the system anymore. It's a dialected falsey part you use to mean what you want (I'm going to use it to mean SPACE in several places...)

And when I think about it... APPEND of NULL has no reason to be an error anymore. The operation producing NULL is the culprit, so if you think the situation is worrying, it's the one who needs a TRY.

Let's say we lived in a world where third [d e] was considered a potential danger. It would then need to work like this:

>> append [a b c] null
== [a b c]

>> third [d e]
** Error: Use TRY with THIRD if NULL result is intentional

>> try third [d e]
; null

>> append [a b c] third [d e]
** Error: Use TRY with THIRD if NULL result is intentional

>> append [a b c] try third [d e]
[a b c]

But as it happens, THIRD doesn't think it's a problem to return NULL. "Nothing bad happened." And APPEND can make perfect sense of it, just like it can an empty block.

This would be a different scenario:

>> third null
** Error: Use TRY with THIRD if NULL input is intentional

>> try third null
; null

But as far as APPEND is concerned, NULL is not a problem it needs to be "protected" from. It can still do something semantically coherent (in contrast to THIRD of NULL, which indicates something off is happening.)

NULL Variables Were Never The Problem

The real issue here hasn't been that variables can -hold- a NULL. It's that sometimes the means of becoming NULL was suspicious.

If you have a variable and you want it to be "noisy" when used due to unassignment, set it to none:

>> item: ~

>> append [a b c] item
** Error: This is how you make a variable cause callsite problems

And conditionals which don't produce a value don't make NULL anymore, they make voids...which become isotopes.

>> number: 3

>> tag: switch number [1 [<one>] 2 [<two>]]
== ~  ; isotope

>> append [a b c] tag
** Error: This is where you actually *do* have a problem

[hostilefork]

@hostilefork

APPEND of NULL has no reason to be an error anymore.

Maybe I spoke too soon on this particular point.

But no matter how this shakes out... there is now an answer: that NULL is how you should append nothing, not some convoluted treatment of BLANK!.

So that means one of these is the answer when var: null

>> append [a b c] var  ; (1): no error scenario
== [a b c]

>> try append [a b c] var  ; (2): append takes NULL, tunnels array in error 
== [a b c]

>> try append [a b c] var  ; (3): parameter marked <try>, null result
; null

So...why would I backtrack and suggest that (2) is looking like it might be likely?

My memory has been refreshed by trying to propagate the decisions through to %rebmake.r ... and remembering just how critical the safety measures can be to knowing what the heck is going on once a codebase gets complicated (especially one you did not write). (!)

Above I said "if you have null in a variable, you should make it an isotope, that's the way to get errors on access". But this neglects the desire to have something lying in wait in a falsey status that you can easily check to see if it's set yet with a simple if var, vs. an if set? 'var ... where you're still casually protected from accidents of thinking the not-set-yet variable is ready to be used.

But really, Rebmake is kind of a good compass.

It's convoluted and all needs to be thrown out. And it has to run in the bootstrap executable, which has been twisted beyond its means.

Yet having something so sprawling and strange sets up a situation when something breaks, it shines a light on when a change makes it harder or easier to get one's bearings. So these kinds of decisions are best done taking that into account--as well as how a brand new and clean codebase might be affected.

AND OF COURSE... you'll be able to change it

Just trying to cover all the bases here.

Certainly, Rebmu will want TRY to be implicit everywhere. So there will be a way of doing that, for people who want brevity and care nothing about error locality...

But I imagine that even on a per-module (or per-function!) basis, redefining the language to suit what you are doing will be common...there may be focused moments where you want an append that quietly accepts nulls...or that will append reified ~null~ BAD-WORD!s, or any behavior you need!

[hostilefork]

@hostilefork

But no matter how this shakes out... there is now an answer: that NULL is how you should append nothing, not some convoluted treatment of BLANK!.

All right, how about an even better answer...

What if we used VOID instead, and NULL was an error?

That's the conclusion I came to with COMPOSE and DELIMIT, and I think it has turned out to be the right one. It has eliminated many casual errors there, and provided new features from responding to the NULL errors definitionally.

So why not make APPEND 100% compatible?

 >> var: null

>> append [a b c] var
** Error: APPEND doesn't take NULL (use MAYBE if intentional)

>> append [a b c] maybe null
== [a b c]

We haven't historically thought of VOID as something to pass a function as a parameter. Because in the olden days, you couldn't receive a void: the evaluator would keep going until it actually produced a value (or NULL).

But non-interstitial invisibility is a thing of the past now (too dangerous, too messy). Hence ^META parameters can detect pure voids. And the interesting property of voidness (not being able to be stored in variables) means you have to do a head-check about "did I really mean this?"

We don't want taking ^META parameters to become too pervasive--because it's a hassle. Only special situations are supposed to do it. But APPEND and friends are now one of those special situations, due to the requirement of taking isotopic blocks (to mean "splice"). So they are already weird in this way.

Using MAYBE Feels Better Than TRY

It's not really an error to be appending nothing, that you should have to use TRY to "defuse". This is a different animal from something like first null.

And having nothingness as the parameter isn't just a no-op... e.g. when using CHANGE.

>> data: [a b c d]

>> change/part data void 2
== [c d]

 >> data
 == [c d]

Conditionals return void when they don't match a branch, which makes this convenient:

 >> append [a b c] switch 1 + 1 [10 [<ten>] 20 [<twenty]]
 == [a b c]

It looks like it's all tying together pretty well. Implementing this in the bootstrap shim is left as an exercise for the author.

[hostilefork]

@hostilefork

Using MAYBE Feels Better Than TRY

So there's a bit of a snag on this, when the thing you're making void isn't a parameter...but a product of something like the body on a MAP-EACH. At first it looks fine:

map-each item [1 <one> 2 <two> 3 <three>] [
    maybe match tag! item  ; leaving it as NULL would be an error
]
== [<one> <two> <three>]

But what if you had something else in the loop body?

map-each item [1 <one> 2 <two> 3 <three>] [
    append log spaced ["Logging:" item]
    maybe match tag! item  ; remember, void vanishes
]
== ["Logging: 1" "Logging: <one>" "Logging..." ...]

Now you've got this stray product of an append being the result of the MAP-EACH :-/

I don't know if this is a good enough reason to disallow a MAP-EACH body from evaluating to void and thus adding nothing. The same property applies everywhere, e.g. in COMPOSE slots--you can't just decide to evaluate the whole slot to void in the last expression. It isn't how invisibility works.

But it has some benefit, as it means you move your MAYBE to the outer level vs something tacked on down at the end:

compose [... (maybe any [...]) ...]

We might look at it this way with the MAP-EACH as well; if each iteration of the loop isn't going to be adding something, we say that at the top:

map-each item [1 <one> 2 <two> 3 <three>] [
    maybe (
        append log ["Logging:" item]
        match tag! item
    )
]

Note that there is another tool here, which is CONTINUE:

map-each item [1 <one> 2 <two> 3 <three>] [
    append log spaced ["Logging:" item]
    match tag! item else [continue]
]
== [<one> <two> <three>]

In any case, this just shows that there's a bit of a twist to the use of void, making it something quite different from NULL...so there's some risk of using it incorrectly.