A Quantum Leap in Testing: Isolating Into Contexts

Long ago @Brett took on the lame-but-important task of breaking the tests into individual files (from a giant, monolithic %core-tests.r file).

And as part of that task, I asked the favor of converting it from the traditional "tests-as-BLOCK!s":

; Original Rebol Test Format
["abba" = reverse "abba"]
[
    foo: func [x] [
        x + 1000
    ]
    true  ; tests have to return *exactly* LOGIC! true
]
[1020 = foo x]
[304 = foo -696]

...to use GROUP!s...

; Ren-C Adjusted Test Format
("abba" = reverse "abba")
(
    foo: func [x] [
        x + 1000
    ]
    true
)
(1020 = foo x)
(304 = foo -696)

...it Wasn't Just Change for the Sake of Change...

I had a plan.

The idea was that I wanted to save the more "barrier-like" blocks to represent isolated groups. So then in the "someday" that we could run these tests into isolated contexts, we'd be able to group only those tests that needed to interact together.

While we were waiting for isolation technology to improve, the blocks were still used. They were just documentation.

But today that is no longer--by the magic of science :man_scientist: they are isolated!

("abba" = reverse "abba")  ; does not need foo, stands alone

[
    (foo: func [x] [
        x + 1000
    ]
    true)

    (1020 = foo x)  ; uses foo, wants to be in same isolated context
    (304 = foo -696)  ; also sees foo
]

But It's Better Than I Thought!

We can actually isolate every test if we feel like it! We can run the thousands of tests and add nothing to the user context in the process!

Is it slower, you ask? No--IT'S MUCH FASTER! The GC can clean things up it's not using.

It Has Been A Good Test For Sea Of Words Modules! Making thousands of mini-modules was a good exercise and found a couple of bugs. Nothing too big, but certainly good to have that extra stress test.

(We still might try a mode where we don't isolate, just to throw a wrench in the GC and cause the user context to puff up. It's a semi-random sort of stress test.)

If BLOCK! Means "Isolate Test", Most Want The Insulation

["abba" = reverse "abba"]  ; we can isolate this!

[
    (foo: func [x] [  ; we can *not* isolate this...
        x + 1000   ; ...and make that also mean "don't check result!"
    ])  ; ...so no need for the "true" here

    [y: 1020, y = foo x]  ; uses foo, doesn't need to expose Y
    [304 = foo -696]  ; uses foo, doesn't need to see Y
]

We don't quite yet have the underlying mechanisms to get that extra level of inheritance needed to isolate the "cluster" of tests, and then get that nested isolation for the blocks in the cluster. But it's likely not too far away!

Could We do Better With the Notation?

I tried some experimentation with separating out the idea of putting freeform code in the "cluster" of tests, and saying the freeform code is what's not isolated:

[
    foo: func [x] [
        x + 1000
    ]

    [1020 = foo x]
    [304 = foo -696]
]

You could imagine it running not just setup code for definitions, but also shutdown:

[
    port: open %test-data.txt

    [port? port]
    [binary? read port]

    close port
]

But trying my hand at it, I think being this freeform isn't a good idea for this particular dialect. There's potential for confusion since you're sensing so much based on when a block is on a newline or not. Seems easy to make mistakes.

Main thing to focus on here is the advancements in the isolation. I'll keep thinking on the dialect!

2 Likes

Few random brainstorms:

Maybe having setup code segments return true isn't so bad? It helps make sure you know what you're doing, and you haven't accidentally run and vaporized some test code result due to a bug in the test system itself. It's more foolproof.

We could go in another direction...foregoing the GROUP!/BLOCK! requirement and using dashes:

[
    port: (open %test-data.txt)
    true
    ----
    port? port
    ----
    binary? read port
    ----
    close port
    true
]

It does start feeling a bit junky. Red's tests have a lot of dashes and squiggles in them to try and put dividers all over the place, and I kind of prefer the more minimalist direction the historical notation was shooting for.

But dashes start to look more appealing the more complicated your tests get:

; AS aliasing of TEXT! as BINARY! constrains binary modifications to UTF-8
https://github.com/metaeducation/ren-c/issues/817
[
    t: "օʊʀֆօռǟɢɢօռ"
    b: as binary! t
    true
    ----
    insert b "ƈ"
    t = "ƈօʊʀֆօռǟɢɢօռ"
    ----
    append b #{C9A8}
    t = "ƈօʊʀֆօռǟɢɢօռɨ"
    ----
    e: trap [insert b #{E08080}]
    e/id = 'bad-utf8-bin-edit
    ----
    b: as binary! const "test"
    e: trap [append b 1]
    e/id = 'const-value
]

This loses the distinction I wanted to make between code that is supposed to be common, and the reliant code bits that are supposed to be all independent. :-/

But with the delimiters, you have their potential ambiguous use in the non-test portions... and it gets uncomfortable knowing where to put them. I'd had some various bad answers historically, that don't get better with code outside the GROUP! coming on the scene:

[
    t: "օʊʀֆօռǟɢɢօռ"
    b: as binary! t
(
    insert b "ƈ"
    t = "ƈօʊʀֆօռǟɢɢօռ"
)(
    append b #{C9A8}
    t = "ƈօʊʀֆօռǟɢɢօռɨ"
)(
    e: trap [insert b #{E08080}]
    e/id = 'bad-utf8-bin-edit
)(
    b: as binary! const "test"
    e: trap [append b 1]
    e/id = 'const-value
)]

Not putting the groups on a separate line for multiline is an option, probably the best one:

[
    (t: "օʊʀֆօռǟɢɢօռ"
    b: as binary! t)

    [insert b "ƈ"
    t = "ƈօʊʀֆօռǟɢɢօռ"]

    [append b #{C9A8}
    t = "ƈօʊʀֆօռǟɢɢօռɨ"]

    [e: trap [insert b #{E08080}]
    e/id = 'bad-utf8-bin-edit]

    [b: as binary! const "test"
    e: trap [append b 1]
    e/id = 'const-value]
]

(Note: The GROUP! vs. BLOCK! distinction seems pretty subtle here, in a way that's surprising considering that we read code all the time and need to know the difference. It looks more obvious without the group.)

(Note2: Unfortunately, we can't use double-spacing to mean anything...because LOAD doesn't preserve double-spacing.) :frowning:

I'm Just Not Going to go Gung-Ho Converting The Tests Yet...

My format choice is compatible with what was there and will give an opportunity to experiment. Once there's more experience with it I'll make some decisions...

2 Likes

So...this has led me to think we may well need to be able to discern double-spacing.

We can find the bits to encode this, as well as other formatting. The main problem is just what happens as you reorganize code and this formatting travels with it. You get weird-looking stuff. :frowning:

Also, some of the conveniences like append/line (which things like keep/line inherit) get more complex. Although I guess you could say:

>> collect [keep/line [a b] keep/line [] keep/line [c d]]
== [
    a b

    c d
]

Tricky, but doable.

I guess the thing I've just noticed visually is that when it comes to horizontal separators, a double-space says a lot by saying nothing. These tests would be much easier to maintain, I think, if they were just spaced.

If you wanted internal spacing in your test, then you could use a GROUP!.

1 Like