Lisps, Kernel, Clojure: limits of "Code is Data and Data is Code"

I had little-to-no Lisp experience when I encountered Rebol. I just knew the general concept of "code is data, and data is code", and assumed they were similar.

But as it turns out, executing arbitrary structures of code constructed at runtime is taboo in most Lisps. This comes under the umbrella of "EVAL", and warned against for all the reasons that people say it's bad in any language...such as why you wouldn't want to run an arbitrary string of JavaScript code. About the only exception they seem to consider is the Read-Eval-Print-Loop: (loop (print (eval (read)))).

There are warnings about the inefficiency of EVAL not being compiled...or the security concerns of running fully arbitrary code that is cobbled together from possibly unsanitized sources (which Rebol deliberately ignores).

If you're willing to ignore that, you still face the Big Mechanical Problem:

"[EVAL] evaluates under the global environment, losing your local context."

But If So... How Do Branches Work in Lisps?

If you have a branching function like EITHER, and it has two legs of the branch with the intent to only run one of them... doesn't that require selectively running an "EVAL"? The EITHER receives a condition and then two "blocks" (lists) specifying arbitrary code. If you lose all your local variables known at the callsite of the EITHER, how can it work?

The answer depends on which Lisp variation you are using. If you are using a classical Lisp (or Clojure), it simply has a list of exceptions... or "special forms". These constructs are treated weirdly by the compiler and it's just swept under the rug. For example:

Special Forms in Clojure

Another possibility would be if there was some kind of environment capture at the callsite, and the EITHER received this environment as a parameter. Then it could pass that environment to EVAL...so the eval would happen as if it were at the callsite.

Were that written in a Rebol-like syntax, it would be quoting all its arguments and look like:

 either: func [
     'condition [group!]  ; conditions would have to be grouped
     'true_branch [block!]
     'false_branch [block!]
     <environment> env  ; implicit capture of environment at callsite
 ][
     if do/environment condition env [  ; let's say IF is a native
         return do/environment true_branch env
     ]
     return do/environment false_branch env
 ]

Now that you're familiar with the idea, here is that written in how the "Kernel" Lisp Variant does it:

($define! $either
   ($vau (condition true_branch false_branch) env
       ($if (>=? (eval condition env) 0)
           (eval true_branch env)
           (eval false_branch env))))

Notably, Kernel is considered a very experimental black-sheep of the Lisp world, due to how much slower this generalized method of thinking is. Special forms understood by the compiler are the norm.

However, being able to pass environments to EVAL is seemingly endorsed in modern Scheme. But that's not a feature of Clojure.

Macros Cover Some Monkeying-With-Code-Structure Cases

Lisp Macros can be used for source-to-source transformations, manipulating structure in the free-wheeling way we might think of doing in Rebol. But that transformation happens only once.

This Reddit question asks about the difference between macros and receiving arguments unevaluated:

"If we put a backtick in front of the body of a function and we pass arguments quoted when calling it, wouldn't the function work the same as a macro (except that macros are evaluated in an earlier stage)? What would be the difference in practice? And how does this approach compare to fexpr?"

The answers outline the difference... macros just run once, and they don't call EVAL on the code they get... they just return new code which will be evaluated later. If you wrote something like an EITHER as a macro, you would have to transform it into IFs, and be dependent on the special forms to do the actual "weird" mechanic.

Isn't This Kind of Weak for the "Data is Code" Mantra?

It does seem disappointing. :-/

But Lisps haven't taken over the world, and maybe this is part of why.

2 Likes