Comments Within Binary Values

Development Naming
#1

Should this be permitted? Here is an annotated PNG:

#{
89504E47  ; PNG Signature
0D0A1A0A  ; Integrity Check

0000000D  ; Chunk Length
49484452  ; Chunk Name (IHDR)
    00000001  ; Width
    00000003  ; Height
    08  ; Colour Depth
    06  ; Colour Type
    00  ; Compression Method
    00  ; Filter Method
    00  ; Interlace Method
52DD6582  ; Chunk CRC

0000000F  ; Chunk Length
49444154  ; Chunk Name (IDAT)
789C63608083F894A8FF00046A021D  ; Image Data (GZIP compressed)
54C10693  ; Chunk CRC

00000000  ; Chunk Length
49454E44  ; Chunk Name (IEND)
AE426082  ; Chunk CRC
}

Neither Rebol 2 nor 3 supported this, however Red does. A couple of notes—the above example would be the case for, against is the already-noted vapourizing comments and the complexity of potential codec that supported comment-preservation

1 Like
#2

It's not the worst idea. Though I'm not sure how realistically useful it is. How often are you documenting a giant chunk of literal binary data...where you're not composing generated values into it?

I'm pretty set on saying that BINARY! is done with &{...} and &"...". We've wondered if that would imply that &decafbad would necessarily need to be the "no delimiter form" or if & could be taken for HTML entities instead.

But perhaps this motivates why non-delimited binaries could be more useful than they seem, by providing these break opportunities for commenting:

join-all [
   &89504E47  ; PNG Signature
   &0D0A1A0A  ; Integrity Check

   (enbin [BE + 4] chunklength)
   &49484452  ; Chunk Name (IHDR)
   ...
]

But I dunno. The delimiters may look better.

join-all [
   &{89504E47}  ; PNG Signature
   &{0D0A1A0A}  ; Integrity Check

   (enbin [BE + 4] chunklength)
   &{49484452}  ; Chunk Name (IHDR)
   ...
]
#3

I see a clear need for this (in fact I see a need for this for other things besides binaries). There will be times when interpretation is unavailable yet complex data needs to be loaded. To prohibit commenting in such a scenario would be more than unfortunate.

1 Like
#4

Worrying a lot about situations when "interpretation is unavailable" is a rather expansive domain in which you can't really overthink too much if you are writing an interpreter.

The rare cases where it might happen sounds like a place to use PARSE to accrue your BINARY! from a format that could be more clever than any single concern we might think to address.

1 Like
#5

So you are saying define a commented-format, write a stand-alone program to translate that commented-format into an uncommented-format, then load the uncommented-format? Three steps and a separate executable utility that needs its own source code and maintenance? I understand wanting to off-load as much work onto the user as possible, but suggesting that commenting is not a worthy concern different from all other ways to dynamically build unreadable things in order to justify requiring this is dodging at best.

#6

I'm saying that despite the claim that this comes up a lot, it simply doesn't. In the rare case "something like it" does come up, I imagine the problem is actually more nuanced and would require a better solution than one we can guess at. All the cases I can imagine caring about would need to be structured for escaped slots, at which point you've got your comment opportunities.

The example given was artificial. You don't have PNG bytes being written up where the fields have been filled in like that in anything but documentation of an example format. Such binaries put to disk would actually be stored as bytes and not text...there is not a market for "annotated hex-encoded bytestreams" that we need to be serving in any more condensed way than &{DECAFBAD} ; comment. (If it were a huge market somehow, then we should adopt whatever that huge market has standardized upon.)

When people embed bytes into a script for convenience to transmit data in band and with the script, they typically want to do it compressed (comments largely pointless on individual compressed bytes)...and more often as base64 vs. hex anyway.

There's a lot of things that need more attention and upkeep.

1 Like
#7

I don't know of any binary format that embeds comments like that (granted, that doesn't say much), so I don't see any win in allowing it.

#8

This was an example Oldes posted in the Red group.