(Archival) Writing a network scheme for Rebol 3

Writing a network scheme for Rebol 3

Graham Chiu
version 1.4, 1 April 2013

This document is a distillation of what I have read and understood regarding networking and URI schemes as defined for Rebol 3. The sources are the Rebol websites along with info obtained in chatting with other Rebol users. Any incorrect information contained herein is entirely due to my own misunderstandings. As an illustration I will talk about creating a basic Scripting Layer For Android (SL4A) scheme from the ground up.

Opening a Port Concepts

When you open a network scheme you are returned a port! object. The port! object contains the functions, objects, and other variables that allow you to track state, and process network events. One of those objects is the underlying sockets-level TCP port which interacts with the TCP device. In Rebol 2, this was generally referred to as the sub-port while Rebol 3 schemes such as HTTP call it conn or a similar name. Make a note as this can be a source of confusion. And if the scheme is anything other than a TCP scheme using a spec block, the underlying TCP port is not opened and will require a second call to open.

Simple Example

To illustrate how the built-in TCP scheme works, we can look at a basic interaction with a SL4A server at network address 192.168.1.103 on port 4321. I will discuss line by line what happens.
basic tcp example with event handler

android-request: {{"params": ["Hello, Android!"], "id":1, "method": "makeToast"}^/}

android-port: make port! tcp://192.168.103:4321
; android-port: make port! [ scheme: 'tcp host: 192.168.1.103 port-id: 4321 ]
android-port/awake: func [event /local tcp-port] [
        tcp-port: event/port
        print ["port equality?" equal? tcp-port android-port]
        print ["==TCP-event:" event/type]
        switch/default event/type [
                lookup [
                        ; this happens when we use a url!, and the IP address has to be resolved
                        ; using DNS, or when using a specification block and the host is provided
                        ; as a name

                        print "opening tcp port"
                        open tcp-port
                ]
                connect [
                        ; these next two lines are equivalent since we are using a basic tcp scheme.
                        ; write tcp-port android-request
                        write android-port android-request
                ]
                wrote [
                        print "have a wrote event, so read response from server"
                        read tcp-port
                ]
                read [
                        ; data has arrived
                        print ["^\read:" length? tcp-port/data]
                        return true
                ]
        ] [
                ; returns if receives other events like done, close, error, custom
                true
        ]
]

open android-port
either port! = type? wait [android-port 3] [
        print to string! android-port/data
] [
        print "Port timed out!"
]
close android-port

First, we define the string android-request to hold the JSON representation of the SL4A function call makeToast, with a parameter of "Hello, Android". The terminating newline is needed as the API appears to be line oriented.

android-port: make port! tcp://192.168.103:4321

Next, we create a new port! based on the URL of tcp://192.168.103:4321. If we now examine android-port, we have this output from probe android-port

make port! [
    spec: make object! [
        title: "TCP Networking"
        scheme: 'tcp
        ref: tcp://192.168.103:4321
        path: none
        host: "192.168.103"
        port-id: 4321
    ]
    scheme: make object! [
        name: 'tcp
        title: "TCP Networking"
        spec: make object! [
            title: none
            scheme: none
            ref: none
            path: none
            host: none
            port-id: 80
        ]
        info: make object! [
            local-ip: none
            local-port: none
            remote-ip: none
            remote-port: none
        ]
        actor: make native! [[port!]]
        awake: make function! [[event][print ['TCP-event event/type] true]]
    ]
    actor: make native! [[port!]]
    awake: make function! [[event][print ['TCP-event event/type] true]]
    state: none
    data: none
    locals: none
]

We can see that the port! object contains a spec object derived from the URL, tcp://192.168.1.103:4321. This was created by the parse-url function which in turn was called by the make-port function. The spec object is filled in with all URI scheme components parsed as per RFC 3986 (e.g. tcp://user:password@hostname:port).

Next we see the scheme object based on the Rebol TCP scheme, followed by actor which is currently a port! (it can also be a block holding actors such as read, write, etc.), the awake function, state, data and locals. The data field will hold any data returned by the tcp port during a read.

The awake function by default just prints the event to the console, and then by return true, exits any wait function that might have passed the event to it. The next lines in our example do something more useful defining an awake function that handles all the different type of events that a port can receive. The events we have not named such as close, error, done will cause our switch statement to return true by default.

We have ordered the events in the switch statement according to the order that we expect them to happen but you might find it more efficient to re-order them in descending order of frequency of use. Now, let’s step through each event to see what happens
Lookup

The lookup event is generated when either the TCP device has to do a DNS lookup to resolve a name, or, if a URL has to be decoded by decode-url. Here we used the url! construction form to create the port!, but with the block form [ scheme: 'tcp host: 192.168.1.103 port-id: 4321 ] the lookup event would not be needed and would not occur as a tuple! is provided for the IP address.
Connect

Here, our TCP device has connected with the server, and a connect event is returned. When handling protocols like POP3, and SMTP, we would want to read the welcome string from the server here, but with other protocols like HTTP and SL4A there wouldn’t be a welcome string so the request is sent immediately.

Wrote

Having written our request to the tcp-port aka android-port, we now get a wrote event. Under normal conditions the server would respond to our request, so we issue an immediate read tcp-port.

Read

The TCP device has received some data which is made available in the android-port/data field, and a read event generated. We print the data and return true since we don’t need to deal with any other events. This allows us to exit the wait function.
Rest of Code

Now, let’s look at the remaining code.

open android-port

passes the port! to the open function. The open function in this instance does not actually open the TCP port, but issues a lookup event which is stored in a system block which holds all events.

If we had done the following instead:

android-port: open tcp://192.168.1.103:4321

then open would have called make-port to first create the port! structure, and then issued the lookup event as above.

But if we had done this instead

android-port: open [ scheme: 'tcp host: 192.168.1.103 port-id: 4321 ]

then the same thing happens as above except no lookup event is generated. In this last instance the underlying TCP port is actually opened. The difference is due to the fact that when using the url! form the host address is converted to a string! i.e. "192.168.1.103" instead of a tuple! by the decode-url function and so still needs to be resolved using a DNS lookup.

either port! = type? wait [android-port 3]

The wait function adds the android-port to the list of system ports, and then does a wake-up android-port lookup, i.e. it sends the lookup event to the android-port’s awake handler. In this case our awake handler on receipt of the lookup event now opens the tcp-port with open tcp-port. Each subsequent event is passed to the port! where they are handled as follows:

connect - We write to the port

wrote - Next, we read the port

read - Finally, we print the data and return true

On receipt of the true value, the wait function removes the port! from the system list of ports, and then returns from the wait returning a port! A timeout is specified so that if the port! fails to return within 3 seconds the wait will exit. If we get a port! returned from the wait, we know we completed without timing out and have data, so we can print it out.

Custom Schemes

It would be tedious to write out the above code each time we wanted to write something to a port when we could do something like this:

result: write sl4a://192.168.1.103 {{"params": ["Hello, Android!"], "id":1, "method": "makeToast"}^/}

The sl4a scheme would handle the opening of the TCP port, set the default port-id and async handler, take our data, write it to the TCP port on connect, and return the data. Since we do not have any reference to the port!, it would then be closed for us by the garbage collector.

In the ideal code above, we have a write function that takes two arguments, a url!, and a string! and returns a result. How do we do this when write is a built-in, native function which knows nothing about our custom sl4a scheme? This is where the magic of the port! actors comes into play. When write, or any of the built-in functions that take a port! as a parameter are called, the port! actor block is examined to see if there is a function of the same name present. If so, it is executed in preference to the built-in function, passing the port! as the first argument, followed by any additional arguments if present.

You will remember from before that we had the following when we inspected the port! object created for the built-in TCP scheme:

actor: make native! [[port!]]

In a typical custom scheme, the signatures of the actor functions would instead look something like this:

actor: [
        write: func [ port [port!] data ][ ... write actions ...]
        read: func [ port [port!]][ ... read actions ...]
        open: [ port [port!]][ ... open actions ... ]
        open?: [ port [port!]][ .. returns a truthy value if open ... ]
        close: [ port [port!]][ .. closes the port ...]
]

and the custom scheme write actor is invoked instead of lib/write. So, the first thing we need to do is write our port! actors. We can have here create, delete, open, close, read, write, open?, query, update, and rename and all these functions take a port! as an argument. We can also add series functions such as length?, append, insert to the actor block, but these function can only take a port! object as the argument, and can not take a url! or spec object.

So, instead of

insert sl4a://192.168.1.103 "some data"

we need to write it something like this:

android: open sl4a://192.168.1.103
insert android "some data"

The minimum actors we need for our scheme example are write, open, open? and close, so let’s start!

Open?

This just needs to return the port/state. When the port! is first created, it should be none, as seen in the probe output above. Once the port! is opened, the port/state is set to an empty string by open. So, when we close the port!, we need to set this back to none.

open?: func [ sl4a-port [port!]][
    sl4a-port/state
]

Close

We need to close the tcp-port, and also set the sl4a-port/state to none. The tcp-port is going to be held in the sl4a-port/state/tcp-port.

close: func [ sl4a-port [port!]][
    close sl4a-port/state/tcp-port
    sl4a-port/state: none
]

Write

Our write actor is going to first check if the sl4a-port is open. If it is not open, it will open the sl4a-port using the sl4a-port open actor (since it is opening a sl4a-port). Once the sl4a-port is opened, we will write the data to the sl4a-port. You will recall from the TCP scheme example that we actually write to the tcp-port in the connect event. We will use function named sync-write that assigns a custom async handler to do this for us. We also need to return some data. Again, recalling the basic TCP scheme, data is returned to the tcp-port/data, or sl4a-port/state/tcp-port/data in the current code. The problem is that this might not be safe if we have more than one read since interleaving reads and writes clears this field. We could store it in sl4a-port/data, but we may not necessarily have a reference to that from the tcp-port, so we will create an additional tcp-port spec field, JSON, to hold this data. We are naming it JSON since we know the SL4A api returns JSON strings.

write: func [sl4a-port [port!] data] [
        if not open? sl4a-port [
                open sl4a-port
        ]
        sync-write sl4a-port data
        ; and this is the data placed here by the sync-write function which we now return
        sl4a-port/state/tcp-port/spec/JSON
]

Open

Our open actor has the task of setting up the sl4a-port/state object which is going to hold our tcp-port, our JSON data field which is part of our port spec(ification), and perhaps other fields we might need later on as we expand our scheme functionality. If the sl4a-port is already open, it is simply returned. If a host address is not specified, we create an error. We then create our tcp-port here using make port! as with the TCP scheme, using a spec block, and make port! creates the scheme and spec objects.

Before opening the tcp-port, we set the awake handler to none. Normally we would set it to the scheme’s awake handler, but we haven’t defined one yet. It won’t matter yet because even though we then open the tcp-port, there wouldn’t be any events to process until we call wait on the tcp-port. Note that since we are now using open on a tcp-port rather than a sl4a port, Rebol will use the open function appropriate for a TCP port. Otherwise we would have a stack overflow.

open: func [
    sl4a-port [port!]
    /local tcp-port
] [
    if sl4a-port/state [return sl4a-port]
    if none? sl4a-port/spec/host [make-sl4a-error "Missing host address"]
    sl4a-port/state: context [
            tcp-port: none
    ]
    sl4a-port/state/tcp-port: tcp-port: make port! [
            scheme: 'tcp
            host: sl4a-port/spec/host
            port-id: sl4a-port/spec/port-id
            timeout: sl4a-port/spec/timeout
            ref: rejoin [tcp:// host ":" port-id]
            port-state: 'init
            json: none
    ]
    comment {
            port/state/tcp-port now looks like this
            [ spec [object!] scheme [object!] actor awake state data locals ]
    }
    tcp-port/awake: none
    open tcp-port
    sl4a-port
]

Sync-write

You will recall we called function sync-write inside the write function but have not defined it. sync-write takes the sl4a-port and a JSON string as arguments. It needs to set up the awake handler for the tcp-port so that a connect event sends the JSON string, then gets data back and stores it in the sl4a-port/state/tcp-port/spec/JSON field. It then kicks off event handling by doing a wait on the tcp-port. If a lookup event occurs, we know we need to open the tcp-port.

If we want to do another write on the port, we can no longer do it on the connect event as that only occurs when we first open the tcp-port. Therefore, we need to check if the tcp-port has been connected. If so, we write to it and then wait to restart the event handler. To track the port state, a flag, port-state, is created in the spec object.

sync-write: func [sl4a-port [port!] JSON-string
    /local tcp-port
] [
    unless open? sl4a-port [
            open sl4a-port
    ]
    tcp-port: sl4a-port/state/tcp-port
    tcp-port/awake: :awake-handler
    either tcp-port/spec/port-state = 'ready [
            write tcp-port to binary! JSON-string
    ][
            tcp-port/locals: copy JSON-string
    ]
    unless port? wait [tcp-port sl4a-port/spec/timeout] [
            make-sl4a-error "SL4A timeout on tcp-port"
    ]
]

Awake Handler

We used an awake-handler in the sync-write function, and define it now. In the connect state, we set the port-state flag to 'ready; in other states we can set it as as appropriate. It is very similar to our prototype awake handler that we discussed at the beginning, but we are also going to use a structured exit at the bottom instead of using returning true, to make things clearer. The value left at the end is the value of the switch statement.

awake-handler: func [event /local tcp-port] [
    print ["=== Client event:" event/type]
    tcp-port: event/port
    switch/default event/type [
            error [
                    print "error event received"
                    tcp-port/spec/port-state: 'error
                    true
            ]
            lookup [
                    open tcp-port
                    false
            ]
            connect [
                    print "connected "
                    write tcp-port tcp-port/locals
                    tcp-port/spec/port-state: 'ready
                    false
            ]
            read [
                    print ["^\read:" length? tcp-port/data]
                    tcp-port/spec/JSON: copy to string! tcp-port/data
                    clear tcp-port/data
                    true
            ]
            wrote [
                    print "written, so read port"
                    read tcp-port
                    false
            ]
            close [
                    print "closed on us!"
                    tcp-port/spec/port-state: 'ready
                    true
            ]
    ] [true]
]

comment {
        the awake handler should return false unless we want to exit the wait
        which we do either as the default condition ( ie. unspecified event ),
        or with Error, Read and Close.
}

Wrapping Up

So we are nearly ready to wrap this up into a scheme. We have to set up the scheme header and define any missing functions that we have used so far without having defined them.
SL4A scheme

Rebol [
    System: "REBOL [R3] Language Interpreter and Run-time Environment"
    title: "R3 SL4A"
    file: %prot-demo.r
    author: ["Graham"]
    name: 'sl4a
    type: 'module
    version: 0.0.1
    Date: [26-Mar-2013]
    Purpose: "R3 send and receive from Scripting Layer 4 Android"
]

make-sl4a-error: func [
    message
] [
    ; the 'do arms the error!
    do make error! [
            type: 'Access
            id: 'Protocol
            arg1: message
    ]
]

awake-handler: func [event /local tcp-port] [
    print ["=== Client event:" event/type]
    tcp-port: event/port
    switch/default event/type [
            error [
                    print "error event received"
                    tcp-port/spec/port-state: 'error
                    true
            ]
            lookup [
                    open tcp-port
                    false
            ]
            connect [
                    print "connected "
                    write tcp-port tcp-port/locals
                    tcp-port/spec/port-state: 'ready
                    false
            ]
            read [
                    print ["^\read:" length? tcp-port/data]
                    tcp-port/spec/JSON: copy to string! tcp-port/data
                    clear tcp-port/data
                    true
            ]
            wrote [
                    print "written, so read port"
                    read tcp-port
                    false
            ]
            close [
                    print "closed on us!"
                    tcp-port/spec/port-state: 'ready
                    true
            ]
    ] [true]
]

sync-write: func [sl4a-port [port!] JSON-string
    /local tcp-port
] [
    unless open? sl4a-port [
            open sl4a-port
    ]
    tcp-port: sl4a-port/state/tcp-port
    tcp-port/awake: :awake-handler
    either tcp-port/spec/port-state = 'ready [
            write tcp-port to binary! JSON-string
    ] [
            tcp-port/locals: copy JSON-string
    ]
    unless port? wait [tcp-port sl4a-port/spec/timeout] [
            make-sl4a-error "SL4A timeout on tcp-port"
    ]
]

sys/make-scheme [
    name: 'sl4a
    title: "SL4A Protocol"
    spec: make system/standard/port-spec-net [port-id: 4321 timeout: 5]

    actor: [
            open: func [
                    sl4a-port [port!]
                    /local tcp-port
            ] [
                    if sl4a-port/state [return sl4a-port]
                    if none? sl4a-port/spec/host [make-sl4a-error "Missing host address"]
                    sl4a-port/state: context [
                            tcp-port: none
                    ]
                    sl4a-port/state/tcp-port: tcp-port: make port! [
                            scheme: 'tcp
                            host: sl4a-port/spec/host
                            port-id: sl4a-port/spec/port-id
                            timeout: sl4a-port/spec/timeout
                            ref: rejoin [tcp:// host ":" port-id]
                            port-state: 'init
                            json: none
                    ]
                    tcp-port/awake: none
                    open tcp-port
                    sl4a-port
            ]
            open?: func [sl4a-port [port!]] [
                    sl4a-port/state
            ]
            write: func [sl4a-port [port!] data] [
                    if not open? sl4a-port [
                            open sl4a-port
                    ]
                    sync-write sl4a-port data
                    sl4a-port/state/tcp-port/spec/JSON
            ]
            close: func [sl4a-port [port!]] [
                    close sl4a-port/state/tcp-port
                    sl4a-port/state: none
            ]
    ]
]

Testing

We can test that it works with the following code, using no port! reference:

android-request: {{"params": ["Hello, Android!"], "id":1, "method": "makeToast"}^/}
result: write sl4a://192.168.1.103 android-request

The output:

console trace

>> result: write sl4a://192.168.1.103 android-request
=== Client event: lookup
=== Client event: connect
connected
=== Client event: wrote
written, so read port
=== Client event: read
?read: 36
== {{"error":null,"id":1,"result":null}

If we want to hold on to the port! we can run this code:

>> android: open sl4a://192.168.1.103
>> result: write android android-request
=== Client event: lookup
=== Client event: connect
connected
=== Client event: wrote
written, so read port
=== Client event: read
?read: 36
== {{"error":null,"id":1,"result":null}
}

>> result: write android android-request
=== Client event: wrote
written, so read port
=== Client event: read
?read: 36
== {{"error":null,"id":1,"result":null}
}

This should provide you with enough information to write your own Rebol 3 network schemes.

References