Android's Back! Modernized, and With Continuous Integration

Development Build System
1

Since I was on a roll with the TCC Bootstrap and the Windows Builds on GitHub Actions, I decided to take a crack at bringing the Android builds back.

It would be underwhelming if I didn't bring things somewhat up-to-date in the process. Android NDK versions are numbered by [r]evisions, and we had been on "r13" from October 2016. Five years ago is basically eternity in cell phone time...and the revisions have counted up steadily since then, currently at "r21" (!)

So I dug in and suffered for another long continuous day-and-night to make it happen.

And They're Back!!!

Big Change: Toolchain Switch to Clang

The Android toolset was originally based on GCC. But they wound up switching to Clang...starting by making it available in revision r16, and then removing the GCC support altogether in r18.

This may seem like a minor change for us (and it probably would have been easier if I knew more about the NDK to start with). But cross compilation is never "simple", and Android is a particularly crazy combinatoric monster. When the .zipped file for installing the SDK is 500MB--with executables for only one of the cross-compiling platforms--you can bet there's a lot in there to trip up.

I'll spare you the details, but you can read config/android-common.r if you want to see comments I wrote to document the process.

Still Supporting The Older NDKs!

The Google Play store will only accept apps that are built with very recent API Levels!

But I think it's interesting to be in the position that we can still build against older NDKs, perhaps for older hardware. It's a distinguishing feature.

Anyway, I made the script detect the version and react accordingly. Maybe there will be a time when it's not worth it to support the older NDKs, but for now it doesn't cost much...and it helps keep a check on making sure that if we use any "newfangled" APIs that only exist on certain modern devices, we do so consciously.

Next Steps... CI Testing!

So I said that I wanted anything we thought was important to be rigged up with automated testing. I did some research into Android emulators that could run "headlessly" with no screen...let you install apps on a virtual phone, and run tests. Such things do exist, and we could run them on GitHub Actions!

They say it only runs on Mac hosts. So that incentivizes hammering on the cross-compile script until it's able to use platforms other than Linux to build for Android. It may not actually be all that hard to do (will have to find out...) Of course we could build on Linux and then just send that product to a mac to run the emulator.

Getting that ability to run a simulator that can test the built Android executable is a priority.

Revive Building the APK

So we definitely want to have the process from @giuliolunati's conference demo be automated and tested. That means building the .APK file which bundles together both the native interpreter (as a backdoor to phone services) along with a WASM interpreter (to live in a webpage and be able to communicate with the native interpreter).

Maybe Giulio can speak to how far we are from having that work again... or if he's busy, I guess I can just go through the video... that's what it's for! :slight_smile:

Bring Back TCC Extension

I resurrected the TCC bootstrapping because I think it is an impressive demo, and it speaks to an important spiritual point about the project.

It's pretty easy to get the ball rolling for TCC on Linux. But a bit more work when you're cross-compiling it, or making it for Windows.

I never tried it, but it seemed like we had TCC working on Android. It would probably be worth doing it again, and interesting if we could see the TCC bootstrap demo work on a phone too (!) We'll have to consult the old Travis build script to see how the ARM configuration was done.

Revisiting This Gives Me More Motivation

Slogging through the nightmare of the Android NDK reminds me that we are really doing something that almost no one does anymore. They just sign away their lives to something like Android Studio and are so far away from the actual mechanical levels that they don't know what the dependencies are. Most people probably wouldn't know how to do any different...and among those who know, it's simply too frustrating to work through the details.

When you add up all the platforms and parts that we have, this is probably a more compelling story than I give it credit for. So we should definitely not take for granted what we've got...and get the great demos like the AndroidNative+WASM packaging back up and running!

:rocket:

2 Likes
2

...and I Have Done It!

We can now cross-compile an ARM executable and put it on a simulated Android device on the web!

Here you can see an R3 built for ARM reading HTTPS from example.com and sending it back to the OS X host machine!

It's probably a good thing I didn't try to do this any sooner than I did. It doesn't seem it would have been very feasible until the most recent release of Android-11 (API Level 30).

What changed in Android-11? Well, instead of emulating an ARM processor on an x86 host, there's a more economical emulation going on. The Android device being emulated is actually the x86 version of Android, which is then smart enough to translate the ARM binary and run it in a fairly efficient way. Doing it this way means that the hardware being emulated is x86-on-x86, which can be optimized by a special layer (Intel HAXM)

I might make these things look easy, but... there was a whole lot of tinkering and learning to know how to get to this point. And as usual, I've documented the findings in the script itself. If you want some "light reading" on the topic of what's going on in Android world, you can read through the comments:

%android-build.yml

3 Likes
3

You never cease to amaze.

2 Likes
4

There turned out to be a lot more work involved in this than I predicted...but...

TCC Natives On Android Are Working! :confetti_ball:

What you see there is a simulated phone in the cloud, running Ren-C with TCC. The phone is "fresh" with nothing installed on it (no development tools or GCC toolchain). And it's able to execute this code:

call-librebol: make-native [
    "Show It's Possible to Call libRebol without GNU toolchain installed"
    n [integer!]
]{
    int n = rebUnboxInteger(rebArgR("n"));
    rebElide("print [{Hello, libRebol World:}", rebI(n), "]");
    return rebInteger(n + 20);
}
compilables: [call-librebol]

opts: [runtime_path %tcc/]  ; tcc-encap.zip is unzipped to tcc/

compile/settings/nostdlib compilables opts
print "COMPILE SUCCEEDED"

result: call-librebol 1000
print [{The result was:} result]

That's pretty badass.

First Time Running With No Toolchain

Whenever I've run TCC demos in the past it has been when the system had directories like /usr/lib and /usr/include on them. It's typical for Linux to have this stuff. Without it, you don't have stdio.h and the like, so there's no definitions for things like printf() and so the C compiler is fairly useless.

We could get into the business of encapping headers and libs for various platforms if we wanted to. I personally do not. If you want to develop for your platform--and the platform isn't like a Linux that has the headers and libs come with it--download the SDK on your own.

However, I realized here for the purposes of testing that we could actually write C programs that weren't completely meaningless by using the libRebol API. Instead of:

int value = 10;
printf("The value is %d\n", value);

You would call into the interpreter:

int value = 10;
rebElide("print [{The value is}", rebI(value), "]");

That's not so bad. It compares even more favorably with the ECHO proposal, where we open our minds to the idea of using WORD!s as plain visible data:

int value = 10;
rebElide("echo [The value is", rebI(value), "]");

Notice that if an API value is a string then ECHO doesn't know it "came from a variable". (I've tried to stress that all API calls are effectively like COMPOSE...the variables/expressions are gone.) This makes it seem like echo [A string "in quotes"] should lose the quotes? But then people who want to literally put quotes in echo would have to do something else.

Something to continue thinking about.

All Part of Anything-We-Care-About-Needs-Tests

Next on the hitlist for Android is seeing Giulio's conference demo running...

1 Like
5

This Is Now Accomplished!

https://github.com/metaeducation/ren-c/runs/2269540487?check_suite_focus=true#step:7:421

  • keystrokes are poked into the ReplPad running from the APK, asking the Wasm version to READ a special URL.
  • that URL is served by the httpd.reb server run by the native Android version of r3 in the APK.
  • When the URL is served, it writes a file into the phone emulator's filesystem.
  • If the file can be successfully pulled off the emulator and onto the host filesystem with the ADB utility ...it means the whole thing worked!

It took quite a bit to get everything together and working. If you look at the script, the emulator is slow enough that if you don't break the simulated typing up into small bits and delay each time you send it, then characters get lost :frowning: Since the emulator thinks it's a brand new phone, it does "helpful" things like pop up hint dialogs for telling you that you are using things for the first time...and those have to be bypassed. Debugging without a screen...inside an emulator...which has a server hidden in it...through a browser... aaaargh.

But there are tools to help--like being able to snap a screenshot of the phone and download it--which can provide guidance.

Anyway, I patched up the file browser and "app store" too... so the things we saw in the conference demo are working in a current interpreter!

4 Likes