MySQL extension

Is your web host running a pre-2013 distribution of an OS? :-/

Well, we do like to have things tested on a wide set of platforms...and within reason, old solid ones help with dependency control.

People apparently have used Docker to run CentOS 7 on Travis. So that would be neat to see used for both the build and the tests.

I have changed the post, I think that is just a part that is not provided for by the CentOS maintainers for any (good?) reason. I believe the hosting is taking care of keeping up to date on security updates, ah and yes they are using MariaDB which may not have that problem. I wanted to stick to MySQL but I'll see if there is a different option, creating another VM soon. (NOTE_1)

Perhaps I could make an compiler directive MYSQL_PRE_2013 or alike that will not include some functions.

I had a look at docker and CentOS 7 when I had some trouble with the VM, but it was a lot of information coming towards me and I did not see how to make it work for me. Free time is even today a scarce commodity.

If I understand the article correct, Travis-CI is using Ubuntu to test stuff on, therefor they use Docker to get an instance of CentOS (an EL type OS, RHEL stands for Red Hat Enterprise Linux(?)) running and execute their testscripts on that.
Docker is not so hard, but getting Travis for our repo is perhaps harder.

(NOTE_1) Created a new VM and installed MariaDB and the -devel package. The mysql.h file still was the old "Copyright (c) 2000, 2012, Oracle and/or its affiliates" version. And "mysql -V
mysql Ver 15.1 Distrib 5.5.65-MariaDB, for Linux (x86_64) using readline 5.1", I could not find a mariadb.h, they use mysql.h also(?).
As turns out this version is coming with the OS, but it will not automatically upgrade to a newer version like Distrib 10.2.33-MariaDB without the admin to do this explicitly by uninstalling the old version and ordering yum to go for the 10.2 version.
So I did that. And guess what? The comments still mention the copyright until 2012 like before, terrible! But... the functions missing before are now present! So it is a newer version

Another soap episode. Off course the program from the new VM crashes on my hostproviders server.
Because it cannot find the libmariadb.so.3 library file on its system.
I checked and these are the mysql_config data on the old

Usage: /bin/mysql_config [OPTIONS]
Options:
--cflags [-I/usr/include/mysql]
--include [-I/usr/include/mysql]
--libs [-L/usr/lib64/mysql -lmysqlclient -lpthread -lz -lm -ldl -lssl -lcrypto]
--libs_r [-L/usr/lib64/mysql -lmysqlclient -lpthread -lz -lm -ldl -lssl -lcrypto]
--plugindir [/usr/lib64/mysql/plugin]
--socket [/var/lib/mysql/mysql.sock]
--port [0]
--version [5.5.65]
--libmysqld-libs [-L/usr/lib64/mysql -lmysqld]
--variable=VAR VAR is one of:
pkgincludedir [/usr/include/mysql]
pkglibdir [/usr/lib64/mysql]
plugindir [/usr/lib64/mysql/plugin]

and on the new VM

Usage: /bin/mysql_config [OPTIONS]
Options:
--cflags [-I/usr/include/mysql -I/usr/include/mysql/mysql]
--include [-I/usr/include/mysql -I/usr/include/mysql/mysql]
--libs [-L/usr/lib64/ -lmariadb]
--libs_r [-L/usr/lib64/ -lmariadb]
--plugindir [/usr/lib64/mysql/plugin]
--socket [/var/lib/mysql/mysql.sock]
--port [3306]
--version [10.2.33]
--libmysqld-libs [-L/usr/lib64 -lmysqld -lpthread -lz -lm -ldl -lssl -lcrypto -lcrypt -llzma -laio -lsystemd]
--variable=VAR VAR is one of:
pkgincludedir [/usr/include/mysql]
pkglibdir [/usr/lib64]
plugindir [/usr/lib64/mysql/plugin]

The error I get is

error while
loading shared libraries: libmariadb.so.3: cannot open shared object
file: No such file or directory

As the providers says this library is present but not at /usr/lib64/ but at /usr/lib64/pleskspecificdirectory/ and indeed (I have SSH connection now) I can affirm it is at that location. It is also in /lib64/.../ .

So I changed the link step to not use `mysql_config --libs` but to use this specific location -L/usr/lib64/pleskspecificdirectory/ -lmariadb and I copied the libfile also to that very same location on my new VM.

Linking without such an additive results in errors, using the above or using the `mysql_config --libs` result libraries from the old VM is OK and gets an executable.

Yet both of those still err on the same problem.

Just one thing I noticed.. the r3's that are produced by linking one way or the other are exactly the same size, where I expect a location of different length could make a difference.

Also libmysqlclient.so.18 is present in the /usr/lib64/ directory.
That is thus why the old VM can create the working version.

I did an ldd of the sources I have created and found that the libmariadb.so.3 is sought inside the common /usr/lib64 directory, where it is not at.
I hoped the -L would make it search for the directory provided BUT APPARENTLY not.
Any suggestions are welcome.

-L tells the linker where to search for libraries at compile time...for putting code directly into the executable. These are .a files.

But the .so files are like DLLs. They aren't linked into your program, but expected to be somewhere around on your system. As with DLLs, sometimes the easiest thing to do is just copy it to the same directory with your executable (or better yet, make a file linkage with ln).

You can use ldd to dump out the .so files that an executable looks for

You can read up on rules about where the dynamic loader looks.

So to put the pieces of the puzzle together, you need to make sure that the executable you build is looking for an .so file available on the target. If all else fails, find the .so on your local machine and put it in the same directory as the executable.

Somehow I tricked the entire thing and convinced it to use the libmysqlclient.so.18, last two lines of my linking step

vector-init.o objs/view/mod-view.o objs/tmp-mod-view-init.o -lm -ldl 
-L/usr/lib64/ -lmysqlclient -lpthread -lz -lm -ldl -lssl -lcrypto objs/main.o

but though things work..

First I got

ldd r3
linux-vdso.so.1 =>  (0x00007ffed2df4000)
libm.so.6 => /lib64/libm.so.6 (0x00007ff098f10000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007ff098d0c000)
libmariadb.so.3 => not found
libc.so.6 => /lib64/libc.so.6 (0x00007ff09893e000)
/lib64/ld-linux-x86-64.so.2 (0x00007ff099212000)

Which already was shorter than the 'complete' list I got before, but look at it now:

linux-vdso.so.1 =>  (0x00007ffe2d692000)
libm.so.6 => /lib64/libm.so.6 (0x00007f4149ba2000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f414999e000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f4149782000)
libz.so.1 => /lib64/libz.so.1 (0x00007f414956c000)
libssl.so.10 => /lib64/libssl.so.10 (0x00007f41492fa000)
libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007f4148e97000)
libc.so.6 => /lib64/libc.so.6 (0x00007f4148ac9000)
/lib64/ld-linux-x86-64.so.2 (0x00007f4149ea4000)
libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007f414887c000)
libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f4148593000)
libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f414838f000)
libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007f414815c000)
libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007f4147f4c000)
libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f4147d48000)
libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f4147b2e000)
libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f4147907000)
libpcre.so.1 => /lib64/libpcre.so.1 (0x00007f41476a5000)

No libmariablahblahblah but also no libmysqlclientetc !
So got it working but totally dissatisfied because I do not understand :frowning:

Yes I think I nailed it!

When searching for "how does gcc know which libs to load at runtime" this sent me To the link on Stackexchange that Brian already posted before And from here I found my definitive clue at howto C Libraries But you need to read carefully.

The hint that sent me of in the right direction was

If you create your own shared object files and do not install them in /usr/lib, then you need to set your LD_LIBRARY_PATH environment variable so that the runtime linker can find them and load them at run time. For example, if I put my .so files in a directory named lib in my home directory, I'd set my LD_LIBRARY_PATH enviroment to the following:

if running bash:
export LD_LIBRARY_PATH=/terrible/new/pathname/lib:$LD_LIBRARY_PATH

if running tcsh:
setenv LD_LIBRARY_PATH /terrible/new/pathname/lib:$LD_LIBRARY_PATH

So I went ahead and appended the cruel directoryname used by the Plesk creators. (My best guess is they did this on purpose because they are running MySQL and MariaDB instances on their platforms and do not want them to interfere with each other)

Now the linking returns an executable with the proper library found

ldd r3
linux-vdso.so.1 =>  (0x00007ffc1e72b000)
libm.so.6 => /lib64/libm.so.6 (0x00007fc9ae615000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007fc9ae411000)
libmariadb.so.3 => /usr/lib64/libmariadbclient-plesk-3.0/libmariadb.so.3 (0x00007fc9ae1bb000)
libc.so.6 => /lib64/libc.so.6 (0x00007fc9added000)
/lib64/ld-linux-x86-64.so.2 (0x00007fc9ae917000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fc9adbd1000)
libz.so.1 => /lib64/libz.so.1 (0x00007fc9ad9bb000)
libssl.so.10 => /lib64/libssl.so.10 (0x00007fc9ad749000)
libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007fc9ad2e6000)
libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007fc9ad099000)
libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007fc9acdb0000)
libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007fc9acbac000)
libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007fc9ac979000)
libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007fc9ac769000)
libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007fc9ac565000)
libresolv.so.2 => /lib64/libresolv.so.2 (0x00007fc9ac34b000)
libselinux.so.1 => /lib64/libselinux.so.1 (0x00007fc9ac124000)
libpcre.so.1 => /lib64/libpcre.so.1 (0x00007fc9abec2000)

That looks a whole lot better.

It's all pretty messy stuff. But in this particular day and age, computers aren't psychic. So if you change anything that anything else depends on, everything has to be explicit.

My biggest complaint about these library dependencies isn't that they exist, but that they are shuffled so far into darkness that you cannot reason about them. Once you learn about tools like ldd you can start to do that... and you are. So you're actually learning some things. Good work. :slight_smile:

2 Likes

Thank you Brian!
Strange thing is how it now looks to me. Because I added that directory to a path, it is like any executable starting up is going "hey listen everybody I need to do some work and I have to use a list of functions. First function is function A anybody familiar with a function A? Ah, according to some magic info I just found, there should be a library by the name of libA that should know function A. Are you there libA?"
:smiley:

My VM froze on me just after I had tested the program on the host and before I could save the correct link step used.
I had to reset.
Now I have a working program again, but the ldd dependencies do not mention the libmariadbclient (nor some libmysql)

linux-vdso.so.1 =>  (0x00007ffcb4f1d000)
libm.so.6 => /lib64/libm.so.6 (0x00007f04c07c3000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f04c05bf000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f04c03a3000)
libz.so.1 => /lib64/libz.so.1 (0x00007f04c018d000)
libssl.so.10 => /lib64/libssl.so.10 (0x00007f04bff1b000)
libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007f04bfab8000)
libc.so.6 => /lib64/libc.so.6 (0x00007f04bf6ea000)
/lib64/ld-linux-x86-64.so.2 (0x00007f04c0ac5000)
libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007f04bf49d000)
libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f04bf1b4000)
libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f04befb0000)
libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007f04bed7d000)
libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007f04beb6d000)
libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f04be969000)
libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f04be74f000)
libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f04be528000)
libpcre.so.1 => /lib64/libpcre.so.1 (0x00007f04be2c6000)

So how is this possible? Did I static link the library, the program is about 0.3 MB larger..

If it works, that's almost a certainty.

If you want to go even deeper into understanding what's in an executable, there is nm.

https://www.mkssoftware.com/docs/man1/nm.1.asp

I got a libmariadbclient.a file in my /usr/lib64 on my local VM, that explains.

1 Like

So I experimented more with this and the conclusion is a little disappointing.
Because the loader searches on runtime in the default directories for the shared library, it will not find the one in the special directory. Using the environment variable $LD_LIBRARY_PATH can help on compile time to find the dependency, and when run. But the environment variable is not persistent between SSH sessions so setting this is of no use in my case.
The best option I have for now is continuing with linking the .a library (so remove or rename the .so one) and making it static.
I suspect the PHP on this system uses the very same method and is also not using the .so lib.

And just when writing this up, another idea pops up. Passing the environment variable through the .htaccess file using SetEnv directive.

Well that would not work because the libname is not .so but .so.3 and a symlink is also not an option.
Putting the .so.3 and a symlink .so in the directory (cgi-bin) is also no option, because that also does not work. (Strange but true)

There's almost always some way to configure environment variables persistently.

There's .profile .bash-profile, and .bashrc, for example.

(Remember to be sure to do export VAR=xxx and not set VAR=xxx, because SET doesn't pass the variable up to the calling environment of a shell script.)

If you're trying to configure an environment variable to be present when the web server is running (e.g. you're not logged in) that's different. But it's certainly something a host that let you run your own processes should let you specify one way or another.

I suspect the PHP on this system uses the very same method and is also not using the .so lib.

Seems you now know enough to dig around in the executables, the libraries they depend on, and that those depend on etc. to find the answers to such questions vs. guess...

1 Like

Great. So having no symlink and no host provider to put one up, I dug deeper again.

Conclusion:

  1. It will be easiest to link with the static library ( .a) and use the resulting executable.

  2. It is possible to have a workaround!

Let me shortly explain the workaround now.

Instead of using the direct method using the Rebol CGI program on the shebang line like this

#!/path/to/cgi-bin/r3 -c

We will use the bash shell as an intermediate.

#!/bin/bash
export LD_LIBRARY_PATH=/path/to/cgi-bin/
sed -n -e '5,$p' < "$0" | /path/to/cgi-bin/r3 -c "$0" "$@"
exit $?
REBOL []
print "Content-type: text/html^/Foo"

(Source Multiline Shebang )
What happens here?
First line calls the bash shell
Second line exports the LD_LIBRARY_PATH environment variable to add the directory where the runtime loader will look for the dynamic libraries used. In this case I made a symbolic link here that points to the appropriate libmariadb.so.3 library.
Third line catches the content of the Rebol script by streaming it from line 5 where it starts until the end ( $p ) into the variable $0 and pipes this through to the Rebol r3 CGI program, using -c to signal it is to be executed as a CGI script (I managed to catch the welcome message of the Rebol console in my error log! :smiley: )
On line 4 the bash shell is exited and the script itself is following after that.

I will document this option, but as you will understand I will be using the static linked version from now on.

sed is one of those unreadable things we're attempting to replace. What keeps you from making the output of the script exactly what you want? And if you need to manipulate it, it would be a better exercise to do it with the r3 executable...

It's often easiest to use statically linked versions of things. But then two programs running on the same machine that both static link a library wind up loading two copies of the library into memory if they both run. So it multiplies the disk space you use, as well as potentially the memory when you are running multiple copies at once.

An additional thing is that when you statically link a library version into your executable, then that means if it's something that gets security updates or patches (like OpenSSL) your executable won't get those patches even if the installed shared library does.

For these reasons (excuses?) you will find static linking is not always made all that easy...and sometimes, even made impossible.

The fact that I cannot use the r3 executable is because it will not load for the missing of the dynamic library for mariadb, and putting one extra version of it there to use just for this sole purpose feels like overkill to me.

However if the r3 executable could load the mysql module on the fly by reading a setup file from the current folder (perhaps also using a symlink to the host provided library from the current folder) ... that would be nice.

I see and understand the principle.
How would this work out?
Will the native mysql-query always have to expect a string and a block of items to be replaced, even if the query is used when there is NO user input at all intended? I have several scripts that are just internal use on the website, like retrieving the number of visitors (and upping it by 1 and saving it). An extra empty block to add seems useless to me.

Perhaps introducing a mysql-execute function for this is a good option.

PHP real escape example So this can be done from within the script handling the input itself when a function mysql-real-escape-string is available.

There is an alternative in prepared statements too.

The prepared statements offer a "placeholder" form, similar to ODBC...which may be better than using the escaping form:

"As an alternative to explicitly escaping special characters, many MySQL APIs provide a placeholder capability that enables you to insert special markers into a statement string, and then bind data values to them when you issue the statement. In this case, the API takes care of escaping special characters in the values for you."

An advantage of using that form would be that it would be more similar to the way the ODBC is implemented.

If the implementation is done in layers, then you can have a higher level form which takes a single parameter...either a string, or a block with the values escaped inline.

See the ODBC-EXECUTE example.

What I suggest is making the C natives about as simple and close to the C API they wrap as possible, and doing the transformative work in usermode code above that.

1 Like

I managed to achieve what I wanted, using the dynamic library that was installed outside one of the regular folders.

I had to link with

-Wl,-rpath,/usr/lib64/irregularfolder

and

-L/usr/lib64/irregularfolder -lmariadb

I checked on the server and ldd gave me the usage as I wanted to see it.
Only hurdle I had to face was an illegal Foo in my header info, I had typed

print "Content-type: text/html^/Foo"

Where I should have typed

print "Content-type: text/html^/^/Foo"

(The second ^/ is implicit if you omit the "Foo" part)
So that scared me for a moment as if it were :jack_o_lantern: Halloween :jack_o_lantern: already :ghost:

Now up to integrating this solution into the build process.

I finally convinced my hosting provider to fix the setup. Pointing at this page about shared libraries I noticed the ldconfig had not been updated. So now I can use the programs I compile by the regular method and do not need to use the -Wl,rpath and other arguments any more. These tricks will be included in some form of documentation though in case somebody will run into similar issues on future occasions.

2 Likes