C plugin system: dlopen fails - c

Question

as a continuation to this post C pluginsystem: symbol lookup error, I am still writing my plugin system and encounter new bugs.
To recap what the plugins are, the program consists of a network application interfaced by a shell, messages has a type and therefore can be use to create applications on the network. For exemple, a possible application would be a chat or a transfert application.
So shell commands can send message of a particular application on the network, when a message is received, if it corresponds to a particular application then an action function is executed with the message content as argument, it could be the application.
A plugin is a shared library with an init function that register it's commands and actions. A command could just be a simple command that doesn't interact with the network, and that's why I achieved at the moment.
The plugin system consists in modules:
plugin_system.c
list.c used by the first module to store plugins
The network part consists in:
protocol.c main part of the protocol
message.c main part for message treatment
application.c main part used to program applications
common.c file with ccommon functions
network.c useful network functions
The modules in protocole are all interdependent, I have split files for conveniency.
All modules are compiled with -fPIC option.
To compile a plugin called plug.c wich doesn't interact with the network, I use:
gcc -Wall -O2 -std=gnu99 -D DEBUG -g -fPIC -c -o plug.o plug.c
gcc -Wall -O2 -std=gnu99 -D DEBUG -g -o plug.so plug.o plugin_system.o list.o -shared
And it works perfectly, the library is loaded with dlopen with no problem, the init function loaded with dlsym and executed correctly so the plugin is registered, I then executed the command and I can see that it work.
Now I wan't to add supports for network communications for the plugins, so I have modified the same test plugin that I used which has just a command to print a message. I have had a call to sendappmessage_all a function that send a message to everyone over the network, defined in message.c.
I can compile the new plugin without adding the network module objects, it compile, the plugin loads correctly, but when it call sendappmessage_all obviously it fails with the message
symbol lookup error: ./plugins/zyva.so: undefined symbol: sendappmessage_all
So to make it work, I should like the plugin with network modules so that's what I have done with
gcc -Wall -O2 -std=gnu99 -D DEBUG -g -o plug.so plug.o plugin_system.o list.o protocol.o message.o thread.o common.o application.o network.o -shared
It compile but when I try to load the plugin, dlopen return NULL.
I have also tried to add just one module, at worst it would only result in an undefined symbol error, but I dlopen still return NULL.
I know it's a lot of informations and on the otherside you probably wan't to see the code but I tried to be the clearer in the most succint way I could be because is way more complex and bigger than the post.
Thank you for your understanding.

Answer 1

The problem is that when you compile the plugin system (i.e. functions called by plugins), and link it to the final executable, the linker does not export the symbols used by the plugins in the dynamic symbol table.
There are two options:
Use -rdynamic when linking the final executable, adding all symbols to the dynamic symbol table.
Use -Wl,-dynamic-list,plugin-system.list when linking the final executable, adding symbols listed in file plugin-system.list to the dynamic symbol table.
The file format is simple:
{
sendappmessage_all;
plugin_*;
};
In other words, you can list either each symbol name (function or data structure), or a glob pattern that matches the desired symbol names. Remember the semicolon after each symbol, and after the closing brace, or you'll get a "syntax error in dynamic list" error at link time.
Note that just marking a function "used" via __attribute__((used)) is not sufficient to make the linker include it in the dynamic symbol table (with GCC 4.8.4 and GNU ld 2.24, at least).
Since the OP thinks what I wrote above is incorrect, here is a fully verifiable proof of the above.
First, a simple main.c that loads plugin files named on the command line, and executes their const char *register_plugin(void); function. Because the function name is shared across all plugins, we need to link them locally (RTLD_LOCAL).
#include <stdlib.h>
#include <string.h>
#include <dlfcn.h>
#include <stdio.h>
static const char *load_plugin(const char *pathname)
{
const char *errmsg;
void *handle; /* We deliberately leak the handle */
const char * (*initfunc)(void);
if (!pathname || !*pathname)
return "No path specified";
dlerror();
handle = dlopen(pathname, RTLD_NOW | RTLD_LOCAL);
errmsg = dlerror();
if (errmsg)
return errmsg;
initfunc = dlsym(handle, "register_plugin");
errmsg = dlerror();
if (errmsg)
return errmsg;
return initfunc();
}
int main(int argc, char *argv[])
{
const char *errmsg;
int arg;
if (argc < 1 || !strcmp(argv[1], "-h") || !strcmp(argv[1], "--help")) {
fprintf(stderr, "\n");
fprintf(stderr, "Usage: %s [ -h | --help ]\n", argv[0]);
fprintf(stderr, " %s plugin [ plugin ... ]\n", argv[0]);
fprintf(stderr, "\n");
return EXIT_SUCCESS;
}
for (arg = 1; arg < argc; arg++) {
errmsg = load_plugin(argv[arg]);
if (errmsg) {
fflush(stdout);
fprintf(stderr, "%s: %s.\n", argv[arg], errmsg);
return EXIT_FAILURE;
}
}
fflush(stdout);
fprintf(stderr, "All plugins loaded successfully.\n");
return EXIT_SUCCESS;
}
The plugins will have access via certain functions (and/or variables), declared in plugin_system.h:
#ifndef PLUGIN_SYSTEM_H
#define PLUGIN_SYSTEM_H
extern void plugin_message(const char *);
#endif /* PLUGIN_SYSTEM_H */
They are implemented in plugin_system.c:
#include <stdio.h>
void plugin_message(const char *msg)
{
fputs(msg, stderr);
}
and listed as dynamic symbols in plugin_system.list:
{
plugin_message;
};
We'll also need a plugin, plugin_foo.c:
#include <stdlib.h>
#include "plugin_system.h"
const char *register_plugin(void) __attribute__((used));
const char *register_plugin(void)
{
plugin_message("Plugin 'foo' is here.\n");
return NULL;
}
and just to remove any confusion about what effect there is having each plugin a registration function by the same name, another plugin named plugin_bar.c:
#include <stdlib.h>
#include "plugin_system.h"
const char *register_plugin(void) __attribute__((used));
const char *register_plugin(void)
{
plugin_message("Plugin 'bar' is here.\n");
return NULL;
}
To make all of this easy to compile, we'll need a Makefile:
CC := gcc
CFLAGS := -Wall -Wextra -O2
LDFLAGS := -ldl -Wl,-dynamic-list,plugin_system.list
PLUGIN_CFLAGS := $(CFLAGS)
PLUGIN_LDFLAGS := -fPIC
PLUGINS := plugin_foo.so plugin_bar.so
PROGS := example
.phony: all clean progs plugins
all: clean progs plugins
clean:
rm -f *.o $(PLUGINS) $(PROGS)
%.so: %.c
$(CC) $(PLUGIN_CFLAGS) $^ $(PLUGIN_LDFLAGS) -shared -Wl,-soname,$# -o $#
%.o: %.c
$(CC) $(CFLAGS) -c $^
plugins: $(PLUGINS)
progs: $(PROGS)
example: main.o plugin_system.o
$(CC) $(CFLAGS) $^ $(LDFLAGS) -o $#
Note that Makefiles require intendation by tabs, not spaces; listing the file here always converts them to spaces. So, if you paste the above to a file, you'll need to fix the indentation, via e.g.
sed -e 's|^ *|\t|' -i Makefile
It is safe to run that more than once; the worst it can do, is mess up your "human-readable" layout.
Compile the above using e.g.
make
and run it via e.g.
./example ./plugin_bar.so ./plugin_foo.so
which shall output
Plugin 'bar' is here.
Plugin 'foo' is here.
All plugins loaded successfully.
to standard error.
Personally, I prefer to register my plugins via a structure, with a version number, and at least one function pointer (to the initialization function). This lets me load all plugins before initializing them, and resolve e.g. interplugin conflicts or dependencies. (In other words, I use a structure with a fixed name, rather than a function with a fixed name, to identify plugins.)
Now, as to __attribute__((used)). If you modify plugin_system.c into
#include <stdio.h>
void plugin_message(const char *msg) __attribute__((used));
void plugin_message(const char *msg)
{
fputs(msg, stderr);
}
and modify the Makefile to have LDFLAGS := -ldl only, the example program and plugins will compile just fine, but running it will yield
./plugin_bar.so: ./plugin_bar.so: undefined symbol: plugin_message.
In other words, if the API exported to plugins is compiled in a separate compilation unit, you will need to use either -rdynamic or -Wl,-dynamic-list,plugin-system.list to ensure the functions are included in the dynamic symbol table in the final executable; the used attribute does not suffice.
If you want all and only non-static functions and symbols in plugin_system.o included in dynamic symbol table in the final binary, you can e.g. modify the end of the Makefile into
example: main.o plugin_system.o
#rm -f plugin_system.list
./list_globals.sh plugin_system.o > plugin_system.list
$(CC) $(CFLAGS) $^ $(LDFLAGS) -o $#
with list_globals.sh:
#!/bin/sh
[ $# -ge 1 ] || exit 0
export LANG=C LC_ALL=C
IFS=:
IFS="$(printf '\t ')"
printf '{\n'
readelf -s "$#" | while read Num Value Size Type Bind Vis Ndx Name Dummy ; do
[ -n "$Name" ] || continue
if [ "$Bind:$Type" = "GLOBAL:FUNC" ]; then
printf ' %s;\n' "$Name"
elif [ "$Bind:$Type:$Ndx" = "GLOBAL:OBJECT:COM" ]; then
printf ' %s;\n' "$Name"
fi
done
printf '};\n'
Remember to make the script executable, chmod u+x list_globals.sh.