I want to write a program using the new SCHED_DEADLINE scheduling policy available since Linux 3.14.
I start out with a simple program trying to use the sched_setattr function.
#include <sched.h>
int main(void)
{
// struct sched_attr attr;
// attr.size = sizeof(struct sched_attr);
// attr.sched_policy = SCHED_DEADLINE;
sched_setattr(0, (void*)0, 0);
return 0;
}
However when compiling I get the following error:
$gcc dead.c
dead.c: In function ‘main’:
dead.c:8:2: warning: implicit declaration of function ‘sched_setattr’ [-Wimplicit-function-declaration]
sched_setattr(0, (void*)0, 0);
^~~~~~~~~~~~~
/tmp/ccGxWxZE.o: In function `main':
dead.c:(.text+0x19): undefined reference to `sched_setattr'
collect2: error: ld returned 1 exit status
My system is running Ubuntu 16.10 Yakkety, with kernel 4.8.0-59-generic. The sched.h file included is found in /usr/include/sched.h and is provided by the package libc6-dev. This headerfile does not contain the function sched_setattr and friends that I am trying to use.
However the kernel (and kernel headers) I have installed comes with a sched.h header file containing the definitions I need. It is located at /usr/src/linux-headers-4.8.0-58/include/linux/sched.h, on my system.
So I naively think lets just build against the newer linux headers instead of the libc6-dev provided headers. My program will only run on this or newer kernels, but that is just fine.
I modify the first line to be: #include <linux/sched.h> and execute:
gcc -I/usr/src/linux-headers-$(uname -r)/include -I/usr/src/linux-headers-$(unam -r)/arch/x86/include dead.c
Now I am getting page after page of errors and warning. This does not seem the way to go.
What is the correct way to build a userspace program against a newer Linux headers than those that are provided by libc?
And subsequently how do I build the program above?
sched_setattr() is a syscall and doesn't seem to have one-to-one libc wrapper. You could do the wrapper yourself, something like this:
#define _GNU_SOURCE
#include <stdio.h>
#include <string.h>
#include <stdint.h>
#include <unistd.h>
#include <linux/sched.h>
#include <sys/syscall.h>
#include <sys/types.h>
struct sched_attr {
uint32_t size; /* Size of this structure */
uint32_t sched_policy; /* Policy (SCHED_*) */
uint64_t sched_flags; /* Flags */
int32_t sched_nice; /* Nice value (SCHED_OTHER, SCHED_BATCH) */
uint32_t sched_priority; /* Static priority (SCHED_FIFO, SCHED_RR) */
/* Remaining fields are for SCHED_DEADLINE */
uint64_t sched_runtime;
uint64_t sched_deadline;
uint64_t sched_period;
};
static int sched_setattr (pid_t pid, const struct sched_attr *attr, unsigned int flags)
{
return syscall (SYS_sched_setattr, pid, attr, flags);
}
int main (int argc, char *argv[])
{
struct sched_attr attr;
int res;
memset (&attr, 0, sizeof (struct sched_attr));
attr.size = sizeof (struct sched_attr);
res = sched_setattr (getpid (), &attr, 0);
if (res < 0) {
perror ("sched_setattr");
return 1;
}
return 0;
}
Looking at the errors reported when trying to include kernel header files required to get the definition of struct sched_attr and reading the comments found by Googling "kernel headers in user space", I really can't suggest trying to include kernel header files just for this.
Related
Ubuntu 18.04
I'm trying to use statx syscall introduced in the Linux Kernel 4.11. There is a manual entry:
#include <sys/types.h>
#include <sys/stat.h>
#include <unistd.h>
#include <fcntl.h> /* Definition of AT_* constants */
int statx(int dirfd, const char *pathname, int flags,
unsigned int mask, struct statx *statxbuf);
So I tried to write an example by myself:
const char *dir_path = NULL;
const char *file_path = NULL;
//read from command line arguments
int dir_fd = open(dir_path, O_DIRECTORY);
struct statx st; //<--------------------------- compile error
statx(dir_fd, file_path, 0, &statx);
But it simply does not compile. The error is the sizeof(statx) is unknown. And actually it is not defined in sys/stat.h, but in linux/stat.h which is not included by sys/stat.h. But after including linux/stat.h the problem is there is no definition for
int statx(int dirfd, const char *pathname, int flags,
unsigned int mask, struct statx *statxbuf);
I expected that since
$ uname -r
4.15.0-39-generic
and 4.15.0-39-generic newer than 4.11 I can use it.
What's wrong?
Currently as the glibc does not provide a wrapper for the statx call, you have to use your kernels definitions. So either copy the statx structure definition from your kernel or just use it from the API the linux kernel provides. The struct statx is currently defined in linux/stat.h.
linux provides a example call to statx available here.
#update library support was added in glibc 2.28
Currently I'm running Ubuntu 16.04 with linux kernel version to be 4.16. I wrote a dummy program that changes its scheduler to SCHED_DEADLINE. But when I tried to compile it, it cannot find definition of structs and macros needed for SCHED_DEADLINE. Most of the code snippet was taken from here (page 24). Below is the test program:
#define _GNU_SOURCE
#include <pthread.h>
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <sched.h>
int main(int argc, char* argv[]) {
struct sched_attr attr;
attr.size = sizeof(attr);
attr.sched_policy = SCHED_DEADLINE;
attr.sched_runtime = 30000000;
attr.sched_period = 100000000;
attr.sched_deadline = attr.sched_period;
if (sched_setattr(gettid(), &attr, 0))
perror("sched_setattr()");
return 0;
}
Here's the output of the compilation:
sched_deadline.c: In function ‘main’:
sched_deadline.c:11:20: error: storage size of ‘attr’ isn’t known
struct sched_attr attr;
^
sched_deadline.c:12:21: error: invalid application of ‘sizeof’ to incomplete type ‘struct attr’
attr.size = sizeof(struct attr);
^
sched_deadline.c:13:22: error: ‘SCHED_DEADLINE’ undeclared (first use in this function)
attr.sched_policy = SCHED_DEADLINE;
My gcc version:
gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.9)
However, the sample code posted in the official website works for me, but the sample code manually defines all the needed macros and system calls in the program. My goal was to compile the application without adding those definitions, which should already be included in the newest kernel version. I have seen various places saying that SCHED_DEADLINE is officially supported after Linux 3.14.10, and upgrading the kernel would automatically solve this issue.
Things I've tried:
Recompiling 4.16 kernel. Previously I thought I need to turn on a switch in config file, but I was not able to find it.
Look into /usr/include/linux/sched.h. Clearly the macros are defined in this header file, but somehow my compiler cannot find it.
I also looked into other posts in the community, but all those questions are for older linux (pre 3.14.10).
You need to include #include <linux/sched.h>
But for the definition of sched_setattr() and gettid(), see the link posted by #CraigEstey
The reason about that, it that glibc will not add function wrappers of linux specific syscall.
For example for gettid(), in the manual we can read this:
Note: There is no glibc wrapper for this system call; see NOTES.
Glibc does not provide a wrapper for this system call; call it using
syscall(2).
The thread ID returned by this call is not the same thing as a POSIX thread ID
Have a look at this article: https://lwn.net/Articles/711058/
#define _GNU_SOURCE
#include <stdint.h>
#include <stdio.h>
#include <unistd.h>
#include <sys/syscall.h>
#include <sched.h>
#include <linux/sched.h>
#include <sys/types.h>
struct sched_attr {
uint32_t size;
uint32_t sched_policy;
uint64_t sched_flags;
/* SCHED_NORMAL, SCHED_BATCH */
int32_t sched_nice;
/* SCHED_FIFO, SCHED_RR */
uint32_t sched_priority;
/* SCHED_DEADLINE (nsec) */
uint64_t sched_runtime;
uint64_t sched_deadline;
uint64_t sched_period;
};
int sched_setattr(pid_t pid, const struct sched_attr *attr, unsigned int flags)
{
return syscall(__NR_sched_setattr, pid, attr, flags);
}
int main(int argc, char* argv[]) {
struct sched_attr attr = {
.size = sizeof(attr),
.sched_policy = SCHED_DEADLINE,
.sched_runtime = 30000000,
.sched_period = 100000000,
.sched_deadline = 100000000
};
pid_t tid = syscall(SYS_gettid);
if (sched_setattr(tid, &attr, 0))
perror("sched_setattr()");
return 0;
}
Or a more shorter code, without the redefinition of struct sched_attr
#define _GNU_SOURCE
#include <stdint.h>
#include <stdio.h>
#include <unistd.h>
#include <sys/syscall.h>
#include <linux/sched/types.h>
#include <linux/sched.h>
#include <sys/types.h>
int sched_setattr(pid_t pid, const struct sched_attr *attr, unsigned int flags)
{
return syscall(__NR_sched_setattr, pid, attr, flags);
}
int main(int argc, char* argv[]) {
struct sched_attr attr = {
.size = sizeof(attr),
.sched_policy = SCHED_DEADLINE,
.sched_runtime = 30000000,
.sched_period = 100000000,
.sched_deadline = 100000000
};
pid_t tid = syscall(SYS_gettid);
if (sched_setattr(tid, &attr, 0))
perror("sched_setattr()");
return 0;
}
But this needs to be executed as root, otherwise I got sched_setattr(): Operation not permitted
Or the application needs to have the right linux capabilities.
I have a structure defined in /usr/src/linux-3.2/include/linux/unistd.h of the linux kernel:
#ifndef _LINUX_UNISTD_H_
#define _LINUX_UNISTD_H_
struct threadinfo_struct {
int pid;
int nthreads;
int *tid;
};
/*
* Include machine specific syscall numbers
*/
#include <asm/unistd.h>
#endif /* _LINUX_UNISTD_H_ */
After compiling and installing the kernel, then booting from it, I try to compile and run this program:
#include <stdio.h>
#include <linux/unistd.h>
int main(void) {
struct threadinfo_struct *ti = (struct threadinfo_struct*) malloc(sizeof(struct threadinfo_struct));
// ...
return 0;
}
However, when I try and do this, I get an error in compilation of the program:
test.c: In function 'main':
test.c:4:78: error: invalid application of 'sizeof' to incomplete type 'struct threadinfo_struct'
Why am I getting this error, and how can I resolve it? This is difficult for me to find much information, given that I am very new to the linux kernel.
The file /usr/src/linux-3.2/include/linux/unistd.h is not on a standard include path.
The user-space applications have their own build environment. You are including the file that is located at /usr/include/linux/unistd.h. Most of the internal kernel structures are not defined for user-space applications.
If you really need this structure to be defined then you need to copy the file from the linux tree to your project directory, or adjust the gcc command by adding the -isystem/usr/src/linux-3.2/include/ option.
However, the latter will create a big mess, so better just copy the file.
Background
I'm working on a cross-platform Zeroconf/Bonjour/DNS-SD library for Haskell, and figured my best bet would bet would be to target the dns_sd.h API. Under Linux, the implementation of this interface is provided by Avahi, which claims to support a subset of the Bonjour API.
Problem
As a sanity check for my library, I've written a small test program in C that just uses the bare bones of the API. It browses for any service on the network of type _http._tcp, prints a message as soon as it sees one, and then dies:
#include <dns_sd.h>
#include <stdio.h>
#include <stdlib.h>
void cb(DNSServiceRef sdRef,
DNSServiceFlags flags,
uint32_t interfaceIndex,
DNSServiceErrorType errorCode,
const char *serviceName,
const char *regtype,
const char *replyDomain,
void *context) {
printf("called!\n");
}
int main() {
DNSServiceRef sd = malloc(sizeof(DNSServiceRef));
const char *regtype = "_http._tcp";
DNSServiceErrorType err1 = DNSServiceBrowse(&sd, 0, 0, regtype, NULL, &cb, NULL);
printf("err1=%d\n", err1);
DNSServiceErrorType err2 = DNSServiceProcessResult(sd);
printf("err2=%d\n", err2);
return 0;
}
On my Mac, this test program works fine in both C and the equivalent Haskell (it finds my printer; exciting!):
$ gcc test.c -o test
$ ./test
err1=0
called!
err2=0
But on my Linux machine, the program berates me before exiting without invoking the callback:
$ gcc test.c -o test -ldns_sd
$ ./test
*** WARNING *** The program 'test' uses the Apple Bonjour compatibility layer of Avahi.
*** WARNING *** Please fix your application to use the native API of Avahi!
*** WARNING *** For more information see <http://0pointer.de/avahi-compat?s=libdns_sd&e=test>
err1=0
err2=0
Questions
Is the Avahi dns_sd compatibility layer still a suitable target for a cross-platform binding? Or is that warning message serious enough about using the native Avahi API that I should consider retargeting?
What is the state of the art for cross-platform Zeroconf in C?
By reason unknown to me, it works only with non-blocking calls. Below is the improved code. Socket from Avahi is set to a non-blocking mode and then select (3) is used to wait for available data. DNSServiceProcessResult(sd) has to be called each time there is data on the socket so it may have been pure luck that your example worked on other platforms.
#include <fcntl.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <dns_sd.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>
static int set_nonblocking(int fd)
{
int flags;
/* If they have O_NONBLOCK, use the Posix way to do it */
#if defined(O_NONBLOCK)
/* Fixme: O_NONBLOCK is defined but broken on SunOS 4.1.x and AIX 3.2.5. */
if (-1 == (flags = fcntl(fd, F_GETFL, 0)))
flags = 0;
return fcntl(fd, F_SETFL, flags | O_NONBLOCK);
#else
/* Otherwise, use the old way of doing it */
flags = 1;
return ioctl(fd, FIOBIO, &flags);
#endif
}
void cb(DNSServiceRef sdRef,
DNSServiceFlags flags,
uint32_t interfaceIndex,
DNSServiceErrorType errorCode,
const char *serviceName,
const char *regtype,
const char *replyDomain,
void *context) {
printf("called %s %s!\n", serviceName, regtype);
}
int main() {
DNSServiceRef sd = malloc(sizeof(DNSServiceRef));
const char *regtype = "_http._tcp";
DNSServiceErrorType err1 = DNSServiceBrowse(&sd, 0, 0, regtype, NULL, &cb, NULL);
printf("err1=%d\n", err1);
int socket = DNSServiceRefSockFD(sd);
set_nonblocking(socket);
fd_set read_fds;
FD_ZERO(&read_fds);
FD_SET(socket, &read_fds);
while(1) {
if(select(socket+1, &read_fds, NULL, NULL, NULL) < 0) {
perror("select");
}
DNSServiceErrorType err2 = DNSServiceProcessResult(sd);
printf("err2=%d\n", err2);
if(err2 != 0)
return 2;
}
return 0;
}
I have an application that has both two external kernel modules and a userspace daemon. I want to load the modules from the daemon code, written in C, at startup, and unload them on clean exit. Can I load them in a cleaner way than doing system("modprobe module"); and unload them using the corresponding rmmod?
init_module / remove_module minimal runnable example
Tested on a QEMU + Buildroot VM and Ubuntu 16.04 host with this simple parameter printer module .
We use the init_module / finit_module and remove_module Linux system calls.
The Linux kernel offers two system calls for module insertion:
init_module
finit_module
and:
man init_module
documents that:
The finit_module() system call is like init_module(), but reads the module to be loaded from the file descriptor fd. It is useful when the authenticity of a kernel module can be determined from its location in the filesystem; in cases where that is possible, the overhead of using cryptographically signed modules to determine the authenticity of a module can be avoided. The param_values argument is as for init_module().
finit is newer and was added only in v3.8. More rationale: https://lwn.net/Articles/519010/
glibc does not seem to provide a C wrapper for them, so we just create our own with syscall.
insmod.c
#define _GNU_SOURCE
#include <fcntl.h>
#include <stdio.h>
#include <sys/stat.h>
#include <sys/syscall.h>
#include <sys/types.h>
#include <unistd.h>
#include <stdlib.h>
#define init_module(module_image, len, param_values) syscall(__NR_init_module, module_image, len, param_values)
#define finit_module(fd, param_values, flags) syscall(__NR_finit_module, fd, param_values, flags)
int main(int argc, char **argv) {
const char *params;
int fd, use_finit;
size_t image_size;
struct stat st;
void *image;
/* CLI handling. */
if (argc < 2) {
puts("Usage ./prog mymodule.ko [args="" [use_finit=0]");
return EXIT_FAILURE;
}
if (argc < 3) {
params = "";
} else {
params = argv[2];
}
if (argc < 4) {
use_finit = 0;
} else {
use_finit = (argv[3][0] != '0');
}
/* Action. */
fd = open(argv[1], O_RDONLY);
if (use_finit) {
puts("finit");
if (finit_module(fd, params, 0) != 0) {
perror("finit_module");
return EXIT_FAILURE;
}
close(fd);
} else {
puts("init");
fstat(fd, &st);
image_size = st.st_size;
image = malloc(image_size);
read(fd, image, image_size);
close(fd);
if (init_module(image, image_size, params) != 0) {
perror("init_module");
return EXIT_FAILURE;
}
free(image);
}
return EXIT_SUCCESS;
}
GitHub upstream.
rmmod.c
#define _GNU_SOURCE
#include <fcntl.h>
#include <stdio.h>
#include <sys/stat.h>
#include <sys/syscall.h>
#include <sys/types.h>
#include <unistd.h>
#include <stdlib.h>
#define delete_module(name, flags) syscall(__NR_delete_module, name, flags)
int main(int argc, char **argv) {
if (argc != 2) {
puts("Usage ./prog mymodule");
return EXIT_FAILURE;
}
if (delete_module(argv[1], O_NONBLOCK) != 0) {
perror("delete_module");
return EXIT_FAILURE;
}
return EXIT_SUCCESS;
}
GitHub upstream.
Busybox source interpretation
Busybox provides insmod, and since it is designed for minimalism, we can try to deduce how it is done from there.
On version 1.24.2, the entry point is at modutils/insmod.c function insmod_main.
The IF_FEATURE_2_4_MODULES is optional support for older Linux kernel 2.4 modules, so we can just ignore it for now.
That just forwards to modutils.c function bb_init_module.
bb_init_module attempts two things:
mmap the file to memory through try_to_mmap_module.
This always sets image_size to the size of the .ko file as a side effect.
if that fails, malloc the file to memory with xmalloc_open_zipped_read_close.
This function optionally unzips the file first if it is a zip, and just mallocs it otherwise.
I don't understand why this zipping business is done, since we can't even rely on it because the try_to_mmap_module does not seem to unzip things.
Finally comes the call:
init_module(image, image_size, options);
where image is the executable that was put into memory, and options are just "" if we call insmod file.elf without further arguments.
init_module is provided above by:
#ifdef __UCLIBC__
extern int init_module(void *module, unsigned long len, const char *options);
extern int delete_module(const char *module, unsigned int flags);
#else
# include <sys/syscall.h>
# define init_module(mod, len, opts) syscall(__NR_init_module, mod, len, opts)
# define delete_module(mod, flags) syscall(__NR_delete_module, mod, flags)
#endif
ulibc is an embedded libc implementation, and it seems to provide init_module.
If it is not present, I think glibc is assumed, but as man init_module says:
The init_module() system call is not supported by glibc. No declaration is provided in glibc headers, but, through a quirk of history, glibc does export an ABI for
this system call. Therefore, in order to employ this system call, it is sufficient to manually declare the interface in your code; alternatively, you can invoke
the system call using syscall(2).
BusyBox wisely follows that advice and uses syscall, which glibc provides, and which offers a C API for system calls.
insmod/rmmod use the functions init_module and delete_module to do this, which also have a man-page available. They both declare the functions as extern instead of including a header, but the man-page says they should be in <linux/module.h>.
I'd recommend against the use of system() in any daemon code that runs with root permissions as it's relatively easy to exploit from a security standpoint. modprobe and rmmod are, indeed, the right tools for the job. However, it'd be a bit cleaner and much more secure to use an explicit fork() + exec() to invoke them.
I'm not sure there's a cleaner way than system.
But for sure, if you want to load/unload the modules from your userspace daemon, then you force yourself to run the daemon as root*, which may not be considered as secure.
*: or you can add the explicit commands in the sudoers file, but this will be a nightmare to manage when deploying your application.
You can perform the same tasks that modprobe and Co. do, but I doubt that could be characterized as cleaner.