So I've been looking to this code for hours and I can't seem to understand what is happening, despite my efforts to debug this with printf's everywhere. The following function is where my main problem is:
void drawFolders() {
printf("Going to call getSubFolder\n");
getSubFolders(".");
printf("DEBUG 1 - Return from getFolderName is: %s\n", getFolderName(3));
printf("DEBUG 2 - Return from getFolderName is: %s\n", getFolderName(5));
int k;
for(k = 0; k < getNumberFolders(); k++) {
printf("DEBUG INLOOP %d - Return from getFolderName is: %s\n",k, getFolderName(k));
draw_sprite(k*60 + 30, 50, folder);
//draw_string(temp[i].name,i*60 + 30, 50);
}
}
Basically, in DEBUG1 and DEBUG2 my function getFolderName() seems to work just fine, ie, a string is returned with no problems. However, when I try to call this function inside my for loop, all I get from getFolderName is garbage (except the first one) , even when k is 3 or 5 (cases which I've tested before the for loop and worked fine). k is not wrong, as I print it too. getFolderName does nothing to the array , here is the function :
char* getFolderName(int index) {
printf("Values inside getFolderName() function : %s\n", currentFolders[index].name);
return currentFolders[index].name ;
}
Also it might be important to show what currentFolders is:
typedef struct Directories {
char* name;
int active;
} Directory;
Directory currentFolders[30];
OUTPUT
Dec 20 22:18:48 192 kernel: Vai chamar o getSubFolder
Dec 20 22:18:48 192 kernel: Values inside getSubFolders function: .
Dec 20 22:18:48 192 kernel: Values inside getSubFolders function: ..
Dec 20 22:18:48 192 kernel: Values inside getSubFolders function: usr
Dec 20 22:18:48 192 kernel: Values inside getSubFolders function: lab4
Dec 20 22:18:48 192 kernel: Values inside getSubFolders function: proj
Dec 20 22:18:48 192 kernel: Values inside getSubFolders function: bin
Dec 20 22:18:48 192 kernel: Values inside getSubFolders function: boot
Dec 20 22:18:48 192 kernel: Values inside getSubFolders function: dev
Dec 20 22:18:48 192 kernel: Values inside getSubFolders function: etc
Dec 20 22:18:48 192 kernel: Values inside getSubFolders function: home
Dec 20 22:18:48 192 kernel: Values inside getSubFolders function: mnt
Dec 20 22:18:48 192 kernel: Values inside getSubFolders function: root
Dec 20 22:18:48 192 kernel: Values inside getSubFolders function: sbin
Dec 20 22:18:48 192 kernel: Values inside getSubFolders function: tmp
Dec 20 22:18:48 192 kernel: Values inside getSubFolders function: var
Dec 20 22:18:48 192 kernel: DEBUG 0Values inside getFolderName() function : lab4
Dec 20 22:18:48 192 kernel: DEBUG 1 - Return from getFolderName is: lab4
Dec 20 22:18:48 192 kernel: Values inside getFolderName() function : bin
Dec 20 22:18:48 192 kernel: DEBUG 2 - Return from getFolderName is: bin
Dec 20 22:18:48 192 kernel: Values inside getFolderName() function : .
Dec 20 22:18:48 192 kernel: DEBUG INLOOP 0 - Return from getFolderName is: .
Dec 20 22:18:48 192 kernel: Values inside getFolderName() function :
Dec 20 22:18:48 192 kernel: DEBUG INLOOP 1 - Return from getFolderName is:
Dec 20 22:18:48 192 kernel: Values inside getFolderName() function :
Dec 20 22:18:48 192 kernel: DEBUG INLOOP 2 - Return from getFolderName is:
Dec 20 22:18:48 192 kernel: Values inside getFolderName() function :
Dec 20 22:18:48 192 kernel: DEBUG INLOOP 3 - Return from getFolderName is:
Dec 20 22:18:48 192 kernel: Values inside getFolderName() function :
Dec 20 22:18:48 192 kernel: DEBUG INLOOP 4 - Return from getFolderName is:
Dec 20 22:18:48 192 kernel: Values inside getFolderName() function :
Dec 20 22:18:48 192 kernel: DEBUG INLOOP 5 - Return from getFolderName is:
Dec 20 22:18:48 192 kernel: Values inside getFolderName() function :
Dec 20 22:18:48 192 kernel: DEBUG INLOOP 6 - Return from getFolderName is:
Dec 20 22:18:48 192 kernel: Values inside getFolderName() function :
Dec 20 22:18:48 192 kernel: DEBUG INLOOP 7 - Return from getFolderName is:
Dec 20 22:18:48 192 kernel: Values inside getFolderName() function :
Dec 20 22:18:48 192 kernel: DEBUG INLOOP 8 - Return from getFolderName is:
Dec 20 22:18:48 192 kernel: Values inside getFolderName() function :
Dec 20 22:18:48 192 kernel: DEBUG INLOOP 9 - Return from getFolderName is:
Dec 20 22:18:48 192 kernel: Values inside getFolderName() function :
Dec 20 22:18:48 192 kernel: DEBUG INLOOP 10 - Return from getFolderName is:
Dec 20 22:18:48 192 kernel: Values inside getFolderName() function :
Dec 20 22:18:48 192 kernel: DEBUG INLOOP 11 - Return from getFolderName is:
Dec 20 22:18:48 192 kernel: Values inside getFolderName() function :
Dec 20 22:18:48 192 kernel: DEBUG INLOOP 12 - Return from getFolderName is:
Dec 20 22:18:48 192 kernel: Values inside getFolderName() function :
Dec 20 22:18:48 192 kernel: DEBUG INLOOP 13 - Return from getFolderName is:
Dec 20 22:18:48 192 kernel: Values inside getFolderName() function :
Dec 20 22:18:48 192 kernel: DEBUG INLOOP 14 - Return from getFolderName is:
Other function called that isn't related to the problem, I think, but you might want to take a look so here it is anyway:
int getSubFolders(char* foldername) {
num_folders = 0;
struct dirent *de=NULL;
DIR *d=NULL;
d=opendir(foldername);
if(d == NULL) {
perror("Couldn't open directory");
return(2);
}
int i = 0;
// Loop while not NULL
while(de = readdir(d)) {
num_folders++;
Directory temp;
currentFolders[i].name = de->d_name;
strcpy( currentFolders[i].name, de->d_name);
// printf("nome %s\n",de->d_name);
//printf("temp.name %s\n",temp.name);
// currentFolders[i] = temp;
printf("Values inside getSubFolders function: %s\n",currentFolders[i].name);
i++;
}
printCurrentFolders();
closedir(d);
return(0);
}
Fix it this way, the struct declare it like this
typedef struct Directories {
char name[256];
int active;
} Directory;
and remove this line
currentFolders[i].name = de->d_name;
The problem is caused by closedir(d); cleaning up resources used by opendir, the struct dirent * being part of those resources.
Related
I want to use DMA in a driver that I'm working on. The ultimate goal is to ensure that the data is in physical RAM and not hidden in a cache. For that I was trying to implement a simple test driver before merging it with my current project.
From what I understand I can setup a DMA mask and allocate a coherent buffer and then simply write to the virtual address. My other device can then read from the physical address. It would be nice if someone could confirm if that is indeed how it works.
Unfortunately the allocation fails and I can't decipher the syslogs for why that could be. I'm probably doing something wrong with the struct device. What is the device reference in dma_alloc_coherent used for?
Here's my current attempt. It's a modified dummy character device (note that I now know that it is complete garbage and not at all how you should do dma):
#include <linux/module.h>
#include <linux/fs.h>
#include <linux/device.h>
#include <linux/cdev.h>
#include <linux/uaccess.h>
#include <linux/dma-mapping.h>
static char hello_world[]="Hello World\n";
static dev_t hello_dev_number;
static struct cdev *driver_object;
static struct class *hello_class;
static struct device *hello_dev;
int errorType;
void *virtAddr;
size_t size = 32;
dma_addr_t *physAddr;
int flag = GFP_KERNEL;
static ssize_t driver_read( struct file *instance, char __user *user, size_t count, loff_t *offset )
{
unsigned long not_copied, to_copy;
to_copy = min( count, strlen(hello_world)+1 );
not_copied=copy_to_user(user,hello_world,to_copy);
*offset += to_copy-not_copied;
return to_copy-not_copied;
}
static struct file_operations fops = {
.owner= THIS_MODULE,
.read= driver_read,
};
static int __init mod_init( void )
{
int debug;
printk("starting insertion");
if (alloc_chrdev_region(&hello_dev_number,0,1,"Hello")<0)
return -EIO;
driver_object = cdev_alloc();
if (driver_object==NULL){
errorType = EIO;
goto free_device_number;
}
driver_object->owner = THIS_MODULE;
driver_object->ops = &fops;
if (cdev_add(driver_object,hello_dev_number,1)){
errorType=EIO;
goto free_cdev;
}
hello_class = class_create( THIS_MODULE, "Hello" );
if (IS_ERR( hello_class )) {
pr_err( "hello: no udev support\n");
errorType=EIO;
goto free_cdev;
}
hello_dev = device_create( hello_class, NULL, hello_dev_number, NULL, "%s", "hello" );
if (IS_ERR( hello_dev )) {
pr_err( "hello: device_create failed\n");
errorType=EIO;
goto free_class;
}
printk("dma is happening");
debug = dma_set_mask_and_coherent(hello_dev, DMA_BIT_MASK(32));
printk("dma mask returns: %d", debug);
if(debug==0){
printk("setting mask failed");
errorType=EIO;
goto free_dev;
}
printk("goodbye crule world!");
virtAddr = dma_alloc_coherent(hello_dev, size, physAddr, flag);
printk("I'm actually alive");
if(virtAddr==NULL){
printk("virtual address null, dma failed!");
errorType=ENOMEM;
goto free_dev;
}
printk("success");
return 0;
free_dev:
device_destroy( hello_class, hello_dev_number );
free_class:
class_destroy( hello_class );
free_cdev:
kobject_put( &driver_object->kobj );
free_device_number:
unregister_chrdev_region( hello_dev_number, 1 );
return -errorType;
}
static void __exit mod_exit( void )
{
dma_free_coherent(hello_dev, size, virtAddr, *physAddr);
device_destroy( hello_class, hello_dev_number );
class_destroy( hello_class );
cdev_del( driver_object );
unregister_chrdev_region( hello_dev_number, 1 );
return;
}
module_init( mod_init );
module_exit( mod_exit );
MODULE_AUTHOR("ME");
MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("Testing DMA.");
Here's the Makefile that I use:
obj-m+=dma_test.o
all:
make -C /lib/modules/$(shell uname -r)/build M=$(PWD) modules
clean:
make -C /lib/modules/$(shell uname -r)/build M=$(PWD) clean
Running this on an ARMv7 AM335x Sitara CPU (Beaglebone Black), Linux Kernel 5.4.106-ti-r27
Here's the kernel log:
Mar 14 02:38:23 beaglebone kernel: [ 100.662640] starting insertion
Mar 14 02:38:23 beaglebone kernel: [ 100.667068] dma is happening
Mar 14 02:38:23 beaglebone kernel: [ 100.667087] dma mask returns: -5
Mar 14 02:38:23 beaglebone kernel: [ 100.667092] goodbye crule world!
Mar 14 02:38:23 beaglebone kernel: [ 100.667100] ------------[ cut here ]------------
Mar 14 02:38:23 beaglebone kernel: [ 100.667128] WARNING: CPU: 0 PID: 2360 at kernel/dma/mapping.c:272 dma_alloc_attrs+0x118/0x128
Mar 14 02:38:23 beaglebone kernel: [ 100.667134] Modules linked in: dma_test(O+) c_can_platform c_can can_dev evdev usb_f_acm u_serial usb_f_ecm usb_f_mass_storage usb_f_rndis u_ether libcomposite uio_pdrv_genirq(O) uio iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 iptable_mangle iptable_filter dmatest(O) ip_tables x_tables icss_iep prueth_ecap spidev
Mar 14 02:38:23 beaglebone kernel: [ 100.667216] CPU: 0 PID: 2360 Comm: insmod Tainted: G O 5.4.106-ti-r27 #1buster
Mar 14 02:38:23 beaglebone kernel: [ 100.667222] Hardware name: Generic AM33XX (Flattened Device Tree)
Mar 14 02:38:23 beaglebone kernel: [ 100.667228] Backtrace:
Mar 14 02:38:23 beaglebone kernel: [ 100.667248] [<c0e37fd8>] (dump_backtrace) from [<c0e38390>] (show_stack+0x20/0x24)
Mar 14 02:38:23 beaglebone kernel: [ 100.667259] r7:600f0113 r6:c14e3154 r5:00000000 r4:c14e3154
Mar 14 02:38:23 beaglebone kernel: [ 100.667274] [<c0e38370>] (show_stack) from [<c0e49af0>] (dump_stack+0xb8/0xcc)
Mar 14 02:38:23 beaglebone kernel: [ 100.667289] [<c0e49a38>] (dump_stack) from [<c013c6cc>] (__warn+0xe0/0x108)
Mar 14 02:38:23 beaglebone kernel: [ 100.667299] r7:00000110 r6:00000009 r5:c01c7090 r4:c11194a8
Mar 14 02:38:23 beaglebone kernel: [ 100.667309] [<c013c5ec>] (__warn) from [<c0e38c00>] (warn_slowpath_fmt+0x70/0xd8)
Mar 14 02:38:23 beaglebone kernel: [ 100.667318] r7:00000110 r6:c11194a8 r5:c1405fc8 r4:00000000
Mar 14 02:38:23 beaglebone kernel: [ 100.667328] [<c0e38b94>] (warn_slowpath_fmt) from [<c01c7090>] (dma_alloc_attrs+0x118/0x128)
Mar 14 02:38:23 beaglebone kernel: [ 100.667339] r9:00000cc0 r8:00000000 r7:00000020 r6:dc78d600 r5:c1405fc8 r4:c0f01734
Mar 14 02:38:23 beaglebone kernel: [ 100.667368] [<c01c6f78>] (dma_alloc_attrs) from [<bf00c1b4>] (mod_init+0x1b4/0x1000 [dma_test])
Mar 14 02:38:23 beaglebone kernel: [ 100.667379] r9:bf0e00cc r8:d29cff30 r7:dc78d600 r6:fffffffb r5:00000000 r4:bf0e0300
Mar 14 02:38:23 beaglebone kernel: [ 100.667399] [<bf00c000>] (mod_init [dma_test]) from [<c0103268>] (do_one_initcall+0x50/0x2d0)
Mar 14 02:38:23 beaglebone kernel: [ 100.667408] r7:00000000 r6:bf0e01f0 r5:bf00c000 r4:c1405fc8
Mar 14 02:38:23 beaglebone kernel: [ 100.667423] [<c0103218>] (do_one_initcall) from [<c01f0878>] (do_init_module+0x70/0x274)
Mar 14 02:38:23 beaglebone kernel: [ 100.667433] r8:d29cff30 r7:bf0e00c0 r6:bf0e01f0 r5:db20cf40 r4:bf0e00c0
Mar 14 02:38:23 beaglebone kernel: [ 100.667444] [<c01f0808>] (do_init_module) from [<c01f2adc>] (load_module+0x1f64/0x2370)
Mar 14 02:38:23 beaglebone kernel: [ 100.667452] r6:bf0e01f0 r5:00000000 r4:bf0e01c0
Mar 14 02:38:23 beaglebone kernel: [ 100.667463] [<c01f0b78>] (load_module) from [<c01f318c>] (sys_finit_module+0xc0/0x110)
Mar 14 02:38:23 beaglebone kernel: [ 100.667474] r10:0000017b r9:d29ce000 r8:c0101204 r7:0043c7e0 r6:00000003 r5:00000000
Mar 14 02:38:23 beaglebone kernel: [ 100.667480] r4:c1405fc8
Mar 14 02:38:23 beaglebone kernel: [ 100.667491] [<c01f30cc>] (sys_finit_module) from [<c0101000>] (ret_fast_syscall+0x0/0x54)
Mar 14 02:38:23 beaglebone kernel: [ 100.667499] Exception stack(0xd29cffa8 to 0xd29cfff0)
Mar 14 02:38:23 beaglebone kernel: [ 100.667511] ffa0: 3d6d5800 00000000 00000003 0043c7e0 00000000 bece2578
Mar 14 02:38:23 beaglebone kernel: [ 100.667522] ffc0: 3d6d5800 00000000 00000000 0000017b 015347e0 00000000 bece26f8 00000000
Mar 14 02:38:23 beaglebone kernel: [ 100.667531] ffe0: bece2528 bece2518 00434e41 b6cbbd92
Mar 14 02:38:23 beaglebone kernel: [ 100.667540] r7:0000017b r6:00000000 r5:00000000 r4:3d6d5800
Mar 14 02:38:23 beaglebone kernel: [ 100.667547] ---[ end trace 870c2d2ad09e80fa ]---
Mar 14 02:38:23 beaglebone kernel: [ 100.667565] Hello hello: coherent DMA mask is unset
Mar 14 02:38:23 beaglebone kernel: [ 100.667571] I'm actually alive
You need to allocate space for the dma_addr_t value.
- dma_addr_t *physAddr;
+ dma_addr_t physAddr;
- virtAddr = dma_alloc_coherent(hello_dev, size, physAddr, flag);
+ virtAddr = dma_alloc_coherent(hello_dev, size, &physAddr, flag);
- dma_free_coherent(hello_dev, size, virtAddr, *physAddr);
+ dma_free_coherent(hello_dev, size, virtAddr, physAddr);
Here is an online example and looking through the source for callers of dma_alloc_coherent() should confirm this.
The API is this way as 'C' does not allow multiple return values. In C++, it might have been a reference. To most kernel developers that have a breadth of experience with 'C', this would seem second nature.
How did I deduce this? This helps,
[<c013c5ec>] (__warn) from [<c0e38c00>] (warn_slowpath_fmt+0x70/0xd8)
[<c01c7090>] (dma_alloc_attrs+0x118/0x128)
[<bf00c1b4>] (mod_init+0x1b4/0x1000 [dma_test])
A call to warn was issued so some parameter seemed to be wrong.
Besides the issue pointed by artless noise, your codes still had 2 bugs.
dev->dma_mask is NULL, so dma_set_mask_and_coherent() returns -5 (-EIO). So you need to add something as follows into between device_create() and dma_set_mask_and_coherent().
u64 dma_mask = DMA_BIT_MASK(32);
hello_dev->dma_mask = &dma_mask;
dma_set_mask_and_coherent() returns non-zero on errors. So you need to change the code to if (debug)
I just switched from POSIX to SysV because the limit is much higher on SysV (1024 vs 10). But I still need an higher limit. The limit should be changed at runtime, because it depends on the data the user chooses.
Using POSIX, it was possible to increase the limit but it was necessary to run the code as root every time and I cannot do that.
Is there a way to increase the limit in SysV?
As SYSV IPC are considered deprecated, it is a pity to design brand new applications with thoses old fashioned services.
POSIX message queues are based on a file system. It is usually mounted on /dev/mqueue:
$ ls -la /dev/mqueue
total 0
drwxrwxrwt 2 root root 40 dec. 25 21:02 .
drwxr-xr-x 20 root root 4600 dec. 25 20:51 ..
A message queue is created with mq_open(). The attr parameter provides the ability to set some attributes:
struct mq_attr {
long mq_flags; /* Flags (ignored for mq_open()) */
long mq_maxmsg; /* Max. # of messages on queue */
long mq_msgsize; /* Max. message size (bytes) */
long mq_curmsgs; /* # of messages currently in queue (ignored for mq_open()) */
};
According to the documentation, the /proc/sys/fs/mqueue/msg_max file defines the ceiling value for the maximum number of messages in a queue. In Linux 5.4, its default value is DFLT_MSGMAX (10) and its upper limit is HARD_MSGMAX (65536).
The defaults values are defined in the Linux source code (cf. /include/linux/ipc_namespace.h):
#define DFLT_QUEUESMAX 256
#define MIN_MSGMAX 1
#define DFLT_MSG 10U
#define DFLT_MSGMAX 10
#define HARD_MSGMAX 65536
#define MIN_MSGSIZEMAX 128
#define DFLT_MSGSIZE 8192U
#define DFLT_MSGSIZEMAX 8192
#define HARD_MSGSIZEMAX (16*1024*1024)
Here is an example program which creates a message queue. It receives as parameters the message queue name and the maximum number of messages in the queue:
#include <errno.h>
#include <pthread.h>
#include <mqueue.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
int main(int argc, char *argv[])
{
mqd_t mqdes;
struct mq_attr attr;
if (argc != 3) {
fprintf(stderr, "Usage: %s <mq-name> <msg_max>\n", argv[0]);
exit(EXIT_FAILURE);
}
attr.mq_flags = 0;
attr.mq_maxmsg = atoi(argv[2]);
attr.mq_msgsize = 2048;
attr.mq_curmsgs = 0;
mqdes = mq_open(argv[1], O_CREAT | O_RDWR, 0777, &attr);
if (mqdes == (mqd_t) -1) {
perror("mq_open");
return 1;
}
return 0;
}
At execution time we can verify that, by default, we can't go above 10 messages for the queue:
$ gcc mq.c -o mq -lrt
$ ./mq
Usage: ./mq <mq-name> <msg_max>
$ ./mq /q0 5
$ ls -la /dev/mqueue/
total 0
drwxrwxrwt 2 root root 60 dec. 25 21:09 .
drwxr-xr-x 20 root root 4600 dec. 25 20:51 ..
-rwxrwxr-x 1 xxxx xxxx 80 dec. 25 21:09 q0
$ ./mq /q1 9
$ ./mq /q2 10
$ ./mq /q3 11
mq_open: Invalid argument
$ ls -la /dev/mqueue/
total 0
drwxrwxrwt 2 root root 100 dec. 25 21:10 .
drwxr-xr-x 20 root root 4600 dec. 25 20:51 ..
-rwxrwxr-x 1 xxxx xxxx 80 dec. 25 21:09 q0
-rwxrwxr-x 1 xxxx xxxx 80 dec. 25 21:10 q1
-rwxrwxr-x 1 xxxx xxxx 80 dec. 25 21:10 q2
Let's change the ceiling value from 10 to 256 for msg_max:
$ cat /proc/sys/fs/mqueue/msg_max
10
$ sudo sh -c "echo 256 > /proc/sys/fs/mqueue/msg_max"
$ cat /proc/sys/fs/mqueue/msg_max
256
Now it is possible to create message queues with up to 256 messages:
$ ./mq /q3 11
$ ./mq /q4 256
$ ./mq /q5 257
mq_open: Invalid argument
$ ls -la /dev/mqueue/
total 0
drwxrwxrwt 2 root root 140 dec. 25 21:16 .
drwxr-xr-x 20 root root 4600 dec. 25 20:51 ..
-rwxrwxr-x 1 xxxx xxxx 80 dec. 25 21:09 q0
-rwxrwxr-x 1 xxxx xxxx 80 dec. 25 21:10 q1
-rwxrwxr-x 1 xxxx xxxx 80 dec. 25 21:10 q2
-rwxrwxr-x 1 xxxx xxxx 80 dec. 25 21:15 q3
-rwxrwxr-x 1 xxxx xxxx 80 dec. 25 21:16 q4
But as you say, increasing the ceiling value requires super user rights. It could be possible to create a "setuid helper" which increases the ceiling value. For example, the following program sets the ceiling value passed as parameter:
#include <errno.h>
#include <fcntl.h>
#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>
#include <string.h>
int main(int argc, char *argv[])
{
int fd;
int rc;
int msg_max;
if (argc != 2) {
fprintf(stderr, "Usage: %s <msg_max>\n", argv[0]);
return 1;
}
fd = open("/proc/sys/fs/mqueue/msg_max", O_RDWR);
if (fd < 0) {
perror("open()");
return 1;
}
rc = write(fd, argv[1], strlen(argv[1]));
if (rc != strlen(argv[1])) {
if (rc >= 0) {
errno = EIO;
}
perror("write()");
return 1;
}
close(fd);
return 0;
}
We can build it, change its owner/group to root and add the setuid bit:
$ gcc mq_helper.c -o mq_helper
$ sudo chown root mq_helper
$ sudo chgrp root mq_helper
$ sudo chmod 4555 mq_helper
$ ls -l mq_helper
-r-sr-xr-x 1 root root 17016 dec. 25 21:45 mq_helper
Then, it is possible to run this program from a non super user account to change the ceiling value of msg_max:
$ cat /proc/sys/fs/mqueue/msg_max
256
$ ./mq_helper 98
$ cat /proc/sys/fs/mqueue/msg_max
98
I've got a problem accessing files programmatically on an Amazon EFS filesystem : the same C program will work with a 64 bits executable, but not with a 32bits executable.
Mount point is /EFS, the program below will list both EFS directory (/EFS/data) and local directory (/home/centos) :
Here is the test program :
#include <stdio.h>
#include <dirent.h>
int main(void) {
struct dirent *d;
char efs_dir[] = "/EFS/data";
char local_dir[] = "/home/centos";
printf("EFS dir : %s\n",efs_dir);
DIR *dirEFS = opendir(efs_dir);
if (dirEFS == NULL) {
printf("Could not open current directory");
return 1;
}
while ((d = readdir(dirEFS)) != NULL)
printf("%s\n", d->d_name);
closedir(dirEFS);
printf("\nlocal filesystem dir : %s\n",local_dir);
DIR *dirLOCAL = opendir(local_dir);
if (dirLOCAL == NULL) {
printf("Could not open current directory");
return 1;
}
while ((d = readdir(dirLOCAL)) != NULL)
printf("%s\n", d->d_name);
closedir(dirLOCAL);
return 0;
}
Unix view :
[centos#ec2-instance ~]$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/nvme0n1p1 20G 3.4G 16G 19% /
tmpfs 3.7G 0 3.7G 0% /dev/shm
/dev/nvme1n1p1 94G 6.1G 83G 7% /opt/data
fs-XXXXXXXX.eu-west-3.amazonaws.com:/
8.0E 5.3G 8.0E 1% /EFS
[centos#ec2-instance ~]$]$ ls -l /EFS/data
-rw-r--r-- 1 centos centos 2796 Sep 15 12:18 efs_file1.txt
-rw-r--r-- 1 centos centos 2812 Sep 15 12:18 efs_file2.txt
-rw-r--r-- 1 centos centos 3625 Sep 15 12:18 efs_file3.txt
-rw-r--r-- 1 centos centos 2768 Sep 15 12:18 efs_file4.txt
[centos#ec2-instance ~]$]$ ls -l /home/centos
-rw-rw-r-- 1 centos centos 2796 Sep 15 14:15 local_file1.txt
-rw-rw-r-- 1 centos centos 2812 Sep 15 14:15 local_file2.txt
-rw-rw-r-- 1 centos centos 3625 Sep 15 14:15 local_file3.txt
-rw-rw-r-- 1 centos centos 2768 Sep 15 14:15 local_file4.txt
-rw-r--r-- 1 centos centos 792 Sep 15 14:21 test_dir.c
-rwxrwxr-x 1 centos centos 7392 Sep 15 14:22 test_dir
When I compile test_dir.c in 32 bits compatibility mode, no results for the EFS directory :
[centos#ec2-instance ~]$ gcc test_dir.c -o test_dir -m32
[centos#ec2-instance ~]$ ./test_dir
EFS dir : /EFS/data
local filesystem dir : /home/centos
test_dir
.
..
local_file1.txt
local_file2.txt
local_file3.txt
local_file4.txt
test_dir.c
But for 64 bits executable it is fine :
[centos#ec2-instance ~]$ gcc test_dir.c -o test_dir
[centos#ec2-instance ~]$ ./test_dir
EFS dir : /EFS/data
.
..
efs_file1.txt
efs_file2.txt
efs_file3.txt
efs_file4.txt
local filesystem dir : /home/centos
test_dir
.
..
local_file1.txt
local_file2.txt
local_file3.txt
local_file4.txt
test_dir.c
Anyone has an idea of what is happening here ?
Thanks.
edit (solution) : compress the 64 bits inodes number to 32 bits with the kernel option nfs.enable_ino64=0
[root#eai ~]# cat /etc/modprobe.d/nfs.conf
options nfs enable_ino64=0
[root#eai ~]# reboot
and it's done ! Thanks DNT.
Function readdir returns an inode. In your 32bit build it expects to find 32bit inodes. But with a file system with 64bit inodes it will fail if the particular inode number cannot be expressed in 32bits.
To see the inode numbers you may want to use ls -li /EFS/data/efs_file1.txt. The leftmost number is the inode number and if it exceeds 4294967295 then it is 64bit and 32bit readdir cannot handle it.
I've written a little linux kernel module, to see, how nowadays implement kernel function hijacking.
https://pastebin.com/99YJFnaq
#include <linux/module.h>
#include <linux/kernel.h>
#include <linux/init.h>
#include <linux/slab.h>
#include <linux/string.h>
#include <linux/syscalls.h>
#include <linux/version.h>
#include <linux/unistd.h>
#include <linux/time.h>
#include <linux/preempt.h>
#include <asm/uaccess.h>
#include <asm/paravirt.h>
#include <asm-generic/bug.h>
#include <asm/segment.h>
#define BUFFER_SIZE 512
#define MODULE_NAME "hacked_read"
#define dbg( format, arg... ) do { if ( debug ) pr_info( MODULE_NAME ": %s: " format , __FUNCTION__ , ## arg ); } while ( 0 )
#define err( format, arg... ) pr_err( MODULE_NAME ": " format, ## arg )
#define info( format, arg... ) pr_info( MODULE_NAME ": " format, ## arg )
#define warn( format, arg... ) pr_warn( MODULE_NAME ": " format, ## arg )
MODULE_DESCRIPTION( MODULE_NAME );
MODULE_VERSION( "0.1" );
MODULE_LICENSE( "GPL" );
MODULE_AUTHOR( "module author <mail#domain.com>" );
static char debug_buffer[ BUFFER_SIZE ];
unsigned long ( *original_read ) ( unsigned int, char *, size_t );
void **sct;
unsigned long icounter = 0;
static inline void rw_enable( void ) {
asm volatile ( "cli \n"
"pushq %rax \n"
"movq %cr0, %rax \n"
"andq $0xfffffffffffeffff, %rax \n"
"movq %rax, %cr0 \n"
"popq %rax " );
}
static inline uint64_t getcr0(void) {
register uint64_t ret = 0;
asm volatile (
"movq %%cr0, %0\n"
:"=r"(ret)
);
return ret;
}
static inline void rw_disable( register uint64_t val ) {
asm volatile(
"movq %0, %%cr0\n"
"sti "
:
:"r"(val)
);
}
static void* find_sym( const char *sym ) {
static unsigned long faddr = 0; // static !!!
// ----------- nested functions are a GCC extension ---------
int symb_fn( void* data, const char* sym, struct module* mod, unsigned long addr ) {
if( 0 == strcmp( (char*)data, sym ) ) {
faddr = addr;
return 1;
} else return 0;
};// --------------------------------------------------------
kallsyms_on_each_symbol( symb_fn, (void*)sym );
return (void*)faddr;
}
unsigned long hacked_read_test( unsigned int fd, char *buf, size_t count ) {
unsigned long r = 1;
if ( fd != 0 ) { // fd == 0 --> stdin (sh, sshd)
return original_read( fd, buf, count );
} else {
icounter++;
if ( icounter % 1000 == 0 ) {
info( "test2 icounter = %ld\n", icounter );
info( "strlen( debug_buffer ) = %ld\n", strlen( debug_buffer ) );
}
r = original_read( fd, buf, count );
strncat( debug_buffer, buf, 1 );
if ( strlen( debug_buffer ) > BUFFER_SIZE - 100 )
debug_buffer[0] = '\0';
return r;
}
}
int hacked_read_init( void ) {
register uint64_t cr0;
info( "Module was loaded\n" );
sct = find_sym( "sys_call_table" );
original_read = (void *)sct[ __NR_read ];
cr0 = getcr0();
rw_enable();
sct[ __NR_read ] = hacked_read_test;
rw_disable( cr0 );
return 0;
}
void hacked_read_exit( void ) {
register uint64_t cr0;
info( "Module was unloaded\n" );
cr0 = getcr0();
rw_enable();
sct[ __NR_read ] = original_read;
rw_disable( cr0 );
}
module_init( hacked_read_init );
module_exit( hacked_read_exit );
Makefile:
CURRENT = $(shell uname -r)
KDIR = /lib/modules/$(CURRENT)/build
PWD = $(shell pwd)
TARGET = hacked_read
obj-m := $(TARGET).o
default:
$(MAKE) -C $(KDIR) M=$(PWD) modules
clean:
#rm -f *.o .*.cmd .*.flags *.mod.c *.order
#rm -f .*.*.cmd *.symvers *~ *.*~ TODO.*
#rm -fR .tmp*
#rm -rf .tmp_versions
Thereafter, I'm making the module and inserting it.
Of-course, the better way is to do it - inside qemu machine. I'm using default Kali 2018.1 installed on image hdd.qcow2 [30Gb]. Kernel 4.14.13 is a default kernel built by me with DEBUG flags:
# diff /boot/config-4.14.13 /boot/config-4.14.0-kali3-amd64
3c3
< # Linux/x86_64 4.14.13 Kernel Configuration
---
> # Linux/x86 4.14.12 Kernel Configuration
7620c7620
< CONFIG_GDB_SCRIPTS=y
---
> # CONFIG_GDB_SCRIPTS is not set
7652,7655c7652
< CONFIG_DEBUG_KMEMLEAK=y
< CONFIG_DEBUG_KMEMLEAK_EARLY_LOG_SIZE=400
< CONFIG_DEBUG_KMEMLEAK_TEST=m
< # CONFIG_DEBUG_KMEMLEAK_DEFAULT_OFF is not set
---
> # CONFIG_DEBUG_KMEMLEAK is not set
CONFIG_DEBUG_KMEMLEAK - is useless on amd64, so there is only CONFIG_GDB_SCRIPTS plays a role.
Back to the game:
# make
# cp hacked_read.ko /lib/modules/4.14.13/hacked_read.ko
# depmod
# modprobe hacked_read
Thereafter, I'm typing different symbols, mostly a and left arrow and delete, as you can see from syslog: icounter = 44000, so it is 44k symbols was typed by me, before bug appears, sometimes more, sometimes less... To get this number faster I'm using /usr/bin/xset r rate 20 60,
or even insert false in if/else statement like this if ( fd != 0 && false ) { // fd == 0 --> stdin (sh, sshd) - this will automate the process.
The Bug
/var/log/syslog/
Aug 30 10:20:37 kali kernel: [ 1540.483650] hacked_read: test2 icounter = 44000
Aug 30 10:20:37 kali kernel: [ 1540.483654] hacked_read: strlen( debug_buffer ) = 202
Aug 30 10:20:42 kali kernel: [ 1546.187954] hacked_read: test2 icounter = 45000
Aug 30 10:20:42 kali kernel: [ 1546.187958] hacked_read: strlen( debug_buffer ) = 376
Aug 30 10:20:58 kali kernel: [ 1561.366421] BUG: unable to handle kernel paging request at ffffffffc071909b
Aug 30 10:20:58 kali kernel: [ 1561.366434] IP: 0xffffffffc071909b
Aug 30 10:20:58 kali kernel: [ 1561.366436] PGD b3a0e067 P4D b3a0e067 PUD b3a10067 PMD 2346c4067 PTE 0
Aug 30 10:20:58 kali kernel: [ 1561.366441] Oops: 0010 [#1] SMP PTI
Aug 30 10:20:58 kali kernel: [ 1561.366443] Modules linked in: hacked_read(O) 9p fscache fuse ppdev bochs_drm sg ttm 9pnet_virtio evdev joydev drm_kms_helper pcspkr serio_raw 9pnet drm parport_pc parport button binfmt_misc ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 crc32c_generic fscrypto ecb sr_mod cdrom sd_mod ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc ata_piix libata scsi_mod aesni_intel aes_x86_64 crypto_simd glue_helper cryptd psmouse floppy virtio_pci virtio_ring virtio e1000 i2c_piix4 [last unloaded: hacked_read]
Aug 30 10:20:58 kali kernel: [ 1561.366488] CPU: 0 PID: 1788 Comm: tee Tainted: G O 4.14.13 #1
Aug 30 10:20:58 kali kernel: [ 1561.366490] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1 04/01/2014
Aug 30 10:20:58 kali kernel: [ 1561.366491] task: ffff9939ac178000 task.stack: ffffb2570359c000
Aug 30 10:20:58 kali kernel: [ 1561.366493] RIP: 0010:0xffffffffc071909b
Aug 30 10:20:58 kali kernel: [ 1561.366494] RSP: 0018:ffffb2570359ff38 EFLAGS: 00010292
Aug 30 10:20:58 kali kernel: [ 1561.366496] RAX: 000000000000005e RBX: 00007ffe554f8940 RCX: 0000000000000000
Aug 30 10:20:58 kali kernel: [ 1561.366497] RDX: 0000000000000000 RSI: ffff9939a0af7c10 RDI: ffff9939c0a20bb8
Aug 30 10:20:58 kali kernel: [ 1561.366498] RBP: 0000000000002000 R08: 0000000000000000 R09: 0000000000000000
Aug 30 10:20:58 kali kernel: [ 1561.366499] R10: 000000000000005e R11: 00000000000003f1 R12: ffffffffc071b360
Aug 30 10:20:58 kali kernel: [ 1561.366501] R13: 000055ae361bb4a0 R14: 0000000000000010 R15: 00007ffe554faa98
Aug 30 10:20:58 kali kernel: [ 1561.366502] FS: 00007f60491184c0(0000) GS:ffff9939ffc00000(0000) knlGS:0000000000000000
Aug 30 10:20:58 kali kernel: [ 1561.366504] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug 30 10:20:58 kali kernel: [ 1561.366505] CR2: ffffffffc071909b CR3: 00000001d9018005 CR4: 00000000000606f0
Aug 30 10:20:58 kali kernel: [ 1561.366514] Call Trace:
Aug 30 10:20:58 kali kernel: [ 1561.366524] ? system_call_fast_compare_end+0xc/0x6f
Aug 30 10:20:58 kali kernel: [ 1561.366526] Code: Bad RIP value.
Aug 30 10:20:58 kali kernel: [ 1561.366532] RIP: 0xffffffffc071909b RSP: ffffb2570359ff38
Aug 30 10:20:58 kali kernel: [ 1561.366532] CR2: ffffffffc071909b
Aug 30 10:20:58 kali kernel: [ 1561.366535] ---[ end trace ca74de96d373ac0b ]---
Could somebody, please, tell me which way to dig?
There is no overflows inside debug_buffer array - it is completely true.
There is no conflicts in asm code, while hijacking is carried out.
It is tiny, light script... Where is the BUG?
Update1:
Looks like I've found a reason why it starts crashing. The BUG appears right after command rmmod hacked_read. So module_exit() is wrong, probably asm's cli & sti not enough.
As the module is removed from the Linux kernel, all memory used by the module (data and code) is released. The exit() function of the module restores the pointer to the original function. However, the kernel may be executing the code of the substitute function at the time the module is removed. Suddenly, right in the middle of that the function disappears as the memory taken by the module's code is released. Hence the bug.
Obviously you can't remove the module after you restore the pointer to the original function until you're sure that there are no kernel threads that (may) execute the code of the substitute function. After the pointer is restored, all new kernel threads will execute the original function, so you need to wait until any current threads finish the execution of the substitute function. How to do that is another issue. You may need to employ some tricks like reference counters, etc.
As #Aleksey mentioned, the issue was outside of the module.
The tee command used read() in its sleeping manner. While I've removed module nothing happened, but there was my little bash script:
#!/bin/bash
logfile="micro-test.log"
while sleep 0;do
echo -n "$(date): $(uptime): "
echo "1 2" | awk '{print $1}'
sleep 60;
done | tee -a $logfile
How I've found the piece of BUG:
As I said, my guest's kernel was compiled with CONFIG_GDB_SCRIPTS=y. Now I'm attaching guest from the host's gdb:
# gdb
(gdb) set logging file gdbcmd2.out
(gdb) set logging on
Copying output to gdbcmd2.out.
(gdb)
Already logging to gdbcmd2.out.
(gdb) target remote :1234
Remote debugging using :1234
warning: No executable has been specified and target does not support
determining executable automatically. Try using the "file" command.
0xffffffff99082e42 in ?? ()
(gdb) add-auto-load-safe-path /usr/src/linux-source-4.14/scripts/gdb/vmlinux-gdb.py
(gdb) file /usr/src/linux-source-4.14/vmlinux
A program is being debugged already.
Are you sure you want to change the file? (y or n) y
Reading symbols from /usr/src/linux-source-4.14/vmlinux...done
On the guest side, I'm extracting addresses:
root#kali:~# cat /sys/module/hacked_read/sections/.text
0xffffffffc06e9000
root#kali:~# cat /sys/module/hacked_read/sections/.bss
0xffffffffc06eb34
On the host side, adding module to debugging:
(gdb) add-symbol-file /usr/src/hacked_read/hacked_read.ko 0xffffffffc06e9000 -s .bss 0xffffffffc06eb34
(gdb) p hacked_read_test
$1 = {unsigned long (unsigned int, char *, size_t)} 0xffffffffc06e9030 <hacked_read_test>
(gdb) maintenance info line-table
... BIG-BIG-OUT-PUT ...
Thereafter, I can see in logfile: gdbcmd2.out - listing of my code with addresses. For example, 0xffffffffc06e9030 - the address of hacked_read_test function:
# grep 0xffffffffc06e9030 gdbcmd2.out
$1 = {unsigned long (unsigned int, char *, size_t)} 0xffffffffc06e9030 <hacked_read_test>
6 77 0xffffffffc06e9030
77 - line of code
$ head -n 77 hacked_read.c | tail -n 1
unsigned long hacked_read_test( unsigned int fd, char *buf, size_t count ) {
Bingo!
Now, on the guest side, I'm doing rmmod hacked_read. After 60+- seconds appears BUG:
Sep 9 06:35:28 kali kernel: [281996.592759] hacked_read: Module was unloaded
Sep 9 06:36:11 kali kernel: [282040.218523] BUG: unable to handle kernel paging request at ffffffffc06e909b
Sep 9 06:36:11 kali kernel: [282040.218530] IP: 0xffffffffc06e909b
Sep 9 06:36:11 kali kernel: [282040.218531] PGD 22980e067 P4D 22980e067 PUD 229810067 PMD 2356e3067 PTE 0
Sep 9 06:36:11 kali kernel: [282040.218534] Oops: 0010 [#9] SMP PTI
Sep 9 06:36:11 kali kernel: [282040.218536] Modules linked in: sctp_diag sctp libcrc32c tcp_diag udp_diag dccp_diag dccp inet_diag unix_diag 9p fscache fuse bochs_drm ttm ppdev drm_kms_helper joydev evdev serio_raw pcspkr sg 9pnet_virtio 9pnet parport_pc parport button drm binfmt_misc ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 crc32c_generic fscrypto ecb sr_mod cdrom sd_mod ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd glue_helper cryptd ata_piix psmouse floppy virtio_pci virtio_ring virtio e1000 i2c_piix4 libata scsi_mod [last unloaded: hacked_read]
Sep 9 06:36:11 kali kernel: [282040.218567] CPU: 0 PID: 32196 Comm: tee Tainted: G D O 4.14.13 #1
Comm: tee & BUG: unable to handle kernel paging request at ffffffffc06e909b
Host:
# grep ffffffffc06e909b gdbcmd2.out
18 88 0xffffffffc06e909b
88 - line of code:
$ head -n 88 hacked_read.c | tail -n 1
strncat( debug_buffer, buf, 1 );
It is easy to see, that kernel unable to give tee the address of the next line after original_read():
77:unsigned long hacked_read_test( unsigned int fd, char *buf, size_t count ) {
78. unsigned long r = 1;
79. if ( fd != 0 ) { // fd == 0 --> stdin (sh, sshd)
80. return original_read( fd, buf, count );
81. } else {
82. icounter++;
83. if ( icounter % 1000 == 0 ) {
84. info( "test2 icounter = %ld\n", icounter );
85. info( "strlen( debug_buffer ) = %ld\n", strlen( debug_buffer ) );
86. }
87. r = original_read( fd, buf, count );
88. strncat( debug_buffer, buf, 1 );
if ( strlen( debug_buffer ) > BUFFER_SIZE - 100 )
debug_buffer[0] = '\0';
return r;
}
}
I am on ubuntu linux 10.04
I have the following
#simplec.c
#include "stdio.h"
int main()
{
printf("Hello World\n");
system("ps -H");
return 12;
}
AND
#callsimplec.c
#include "stdio.h"
int main()
{
int ret =0;
ret = system("./simplec");
printf("In callsimplec ret is %d\n", ret);
}
When I do
gcc callsimplec.c -o callsimplec
gcc simplec.c -o simplec
./callsimplec
I get:
Hello World
PID TTY TIME CMD
27238 pts/2 00:00:00 bash
28066 pts/2 00:00:00 callsimplec
28067 pts/2 00:00:00 simplec
28068 pts/2 00:00:00 ps
In callsimplec ret is 3072
So I figured out that 3072 is printed because 256 times 12 is 3072. Whatever return value I use in simplec.c I get that value multiplied by 256 as the output in print. Why is that? I am just trying to make sense of it.
The value returned by system should be used with the macros: WEXITED, WIFEXITSTATUS etc.
The value returned by system (and by the wait family) is, acording to case:
Normal termination:
15 ............. 8 ............. 0
exit status 0x00
Killed by signal:
15 ............. 8 7............ 0
unused killer signal
Stopped by signal
15 ............. 8 ............. 0
stop signal 0x7f
Continued by signal
15 ............................. 0
0xFFF
So in your case the process exited normally and system returned 12 shifted 8 times.