Understanding per cpu structure in linux kernel - c

I am studying linux kernel and sometimes I don't understand what kernel developers want in a particular piece of code. So I was reading through timers in kernel and a timer is created using a struct timer_list variable, that contains a per cpu pointer. I tried to understand a little better this per cpu variable so I was looking in linux kerenl, how things are getting created.
So I have taken different structures from kernel and listed the #defines to integrate things and see a clear picture, what actually is happening.
structures from all this started
struct timer_list {
/*
* All fields that change during normal runtime grouped to the
* same cacheline
*/
struct list_head entry;
unsigned long expires;
struct tvec_base *base;//pointer to per cpu variable, so I checked what is inside
void (*function)(unsigned long);
unsigned long data;
int slack;
#ifdef CONFIG_TIMER_STATS
int start_pid;
void *start_site;
char start_comm[16];
#endif
#ifdef CONFIG_LOCKDEP
struct lockdep_map lockdep_map;
#endif
};
the base pointer is a struct like this
struct tvec_base {
spinlock_t lock;
struct timer_list *running_timer;
unsigned long timer_jiffies;
unsigned long next_timer;
unsigned long active_timers;
struct tvec_root tv1;
struct tvec tv2;
struct tvec tv3;
struct tvec tv4;
struct tvec tv5;
} ____cacheline_aligned;//??why such a name __cacheline_aligned
struct tvec_base boot_tvec_bases;
EXPORT_SYMBOL(boot_tvec_bases);
static DEFINE_PER_CPU(struct tvec_base *, tvec_bases) = &boot_tvec_bases;//from here I am a little puzzeled as the way things are written been assigned.
DEFINE PER CPU is such a simple #define to understand, wish it was
#define DEFINE_PER_CPU(type, name) \
DEFINE_PER_CPU_SECTION(type, name, "")
#define DEFINE_PER_CPU_SECTION(type, name, sec) \//?? what exactly we achieve here
__PCPU_DUMMY_ATTRS char __pcpu_scope_##name; \
extern __PCPU_DUMMY_ATTRS char __pcpu_unique_##name; \
__PCPU_DUMMY_ATTRS char __pcpu_unique_##name; \
__PCPU_ATTRS(sec) PER_CPU_DEF_ATTRIBUTES __weak \
__typeof__(type) name
#define __PCPU_DUMMY_ATTRS \
__attribute__((section(".discard"), unused))//i think it is a section in map file, but already kernel is built and I am builing a timer module, so what does it do?
If anyone have good experience in linux internal can you just point me in the right direction.
Some specific questions which if answered can make me understand whole thing,
`static DEFINE_PER_CPU(struct tvec_base *, tvec_bases) = &boot_tvec_bases;
1)what does this mean? this address '&boot_tvec_bases' is going where?
2) why a name ____cacheline_aligned;
choosen. Does it do anything special?
3)what is
#define DEFINE_PER_CPU_SECTION(type, name, sec)
sec here?

timer.h:36: extern struct tvec_base boot_tvec_bases;
See ____cacheline_aligned_in_smp for structure in linux
____cacheline_aligned instructs the compiler to instantiate a struct or variable at an address corresponding to the beginning of an L1
cache line, for the specific architecture, i.e., so that it is L1
cache-line aligned. ____cacheline_aligned_in_smp is similar, but is
actually L1 cache-line aligned only when the kernel is compiled in SMP
configuration (i.e., with option CONFIG_SMP). These are defined in
file include/linux/cache.h
3.http://www.makelinux.net/ldd3/chp-8-sect-5
Per-CPU variables are an interesting 2.6 kernel feature. When you
create a per-CPU variable, each processor on the system gets its own
copy of that variable. This may seem like a strange thing to want to
do, but it has its advantages. Access to per-CPU variables requires
(almost) no locking, because each processor works with its own copy.

Related

Forward declaration issue, two compilers

I've been developing in C using eclipse as my IDE in my virtual machine with ubuntu, I've made some progress and I wanted to test them in the real product which is an embedded system using powerpc.
In order to compile that program for our product I use Code::Blocks in Windows but the compiler is a powerpc version of the gcc.
The same code is giving me an error in the powerpc version that doesn't appear in the ubuntu version.
I have two header files gral.h and module_hand.h as follows:
The gral.h file:
#ifndef HEADERS_GRAL_H_
#define HEADERS_GRAL_H_
#include "module_hand.h"
typedef struct PROFILE
{
module_t mod; // this one comes from module_hand.h
int var1; // some other random variables
} profile_t;
#endif /* HEADERS_GRAL_H_ */
The module_hand.h is defined as follows
#ifndef HEADERS_MODULE_HAND_H_
#define HEADERS_MODULE_HAND_H_
#include <stdint.h>
#include "gral.h"
typedef struct PROFILE profile_t;
typedef struct module
{
char name[30]; // name of module
char rev[30]; // module revision
char mfr[30]; // manufacturer
} module_t;
int Mod_Init(profile_t *profile);
/* some other random functions */
#endif /* HEADERS_MODULE_HAND_H_*/
As you'll see, I don't use the PROFILE struct in the module struct, But I declare it forward to use it in the declaration of the Mod_Init function
This gives me a Error: redefinition of typedef 'profile_t'
and error: previous declaration of 'profile_t' was here
If I remove the forward declaration the error is Error: parse error before '*' token
where the line number is the line of the function declaration.
My doubt is what am I missing, and why gcc in Ubuntu does compile it with no problem.
Your powerpc compiler is enforcing the C99 rule that
If an identifier has no linkage, there shall be no more than one
declaration of the identifier (in a declarator or type specifier) with
the same scope and in the same name space, except for tags as
specified in 6.7.2.3.
(C99 6.7/3)
Your Linux compiler is observing the relaxed version of that rule that was introduced in C11:
If an identifier has no linkage, there shall be no more than one
declaration of the identifier (in a declarator or type specifier) with
the same scope and in the same name space, except that:
a typedef name may be redefined to denote the same type as it currently does, provided that type is not a variably modified type;
tags may be redeclared as specified in 6.7.2.3.
(C11 6.7/3; also C17 6.7/3)
Supposing that the compilation options are the same, the behavior difference surely arises from using different versions of GCC. More recent versions default to more recent versions of the language.
You could try adding -std=gnu11 or -std=c11 to the command-line options (for both targets) to try to get consistency. If your powerpc version of GCC is too old to accept those then you really need to update to a newer version.
Note also, however, that you don't need to have this problem in the first place. Given that module_hand.h includes gral.h, the former has no need whatever to redefine a typedef that the latter already defines.
Moreover, the fact that these two headers each include the other is a strong suggestion that they ought to be combined into one. Multiple-inclusion guards prevent an actual loop, but they are not an adequate solution.
In the gral.h header file, you define profile_t using typedef, then you redefine profile_t with another typedef in module_hand.h. You should just define the struct PROFILE in gral_h and include gral.h in module_hand.h.
gral.h:
#ifndef HEADERS_GRAL_H_
#define HEADERS_GRAL_H_
#include "module_hand.h"
typedef struct PROFILE {
module_t mod; // this one comes from module_hand.h
int var1; // some other random variables
} profile_t;
#endif /* HEADERS_GRAL_H_ */:
module_hand.h:
#ifndef HEADERS_MODULE_HAND_H_
#define HEADERS_MODULE_HAND_H_
#include <stdint.h>
typedef struct module
{
char name[30]; // name of module
char rev[30]; // module revision
char mfr[30]; // manufacturer
} module_t;
int Mod_Init(struct PROFILE *profile);
/* some other random functions */
#endif /* HEADERS_MODULE_HAND_H_*/
Well I read your answers and comments and decided to try another approach.
As some of you said, I had some kind of recursion, I wanted to keep every structure within its respective header file, but now, I dropped the idea and merged the structures in one file only.
My new approach:
Gral.h
#ifndef HEADERS_GRAL_H_
#define HEADERS_GRAL_H_
typedef struct module
{
char name[30]; // name of module
char rev[30]; // module revision
char mfr[30]; // manufacturer
} module_t;
typedef struct PROFILE {
module_t mod; // this one comes from module_hand.h
int var1; // some other variables
} profile_t;
#endif /* HEADERS_GRAL_H_ */:
Module.h
#ifndef HEADERS_MODULE_HAND_H_
#define HEADERS_MODULE_HAND_H_
#include <Gral.h>
int Mod_Init(profile_t *profile);
/* some other functions */
#endif /* HEADERS_MODULE_HAND_H_*/
And when any other structure comes up, I'll declare it in Gral.h and include the header file.
Regarding the compilers, they aren't the same version. The powerpc is quite old now. That would explain the powerpc compilation errors.
Thank you again.

Array members defined as constants

I'm trying to implement a framework, where I would need to declare (in .h file) the list of available "drivers" (struct variables) which would be defined in specific .c modules. As the list would probably grow in the future I would like to have it all in one place in the .h file to keep it easily extensible.
E.g. let's have "driver.h"
typedef struct driver {
int id;
char name[10];
int(*init)();
void (*deinit)();
int (*doTheJob)(int);
} driver_t;
#define DRIVERLIST driver1, driver2, driver3
#define DRIVERS extern driver_t DRIVERLIST;
DRIVERS
Then the specific drivers (driver1, driver2, driver3) would be defined in dedicated modules.. e.g. driver1.c, driver2.c .. etc...
But then I would like to have a module e.g. manager.c where I would like to define the array of available drivers as declared in driver.h so that I'm able to iterate the array and get the drivers for usage in other parts of the framework..
So in manager.c I would need something like:
driver_t drivers[MAX_DRIVERS] = {DRIVERS}
But obviously it does not compile this way..
The main idea is to edit only driver.h when I need to add declaration for additional driver in the future and then just implement it in dedicated module, whithout the necessity to edit e.g. manager.c or other parts of the framework..
Do you have any idea, how to implement such mechanism in c?
In C you can't initialize an array with copies of some objects (in C++ can but it is not good practice because they are copies and will be changed independently with original objects).
drivers array should contain pointers to original objects. I suggest something like
/* driver.h */
typedef struct driver {
int id;
char name[10];
int(*init)();
void (*deinit)();
int (*doTheJob)(int);
} driver_t;
#define MAX_DRIVERS 10
#define DRIVERLIST driver1, driver2, driver3
#define DRIVERS_INIT {&driver1, &driver2, &driver3}
#define DRIVERS extern driver_t DRIVERLIST;
DRIVERS
/* manager.c */
#include "driver.h"
/* ... */
driver_t * drivers[MAX_DRIVERS] = DRIVERS_INIT;
Manager code will use drivers[i]->id instead of drivers[i].id.
The proper way to do this in C is to immediately get rid of all extern-spaghetti with globals.
Instead you could put your struct definition inside driver.h and in driver.c initialize it through a "constructor":
// driver.c
#include "driver.h"
#include "specific_driver_x.h"
void driver_init (driver_t* driver)
{
driver->init = specific_driver_init;
driver->doTheJob = specific_driver_job;
}
For professional code, this can be further improved with the concept of "opaque type" as explained here, to achieve private encapsulation (and if needed polymorphism). In which case the struct definition can be (partially) hidden in driver.c and the constructor also handles memory allocation.
I think I found a solution. I took the inspiration from the rtl_433 project https://github.com/merbanan/rtl_433/blob/master/include/rtl_433_devices.h where they defined something similar for the devices declarations.
So it should be in header file:
/* driver.h */
#define DRIVERS \
DECL(driver1) \
DECL(driver2)
#define DECL(name) extern driver_t name;
DRIVERS
#undef DECL
And then in module:
/* driver.c */
driver_t* drivers[] = {
#define DECL(name) &name,
DRIVERS
#undef DECL
};

Compile the same C code for user space as well as kernel space

lets say i have a .h and .c file, originally written for kernel space.
#ifndef __TREE_H__
#define __TREE_H__
/* #includes for kernel headers*/
struct tree_node_t{
int data;
struct tree_node_t *left;
struct tree_node_t *right;
};
struct tree{
int data;
struct tree_node_t *root;
};
void insert_node(struct tree *tree, int data);
#endif
Lets say, the corresponding .c file contains the definition of void insert_node fn, which create a tree_node using kmalloc() and insert it into a tree. I am able to compile the code using Makefile to compile the kernel modules as the .h/.c file are part of kernel module.
Now, i want to use the same data structure and its fns in userspace as well. I need to compile it using new user space specific makefile which takes care to replace the kernel specific calls like kmalloc and kfree with malloc and free instead ? How can i use various combination of #ifdef #else #endif so that the files are compilable in user space and kernel space by using respective make files.
pls Nte : kmalloc/kfree are just simple examples to explain. Files contain other kernel specific APIs which needs to be excluded/replaced at the time compilation of file in user space.
You may use __KERNEL__ macro for differentiate target of compilation: kernel code (macro is defined) or application (macro is not defined).
Exactly this macro is used in Linux kernel for uapi headers and some other files. See, e.g., include/uapi/linux/ethtool.h.

Create dispatch table registering functions across multiple source files in C

How can I implement a dynamic dispatch table in C
It's essentially the same question as the linked issue, so ...
As your Strategy.c obviously already knows about the strategy
instances by name (#include "XYstrategy.h") you could go the whole
mile and use the header files instead of the implementation files to
communicate your strategy to the central dispatcher:
This is contrary to the clear intent in the question. This was an example of how he could do it statically, but wanted to have modules dynamically register themselves at compile time.
Let me try providing an example I'm struggling with for my own purposes...
I have a micro-controller which I want to use to read a variety of sensors that report temperature and/or humidity. I have a central core program which takes care of formatting the returned data and submitting it to a web server where it is recorded in an RRD.
Rather than build a large monolithic program which contains all the different functions for each sensor type, I want to be able to build a specific subset into the software loaded onto the micro-controller which corresponds to the sensors installed on that particular controller.
To do this I would like to be able to write a generic driver for each sensor that has three functions:
bool _sensor_startup();
bool _read_sensor(float *temp, float *humidity, uint8_t max_count, uint8_t *count);
bool _sensor_shutdown();
The sensor_startup function will take care of powering up the sensors, making sure that they are properly configured and in a state of readiness for read_sensor to be called. If this process fails for any reason, it returns false, otherwise, it returns true.
The read_sensor function will cause up to max_count sensors to be read with their reults stored in the arrays pointed to by temp and humidity, respectively. The number of sensors read will be stored in count.
The sensor_shutdown function will do any housekeeping necessary to return the sensors and supporting electronics into their lowest power consumption configuration.
Each of these is contained in a separate .c file which may have a corresponding .h file to define relevant constants, call relevant libraries, etc.
I'd like to have a master Sensor.h file which is included by the .c or .h files and which defines:
typedef struct { startup_func, read_func, shutdown_func } sensor_driver_entry;
extern sensor_driver_entry sensor_table[];
Then I'd like each Driver file to be able to use a macro (or a function) to register the type-specific functions in the next open slot in sensor_table at compile time.
I'd like sensor table to be declared in the global namespace of Sensor.c as:
sensor_driver_entry sensor_table[MAX_SENSOR_TYPES];
(MAX_SENSOR_TYPES would be defined in Sensor.h reflecting the maximum possible number of drivers that could be selected).
Is this even possible? If so, can someone provide a syntactic example? In this specific case, I'm coding in the Particle Dev environment for a Particle Photon, but I'd like it if I could make the code also portable to the Arduino IDE to use it with ESP8266 boards as well.
One possibility is to make use of constructors. Below is a simple example with two drivers registering their functions respectively.
If the application is compiled with both drivers (gcc main.c driver1.c driver2.c) the output shows both driver functions registered:
driver1_init
driver2_init
driver1_func
driver2_func
If only the first driver is compiled in (gcc main.c driver1.c) the output shows only that driver's function registered:
driver1_init
driver1_func
driver.h
typedef void (*driver_func_t)(void);
typedef struct { driver_func_t func; } driver_entry_t;
#define MAX_TYPES 10
extern driver_entry_t driver_table[MAX_TYPES];
extern unsigned int num_driver_entries;
main.c
#include <stdio.h>
#include "driver.h"
driver_entry_t driver_table[MAX_TYPES];
unsigned int num_driver_entries;
int main (void)
{
unsigned int ix;
for (ix = 0; ix < num_driver_entries; ix++) {
driver_table[ix].func();
}
return 0;
}
driver1.c
#include <stdio.h>
#include "driver.h"
void driver1_func (void)
{
printf("%s\n", __FUNCTION__);
}
void driver1_init (void) __attribute__ ((constructor));
void driver1_init (void)
{
printf("%s\n", __FUNCTION__);
driver_table[num_driver_entries++].func = driver1_func;
}
driver2.c
#include <stdio.h>
#include "driver.h"
void driver2_func (void)
{
printf("%s\n", __FUNCTION__);
}
void driver2_init (void) __attribute__ ((constructor));
void driver2_init (void)
{
printf("%s\n", __FUNCTION__);
driver_table[num_driver_entries++].func = driver2_func;
}

cl_float4 on host and float4 on device in openCL

So I want to define a struct thats usable on both the host and on the device in openCL that makes use of the built in openCL float4 datatypes.
On the host side, the framework comes with a cl_float4 type but on the device it's just float4.
So if I create a struct like this...
typedef struct
{
cl_float4 a, b;
} MyStruct;
...and then try to pass that struct into a kernel (via a buffer) I get an error.
If I declare it as follows...
typedef struct
{
float4 a,b;
}
...that would work on the device but doesn't work on the host.
So is there a way to get it so that I can make use of openCLs built in vector types on both sides of my program within the same structs?
The C preprocessor can help you here, by treating the code differently depending on whether it is being compiled on the host or the device.
Here's some possible solutions:
typedef struct
{
#ifdef __OPENCL_C_VERSION__
float4
#else
cl_float4
#endif
a, b;
} MyStruct;
or:
#ifdef __OPENCL_C_VERSION__
typedef float4 cl_float4;
#endif
typedef struct
{
cl_float4 a, b;
} MyStruct;
or:
#ifndef __OPENCL_C_VERSION__
typedef cl_float4 float4;
#endif
typedef struct
{
float4 a, b;
} MyStruct;
or just use cl_float4, and compile the OpenCL code like this:
clBuildProgram(program, 1, &device, "-Dcl_float4=float4", NULL, NULL);
I'd avoid this approach unless you're very careful with your struct definition. The data structure alignment rules of your OpenCL device's architecture may very well be different than those of your OpenCL host's architecture. See this Wikipedia article for an overview on data structure alignment.
TLDR: the size of your struct may vary from device to host, as might the offsets from of each struct's member from the beginning of the struct. If this happens your program will break. Even if you get away with this on your current host/device combination it's not guaranteed to work on other hardware combinations.

Resources