I have a system that needs at least 10 mseconds of accuracy for timers.
I went for timerfd as it suits me perfectly, but found that even for times up to 15 milliseconds it is not accurate at all, either that or I don't understand how it works.
The times I have measured were up to 21 mseconds on a 10 mseconds timer.
I have put together a quick test that shows my problem.
Here a test:
#include <sys/timerfd.h>
#include <time.h>
#include <string.h>
#include <stdint.h>
#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>
#include <inttypes.h>
int main(int argc, char *argv[]){
int timerfd = timerfd_create(CLOCK_MONOTONIC,0);
int milliseconds = atoi(argv[1]);
struct itimerspec timspec;
bzero(&timspec, sizeof(timspec));
timspec.it_interval.tv_sec = 0;
timspec.it_interval.tv_nsec = milliseconds * 1000000;
timspec.it_value.tv_sec = 0;
timspec.it_value.tv_nsec = 1;
int res = timerfd_settime(timerfd, 0, &timspec, 0);
if(res < 0){
perror("timerfd_settime:");
return 1;
}
uint64_t expirations = 0;
int iterations = 0;
while( res = read(timerfd, &expirations, sizeof(expirations))){
if(res < 0){ perror("read:"); continue; }
if(expirations > 1){
printf("%" PRIu64 " expirations, %d iterations\n", expirations, iterations);
break;
}
iterations++;
}
return 0;
}
And executed like this:
Zack ~$ for i in 2 4 8 10 15; do echo "intervals of $i milliseconds"; ./test $i;done
intervals of 2 milliseconds
2 expirations, 1 iterations
intervals of 4 milliseconds
2 expirations, 6381 iterations
intervals of 8 milliseconds
2 expirations, 21764 iterations
intervals of 10 milliseconds
2 expirations, 1089 iterations
intervals of 15 milliseconds
2 expirations, 3085 iterations
Even assuming some possible delays, 15 milliseconds delays sounds too much for me.
Try altering it as follows, this should pretty much garuntee that it'll never miss a wakeup, but be careful with it since running realtime priority can lock your machine hard if it doesn't sleep, also you may need to set things up so that your user has the ability to run stuff at realtime priority (see /etc/security/limits.conf)
#include <sys/timerfd.h>
#include <time.h>
#include <string.h>
#include <stdint.h>
#include <stdio.h>
#include <sched.h>
int main(int argc, char *argv[])
{
int timerfd = timerfd_create(CLOCK_MONOTONIC,0);
int milliseconds = atoi(argv[1]);
struct itimerspec timspec;
struct sched_param schedparm;
memset(&schedparm, 0, sizeof(schedparm));
schedparm.sched_priority = 1; // lowest rt priority
sched_setscheduler(0, SCHED_FIFO, &schedparm);
bzero(&timspec, sizeof(timspec));
timspec.it_interval.tv_sec = 0;
timspec.it_interval.tv_nsec = milliseconds * 1000000;
timspec.it_value.tv_sec = 0;
timspec.it_value.tv_nsec = 1;
int res = timerfd_settime(timerfd, 0, &timspec, 0);
if(res < 0){
perror("timerfd_settime:");
}
uint64_t expirations = 0;
int iterations = 0;
while( res = read(timerfd, &expirations, sizeof(expirations))){
if(res < 0){ perror("read:"); continue; }
if(expirations > 1){
printf("%ld expirations, %d iterations\n", expirations, iterations);
break;
}
iterations++;
}
}
If you are using threads you should use pthread_setschedparam instead of sched_setscheduler.
Realtime also isn't about low latency, it's about guarantees, RT means that if you want to wake up exactly once every second on the second, you WILL, the normal scheduling does not give you this, it might decide to wake you up 100ms later, because it had some other work to do at that time anyway. If you want to wake up every 10ms and you REALLY do need to, then you should set yourself to run as a realtime task then the kernel will wake you up every 10ms without fail. Unless a higher priority realtime task is busy doing stuff.
If you need to guarantee that your wakeup interval is exactly some time it doesn't matter if it's 1ms or 1 second, you won't get it unless you run as a realtime task. There are good reasons the kernel will do this to you (saving power is one of them, higher throughput is another, there are others), but it's well within it's rights to do so since you never told it you need better guarantees. Most stuff doesn't actually need to be this accurate, or need to never miss so you should think hard about whether or not you really do need it.
quoting from http://www.ganssle.com/articles/realtime.htm
A hard real time task or system is
one where an activity simply must be
completed - always - by a specified
deadline. The deadline may be a
particular time or time interval, or
may be the arrival of some event. Hard
real time tasks fail, by definition,
if they miss such a deadline.
Notice this definition makes no
assumptions about the frequency or
period of the tasks. A microsecond or
a week - if missing the deadline
induces failure, then the task has
hard real time requirements.
Soft realtime is pretty much the same, except that missing a deadline, while undesirable, is not the end of the world (for example video and audio playback are soft realtime tasks, you don't want to miss displaying a frame, or run out of buffer, but if you do it's just a momentary hiccough, and you simply continue). If what you are trying to do is 'soft' realtime I wouldn't bother with running at realtime priority since you should generally get your wakeups in time (or at least close to it).
EDIT:
If you aren't running realtime the kernel will by default give any timers you make some 'slack' so that it can merge your request to wake up with other events that happen at times close to the one you asked for (that is if the other event is within your 'slack' time it will not wake you at the time you asked, but a little earlier or later, at the same time it was already going to do something else, this saves power).
For a little more info see High- (but not too high-) resolution timeouts and Timer slack (note I'm not sure if either of those things is exactly what's really in the kernel since both those articles are about lkml mailing list discussions, but something like the first one really is in the kernel.
I've got a feeling that your test is very hardware dependent. When I ran your sample program on my system, it appeared to hang at 1ms. To make your test at all meaningful on my computer, I had to change from milliseconds to microseconds. (I changed the multiplier from 1_000_000 to 1_000.)
$ grep 1000 test.c
timspec.it_interval.tv_nsec = microseconds * 1000;
$ for i in 1 2 4 5 7 8 9 15 16 17\
31 32 33 47 48 49 63 64 65 ; do\
echo "intervals of $i microseconds";\
./test $i;done
intervals of 1 microseconds
11 expirations, 0 iterations
intervals of 2 microseconds
5 expirations, 0 iterations
intervals of 4 microseconds
3 expirations, 0 iterations
intervals of 5 microseconds
2 expirations, 0 iterations
intervals of 7 microseconds
2 expirations, 0 iterations
intervals of 8 microseconds
2 expirations, 0 iterations
intervals of 9 microseconds
2 expirations, 0 iterations
intervals of 15 microseconds
2 expirations, 7788 iterations
intervals of 16 microseconds
4 expirations, 1646767 iterations
intervals of 17 microseconds
2 expirations, 597 iterations
intervals of 31 microseconds
2 expirations, 370969 iterations
intervals of 32 microseconds
2 expirations, 163167 iterations
intervals of 33 microseconds
2 expirations, 3267 iterations
intervals of 47 microseconds
2 expirations, 1913584 iterations
intervals of 48 microseconds
2 expirations, 31 iterations
intervals of 49 microseconds
2 expirations, 17852 iterations
intervals of 63 microseconds
2 expirations, 24 iterations
intervals of 64 microseconds
2 expirations, 2888 iterations
intervals of 65 microseconds
2 expirations, 37668 iterations
(Somewhat interesting that I got the longest runs from 16 and 47 microseconds, but 17 and 48 were awful.)
time(7) has some suggestions on why our platforms are so different:
High-Resolution Timers
Before Linux 2.6.21, the accuracy of timer and sleep system
calls (see below) was also limited by the size of the jiffy.
Since Linux 2.6.21, Linux supports high-resolution timers
(HRTs), optionally configurable via CONFIG_HIGH_RES_TIMERS. On
a system that supports HRTs, the accuracy of sleep and timer
system calls is no longer constrained by the jiffy, but instead
can be as accurate as the hardware allows (microsecond accuracy
is typical of modern hardware). You can determine whether
high-resolution timers are supported by checking the resolution
returned by a call to clock_getres(2) or looking at the
"resolution" entries in /proc/timer_list.
HRTs are not supported on all hardware architectures. (Support
is provided on x86, arm, and powerpc, among others.)
All the 'resolution' lines in my /proc/timer_list are 1ns on my (admittedly ridiculously powerful) x86_64 system.
I decided to try to figure out where the 'breaking point' is on my computer, but gave up on the 110 microsecond run:
$ for i in 70 80 90 100 110 120 130\
; do echo "intervals of $i microseconds";\
./test $i;done
intervals of 70 microseconds
2 expirations, 639236 iterations
intervals of 80 microseconds
2 expirations, 150304 iterations
intervals of 90 microseconds
4 expirations, 3368248 iterations
intervals of 100 microseconds
4 expirations, 1964857 iterations
intervals of 110 microseconds
^C
90 microseconds ran for three million iterations before it failed a few times; that's 22 times better resolution than your very first test, so I'd say that given the right hardware, 10ms shouldn't be anywhere near difficult. (90 microseconds is 111 times better resolution than 10 milliseconds.)
But if your hardware doesn't provide the timers for high resolution timers, then Linux can't help you without resorting to SCHED_RR or SCHED_FIFO. And even then, perhaps another kernel could better provide you with the software timer support you need.
Good luck. :)
Here's a theory. If HZ is set to 250 for your system ( as is typical ) then you have a 4 millisecond timer resolution. Once your process is swapped out by the scheduler, it's likely that a number of other processes will be scheduled and run before your process gets another time slice. This might explain you seeing timer resolutions in the 15 to 21 millisecond range. The only way to get around this would be to run a real-time kernel.
The typical solution for high resolution timing on non-realtime systems is basically to busy wait with a call to select.
Depending on what else the system is doing, it may be a bit slow in switching back to your task. Unless you have a "real" realtime system, there's no guarantee it will do better than what you're seeing, although I agree that result is a bit disappointing.
You can (mostly) eliminate that task switch / scheduler time. If you have CPU power (and electrical power!) to spare, a brutal but effective solution would be a busy wait spin loop.
The idea is to run your program in a tight loop that continuously polls the clock for what time it is, and then calls your other code when the time is right. At the expense of making your system act very sluggish for everything else and heating up your CPU, you will end up with task scheduling that is mostly jitter free.
I wrote a system like this once under Windows XP to spin a stepper motor, supplying evenly spaced pulses up to 40K times per second, and it worked fine. Of course, your mileage may vary.
Related
I want to sleep in the kernel for a specific amount of time, and I use time_before and jiffies to calculate the amount of time I should sleep, however I don't understand how the calculation actually works. I know that HZ is 250 and jiffies is a huge dynamic value. I know what them both are and what they are used for.
I calculate the time with jiffies + (10 * HZ).
static unsigned long j1;
static int __init sys_module_init(void)
{
j1 = jiffies + (10 * HZ);
while (time_before(jiffies, j1))
schedule();
printk("Hello World - %d - %ld\n", HZ, jiffies); // Hello World - 250 - 4296485594 (dynamic)
return 0;
}
How does the calculation work and how many seconds will I sleep? I want to know that because in the future I'll probably want to sleep for a specific time.
HZ represents the amount of ticks in a second, and multiplying that by 10 gives the amount of ticks in 10 seconds. So the calculation jiffies + 10 * HZ yields the expected value of jiffies 10 seconds from now.
However, calling schedule() in a loop until that value is hit is not the recommended way to go. If you want to sleep in the kernel, you don't need to reinvent the wheel, there already is a set of APIs just for this purpose, documented here, which will make your life a lot easier. The simplest way to sleep in your specific case would be to use msleep(), passing the number of milliseconds you want to sleep for:
#include <linux/delay.h>
static int __init sys_module_init(void)
{
msleep(10 * 1000);
return 0;
}
I have a 16 core Linux machine that is idle. If I run a trivial, single threaded C program that sits in a loop reading the cycle counter forever (using the rdtsc instruction), then every 0.5 seconds, I see a 0.17 ms jump in the timer value. In other words, every 0.5 seconds it seems that my application is stopped for 0.17ms. I would like to understand why this happens and what I can do about it. I understand Linux is not a real time operating system. I'm just trying to understand what is going on, so I can make the best use of what Linux provides.
I found someone else's software for measuring this problem - https://github.com/nokia/clocktick_jumps. Its results are consistent with my own.
To answer the "tell us what specific problem you are trying to solve" question - I work on high-speed networking applications using DPDK. About 60 million packets arrive per second. I need to decide what size to make the RX buffers and have reasons that the number I pick is sensible. The answer to this question is one part of that puzzle.
My code looks like this:
// Build with gcc -O2 -Wall
#include <stdio.h>
#include <unistd.h>
#include <x86intrin.h>
int main() {
// Bad way to learn frequency of cycle counter.
unsigned long long t1 = __rdtsc();
usleep(1000000);
double millisecs_per_tick = 1e3 / (double)(__rdtsc() - t1);
// Loop forever. Print message if any iteration takes unusually long.
t1 = __rdtsc();
while (1) {
unsigned long long t2 = __rdtsc();
double delta = t2 - t1;
delta *= millisecs_per_tick;
if (delta > 0.1) {
printf("%4.2f - Delay of %.2f ms.\n", (double)t2 * millisecs_per_tick, delta);
}
t1 = t2;
}
return 0;
}
I'm running on Ubuntu 16.04, amd64. My processor is an Intel Xeon X5672 # 3.20GHz.
I expect your system is scheduling another process to run on the same CPU, and you're either replaced or moved to another core with some timing penalty.
You can find the reason by digging into kernel events happening at the same time. For example systemtap, or perf can give you some insight. I'd start with the scheduler events to eliminate that one first: https://github.com/jav/systemtap/blob/master/tapset/scheduler.stp
My CPU has four cores,MAC os. I use 4 threads to calculate an array. But the time of calculating does't being reduced. If I don't use multithread, the time of calculating is about 52 seconds. But even I use 4 multithreads, or 2 threads, the time doesn't change.
(I know why this happen now. The problem is that I use clock() to calculate the time. It is wrong when it is used in multithread program because this function will multiple the real time based on the num of threads. When I use time() to calculate the time, the result is correct.)
The output of using 2 threads:
id 1 use time = 43 sec to finish
id 0 use time = 51 sec to finish
time for round 1 = 51 sec
id 1 use time = 44 sec to finish
id 0 use time = 52 sec to finish
time for round 2 = 52 sec
id 1 and id 0 is thread 1 and thread 0. time for round is the time of finishing two threads. If I don't use multithread, time for round is also about 52 seconds.
This is the part of calling 4 threads:
for(i=1;i<=round;i++)
{
time_round_start=clock();
for(j=0;j<THREAD_NUM;j++)
{
cal_arg[j].roundth=i;
pthread_create(&thread_t_id[j], NULL, Multi_Calculate, &cal_arg[j]);
}
for(j=0;j<THREAD_NUM;j++)
{
pthread_join(thread_t_id[j], NULL);
}
time_round_end=clock();
int round_time=(int)((time_round_end-time_round_start)/CLOCKS_PER_SEC);
printf("time for round %d = %d sec\n",i,round_time);
}
This is the code inside the thread function:
void *Multi_Calculate(void *arg)
{
struct multi_cal_data cal=*((struct multi_cal_data *)arg);
int p_id=cal.thread_id;
int i=0;
int root_level=0;
int leaf_addr=0;
int neighbor_root_level=0;
int neighbor_leaf_addr=0;
Neighbor *locate_neighbor=(Neighbor *)malloc(sizeof(Neighbor));
printf("id:%d, start:%d end:%d,round:%d\n",p_id,cal.start_num,cal.end_num,cal.roundth);
for(i=cal.start_num;i<=cal.end_num;i++)
{
root_level=i/NUM_OF_EACH_LEVEL;
leaf_addr=i%NUM_OF_EACH_LEVEL;
if(root_addr[root_level][leaf_addr].node_value!=i)
{
//ignore, because this is a gap, no this node
}
else
{
int k=0;
locate_neighbor=root_addr[root_level][leaf_addr].head;
double tmp_credit=0;
for(k=0;k<root_addr[root_level][leaf_addr].degree;k++)
{
neighbor_root_level=locate_neighbor->neighbor_value/NUM_OF_EACH_LEVEL;
neighbor_leaf_addr=locate_neighbor->neighbor_value%NUM_OF_EACH_LEVEL;
tmp_credit += root_addr[neighbor_root_level][neighbor_leaf_addr].g_credit[cal.roundth-1]/root_addr[neighbor_root_level][neighbor_leaf_addr].degree;
locate_neighbor=locate_neighbor->next;
}
root_addr[root_level][leaf_addr].g_credit[cal.roundth]=tmp_credit;
}
}
return 0;
}
The array is very large, each thread calculate part of the array.
Is there something wrong with my code?
It could be a bug, but if you feel the code is correct, then the overhead of parallelization, mutexes and such, might mean the overall performance (runtime) is the same as for the non-parallelized code, for the size of elements to compute against.
It might be an interesting study, to do looped code, single-threaded, and the threaded code, against very large arrays (100k elements?), and see if the results start to diverge to be faster in the parallel/threaded code?
Amdahl's law, also known as Amdahl's argument,[1] is used to find the maximum expected improvement to an overall system when only part of the system is improved. It is often used in parallel computing to predict the theoretical maximum speedup using multiple processors.
https://en.wikipedia.org/wiki/Amdahl%27s_law
You don't always gain speed by multi-threading a program. There is a certain amount of overhead that comes with threading. Unless there is enough inefficiencies in the non-threaded code to make up for the overhead, you'll not see an improvement. A lot can be learned about how multi-threading works even if the program you write ends up running slower.
I know why this happen now. The problem is that I use clock() to calculate the time. It is wrong when it is used in multithread program because this function will multiple the real time based on the num of threads. When I use time() to calculate the time, the result is correct.
I am trying to see how much does it take for about 10000 names to be inserted into a BST(writing in c).
I am reading these names from a txt file using fscanf. I have declared a file pointer(fp) at the main function. Calling a function that is at another .c file passing the fp through its arguments. I want to count the time needed for 2,4,8,16,32...,8192 names to be inserted saving the time at a long double array. I have included the time.h library at the .c file where the function is located.
Code:
void myfunct(BulkTreePtr *Bulktree, FILE* fp,long double time[])
{
double tstart, tend, ttemp;
TStoixeioyTree datainput;
int error = 0,counter=0,index=0,num=2,i;
tstart = ((double) clock())/CLOCKS_PER_SEC;
while (!feof(fp))
{
counter++;
fscanf(fp,"%s %s", datainput.lname, datainput.fname);
Tree_input(&((*Bulktree)->TreeRoot), datainput, &error);
if (counter == num)
{
ttemp = (double) clock()/CLOCKS_PER_SEC;
time[index] = ttemp-tstart;
num = num * 2;
index++;
}
}
tend = ((double) clock())/CLOCKS_PER_SEC;
printf("Last value of ttemp is %f\n",ttemp-tstart);
time[index] = (tend-tstart);
num = 2;
for(i=0;i<14;i++)
{
printf("Time after %d names is %f sec \n", num, (float)time[i]);
num=num*2;
}
I am getting this:
Last value of ttemp is 0.000000
Time after 2 names is 0.000000 sec
Time after 4 names is 0.000000 sec
Time after 8 names is 0.000000 sec
Time after 16 names is 0.000000 sec
Time after 32 names is 0.000000
ms Time after 64 names is
0.000000 sec Time after 128 names is 0.000000 sec Time after 256
names is 0.000000 sec Time after
512 names is 0.000000 sec Time
after 1024 names is 0.000000 sec
Time after 2048 names is 0.000000 sec
Time after 4096 names is 0.000000
sec Time after 8192 names is
0.000000 sec Time after 16384 names is 0.010000 sec
What am I doing wrong? :S
Use clock_getres() and clock_gettime(). Most likely you will find your system doesn't have a very fast clock. Note that the system might return different numbers when calling gettimeofday or clock_gettime(), but often times (depending on kernel) those numbers at greater than HZ resolution are lies generated to simulate time advancing.
You might find a better test to do fixed time tests. Find out how many inserts you can do in 10 seconds. Or have some kind of fast reset method (memset?) and find out how many groups of inserts of 1024 names you can do in 10 seconds.
[EDIT]
Traditionally, the kernel gets interrupted at HZ frequency by the hardware. Only when it gets this hardware interrupt does it know that time had advanced by 1/HZ of a second. The traditional value for HZ was 1/100 of a second. Surprise, surprise, you saw a 1/100th of a second increment in time. Now some systems and kernels have recently started providing other methods of getting higher resolution time, looking at the RTC device or whatever.
However, you should use the clock_gettime() function I pointed you to along with the clock_getres() function to find out how often you will get accurate time updates. Make sure your test runs many many multiples of clock_getres() unless you want it to be a total lie.
clock() returns the number of "ticks"; there are CLOCKS_PER_SEC ticks per second. For any operation which takes less than 1/CLOCKS_PER_SEC seconds, the return value of clock() will either be unchanged or changed by 1 tick.
From your results it looks like even 16384 insertions take no more than 1/100 seconds.
If you want to know how long a certain number of insertions take, try repeating them many, many times so that the total number of ticks is significant, and then divide that total time by the number of times they were repeated.
clock returns the amount of cpu time used, not the amount of actual time elapsed, but that might be what you want here. Note that Unix standards requires CLOCKS_PER_SECOND to be exactly one million (1000000), but the resolution can be much worse (e.g. it might jump by 10000 at a time). You should be using clock_gettime with the cpu time clock if you want to measure cpu time spent, or otherwise with the monotonic clock to measure realtime spent.
ImageMagick includes stopwatch functions such as these.
#include "magick/MagickCore.h"
#include "magick/timer.h"
TimerInfo *timer_info;
timer_info = AcquireTimerInfo();
<your code>
printf("elapsed=%.6f sec", GetElapsedTime(timer_info));
But that only seems to have a resolution of 1/100 second. Plus it requires installing ImageMagic. I suggest this instead. It's simple and has usec resolution in Linux.
#include <time.h>
double timer(double start_secs)
{
static struct timeval tv;
static struct timezone tz;
gettimeofday(&tv, &tz);
double now_secs = (double)tv.tv_sec + (double)tv.tv_usec/1000000.0;
return now_secs - start_secs;
}
double t1 = timer(0);
<your code>
printf("elapsed=%.6f sec", timer(t1));
Need a running (moving, rolling) average algorithm to calculate the 5-minute average bits that are passed in. All I have to work with is an accumulative value for the bits that are passed in.
For example: I start with 0 bits, 5 minutes later, I have 10 bits, so my average is 10 bits. 5 minutes later, I have 15 bits, so now my average is 7.5 bits. Another 5 minutes later, I have 30 bits, so my average now is 10.8 bits.
My question is, how can I implement a timer\counter in C++ so it would poll the bits value in exact 5 minutes intervals? Obviously I can't use delay 300 seconds. But can I make a timer in the background which would only fire an event (poll the bit value) every 5 minutes?
The Code to my Previous Answer
#define REENTRANT
//The above is neccessary when using threads. This must be defined before any includes are made
//Often times, gcc -DREENTRANT is used instead of this, however, it produces the same effect
#include <pthread.h>
char running=1;
void* timer(void* dump){
unsigned char i=0;
while(running){
for(i=0;i<300 && running;i++){
sleep(1);//so we don't need to wait the 300 seconds when we want to quit
}
if(running)
callback();//note that this is called from a different thread from main()
}
pthread_exit(NULL);
}
int main(){
pthread_t thread;
pthread_create(&thread,NULL,timer,NULL);
//do some stuff
running=0;
pthread_join(thread,NULL);//we told it to stop running, however, we might need to wait literally a second
pthread_exit(NULL);
return 0;
}
The "dumb" solution is to use POSIX threads. You can make a thread and then put it in an infinite loop with sleep() in it.