I would like to measure the ping time between a sender process and a consumer.
Both processes run on the same physical host, Linux 64 bit.
I'm currently using clock_gettime(CLOCK_REALTIME, &cur_ts);.
I basically capture the current timestamp cur_ts and send it to consumer process; as soon as it's being received on the other end I then invoke clock_gettime(CLOCK_REALTIME, &cur_ts); again in the second process and subtract the two times.
Is this procedure accurate to measure ping between two processes? Should I be using something different?
Thanks
Yes
You are using the correct interface. This is a Posix-specified HRT (High Resolution Timer) that is available in modern Loonix, and your use is the canonical standard.
For ping resolutions it probably doesn't matter, but it's possible to use CLOCK_MONOTONIC, especially for short intervals. Under other conditions, CLOCK_REALTIME may be more accurate. However, the exact meaning of these clocks is not specified by Posix, and on Linux, I believe all of them are subject to NTP adjustments. That's a good thing for long intervals, at least.
On Linux, to get access to the underlying hardware clock without NTP adjustments, you have to go off the Posix reservation and use CLOCK_MONOTONIC_RAW.
http://pubs.opengroup.org/onlinepubs/9699919799/functions/clock_gettime.html
Related
I am writing a device driver and want to benchmark a few pieces of code to get a feel for where I could be experiencing some bottlenecks. As a result, I want to time a few segments of code.
In userspace, I'm used to using clock_gettime() with CLOCK_MONOTONIC. Looking at the kernel sources (note that I am running kernel 4.4, but will be upgrading eventually), it appears I have a few choices:
getnstimeofday()
getrawmonotonic()
get_monotonic_coarse()
getboottime()
For convenience, I have written a function (see below) to get me the current time. I am currently using getrawmonotonic() because I figured this is what I wanted. My function returns the current time as a ktime_t, so then I can use ktime_sub() to get the elapsed time between two times.
static ktime_t get_time_now(void) {
struct timespec time_now;
getrawmonotonic(&time_now);
return timespec_to_ktime(time_now);
}
Given the available high resolution clocking functions (jiffies won't work for me), what is the best function for my given application? More generally, I'm interested in any/all documentation about these functions and the underlying clocks. Primarily, I am curious if the clocks are affected by any timing adjustments and what their epochs are.
Are you comparing measurements you're making in the kernel directly with measurements you've made in userspace? I'm wondering about your choice to use CLOCK_MONOTONIC_RAW as the timebase in the kernel, since you chose to use CLOCK_MONOTONIC in userspace. If you're looking for an analogous and non-coarse function in the kernel which returns CLOCK_MONOTONIC (and not CLOCK_MONOTONIC_RAW) time, look at ktime_get_ts().
It's possible you could also be using raw kernel ticks to be measuring what you're trying to measure (rather than jiffies, which represent multiple kernel ticks), but I do not know how to do that off the top of my head.
In general if you're trying to find documentation about Linux timekeeping, you can take a look at Documentation/timers/timekeeping.txt. Usually when I try to figure out kernel timekeeping I also unfortunately just spend a lot of time reading through the kernel source in time/ (time/timekeeping.c is where most of the functions you're thinking of using right now live... it's not super well-commented, but you can probably wrap your head around it with a little bit of time). And if you're feeling altruistic after learning, remember that updating documentation is a good way to contribute to the kernel :)
To your question at the end about how clocks are affected by timing adjustments and what epochs are used:
CLOCK_REALTIME always starts at Jan 01, 1970 at midnight (colloquially known as the Unix Epoch) if there are no RTC's present or if it hasn't already been set by an application in userspace (or I guess a kernel module if you want to be weird). Usually the userspace application which sets this is the ntp daemon, ntpd or chrony or similar. Its value represents the number of seconds passed since 1970.
CLOCK_MONTONIC represents the number of seconds passed since the device was booted up, and if the device is suspended at a CLOCK_MONOTONIC value of x, when it's resumed, it resumes with CLOCK_MONOTONIC set to x as well. It's not supported on ancient kernels.
CLOCK_BOOTTIME is like CLOCK_MONOTONIC, but has time added to it across suspend/resume -- so if you suspend at a CLOCK_BOOTTIME value of x, for 5 seconds, you'll come back with a CLOCK_BOOTTIME value of x+5. It's not supported on old kernels (its support came about after CLOCK_MONOTONIC).
Fully-fleshed NTP daemons (not SNTP daemons -- that's a more lightweight and less accuracy-creating protocol) set the system clock, or CLOCK_REALTIME, using settimeofday() for large adjustments ("steps" or "jumps") -- these immediately affect the total value of CLOCK_REALTIME, and using adjtime() for smaller adjustments ("slewing" or "skewing") -- these affect the rate at which CLOCK_REALTIME moves forward per CPU clock cycle. I think for some architectures you can actually tune the CPU clock cycle through some means or other, and the kernel implements adjtime() this way if possible, but don't quote me on that. From both the bulk of the kernel's perspective and userspace's perspective, it doesn't actually matter.
CLOCK_MONOTONIC, CLOCK_BOOTTIME, and all other friends slew at the same rate as CLOCK_REALTIME, which is actually fairly convenient in most situations. They're not affected by steps in CLOCK_REALTIME, only by slews.
CLOCK_MONOTONIC_RAW, CLOCK_BOOTTIME_RAW, and friends do NOT slew at the same rate as CLOCK_REALTIME, CLOCK_MONOTONIC, and CLOCK_BOOTIME. I guess this is useful sometimes.
Linux provides some process/thread-specific clocks to userspace (CLOCK_PROCESS_CPUTIME_ID, CLOCK_THREAD_CPUTIME_ID), which I know nothing about. I do not know if they're easily accessible in the kernel.
I have a small program running on Linux (on an embedded PC, dual-core Intel Atom 1.6GHz with Debian 6 running Linux 2.6.32-5) which communicates with external hardware via an FTDI USB-to-serial converter (using the ftdi_sio kernel module and a /dev/ttyUSB* device). Essentially, in my main loop I run
clock_gettime() using CLOCK_MONOTONIC
select() with a timeout of 8 ms
clock_gettime() as before
Output the time difference of the two clock_gettime() calls
To have some level of "soft" real-time guarantees, this thread runs as SCHED_FIFO with maximum priority (showing up as "RT" in top). It is the only thread in the system running at this priority, no other process has such priorities. My process has one other SCHED_FIFO thread with a lower priority, while everything else is at SCHED_OTHER. The two "real-time" threads are not CPU bound and do very little apart from waiting for I/O and passing on data.
The kernel I am using has no RT_PREEMPT patches (I might switch to that patch in the future). I know that if I want "proper" realtime, I need to switch to RT_PREEMPT or, better, Xenomai or the like. But nevertheless I would like to know what is behind the following timing anomalies on a "vanilla" kernel:
Roughly 0.03% of all select() calls are timed at over 10 ms (remember, the timeout was 8 ms).
The three worst cases (out of over 12 million calls) were 31.7 ms, 46.8 ms and 64.4 ms.
All of the above happened within 20 seconds of each other, and I think some cron job may have been interfering (although the system logs are low on information apart from the fact that cron.daily was being executed at the time).
So, my question is: What factors can be involved in such extreme cases? Is this just something that can happen inside the Linux kernel itself, i.e. would I have to switch to RT_PREEMPT, or even a non-USB interface and Xenomai, to get more reliable guarantees? Could /proc/sys/kernel/sched_rt_runtime_us be biting me? Are there any other factors I may have missed?
Another way to put this question is, what else can I do to reduce these latency anomalies without switching to a "harder" realtime environment?
Update: I have observed a new, "worse worst case" of about 118.4 ms (once over a total of around 25 million select() calls). Even when I am not using a kernel with any sort of realtime extension, I am somewhat worried by the fact that a deadline can apparently be missed by over a tenth of a second.
Without more information it is difficult to point to something specific, so I am just guessing here:
Interrupts and code that is triggered by interrupts take so much time in the kernel that your real time thread is significantly delayed. This depends on the frequency of interrupts, which interrupt handlers are involved, etc.
A thread with lower priority will not be interrupted inside the kernel until it yields the cpu or leaves the kernel.
As pointed out in this SO answer, CPU System Management Interrupts and Thermal Management can also cause significant time delays (up to 300ms were observed by the poster).
118ms seems quite a lot for a 1.6GHz CPU. But one driver that accidently locks the cpu for some time would be enough. If you can, try to disable some drivers or use different driver/hardware combinations.
sched_rt_period_us and sched_rt_period_us should not be a problem if they are set to reasonable values and your code behaves as you expect. Still, I would remove the limit for RT threads and see what happens.
What else can you do? Write a device driver! It's not that difficult and interrupt handlers get a higher priority than realtime threads. It may be easier to switch to a real time kernel but YMMV.
I am writing a network program which can calculate accurate data packet rate (packet per second, frame per second, bps). Now i have a device called testcenter which can send accurate flow to a specific pc (protocol is UDP/IP) on Linux, i like to know the accurate pps(packets per second) with my program , i have considered the gettimeofday(&start,NULL)function before i call recvfrom() and update the counter for packets, after that call gettimeofday(&end,NULL) and get the pps rate. I hope there is better solution than this since the user/kernel barrier is traversed on system calls.
Best regards.
I think you should use clock_gettime() with CLOCK_MONOTONIC_COARSE. But it will only be accurate till the last tick .. So may be off by 10s of millisec. But its definitely faster that using it with CLOCK_MONOTONIC_RAW. You can also use gettimeofday but clock_gettime with CLOCK_MONOTONIC_RAW is slightly faster and higher resolution than gettimeofday.
Also gettimeofday() gives wall clock time, which might change even for daylight saving ... I don't think you should use it to measure traffic rate.
Your observation that gettimeofday switches to kernel mode is incorrect for Linux on a few popular architectures due to the use of vsyscalls. Clearly using gettimeofday here is not a bad option. You should however consider using a monotonic clock, see man 3 clock_gettime. Note that clock_gettime is not yet converted to vsyscall for as many architectures as gettimeofday.
Beyond this option you may be able to set the SO_TIMESTAMP socket option and obtain precise timestamps via recvmsg.
I have a small program that needs to be run in a small Linux embedded system (ARM). It is written in C. It needs to poll some data (2x64-bit) from an API provided by the system manufacturer, and then do some calculations and send the data through the network. The data should be polled around 30 times every second (30Hz).
What would be the best way to do it in C? I've seen solutions using sleep(), but it does not seem to be the best option for the job.
I suggest consider using the poll(2) multiplexing syscall to do the polling.
notice that when poll is waiting and polling for input, it does not consume any CPU
If the processing of each event takes some significant time (e.g. a millisecond or more) you may want to recompute the delay.
You could use timerfd_create(2) (and give both your device file descriptor and your timer fd to poll). See also timer_create(2)...
Perhaps clock_gettime(2) could be useful.
And reading time(7) is definitely useful. Perhaps also the Advanced Linux Programming book.
sleep() suspends execution in seconds, if you are looking for a more accurate sleep()-like function, use usleep() which suspends execution in microseconds, or nanosleep() in nanoseconds.
In a long-running server program (built in C) in a Posix (Linux) environment: what is the best approach to get a function to execute at a specific time in the future? It doesn't need to execute in any particular thread, but the accuracy of the execution time needs to be a few milliseconds. General approaches or specific code is appreciated.
There are some high resolution clock functions in the GNU C library (sys/timex.h), so although they are not POSIX, they will be portable linux wise.
High Accuracy Clock -- The GNU C Library
Those functions are prefixed 'ntp' although they do not require or make use of any ntp service, so the relationship is purely superficial.
Beware that although the granularity is in microseconds, the linux kernel has a userspace latency of 10ms so don't bother or expect anything more accurate than that.
Once you have the current high resolution time, you could then calculate a duration and use (posix) nanosleep (but again, round to 10ms) to set a delay. There is also a clock_nanosleep which might be of interest.
You should look up posix timers. It gives you a simple interface to scheduling future work. You can manage it to send you a signal in x seconds/ nanoseconds (ad then you can put your function as it's signal handler). Lookup timer_create