It seems that glibc's implementation of fprintf() is thread-safe, but is that so for Microsoft's CRT, as well?
By thread-safe, I don't mean just crashing, but also that if multiple threads (in the same process) call fprintf(), the texts will not be mixed.
That is, for example, if thread A calls fprintf(stdout, "aaaa"); and thread B calls fprintf(stdout, "bbbb"); it's guaranteed not to mix to become aabbaabb.
Is there such a guarantee?
Yes. In the multithreaded runtime libraries, every stream has an associated lock. This lock is acquired at the beginning of any call to a printf function and not released until just before that printf function returns.
This behavior is required by C11 (there was no concept of "threads" in standard C until C11). C11 §7.21.2/7-8 states:
Each stream has an associated lock that is used to prevent data races when multiple
threads of execution access a stream, and to restrict the interleaving of stream operations performed by multiple threads. Only one thread may hold this lock at a time. The lock is reentrant: a single thread may hold the lock multiple times at a given time.
All functions that read, write, position, or query the position of a stream lock the stream before accessing it. They release the lock associated with the stream when the access is complete.
Visual C++ does not fully support C11, but it does conform to this requirement. A couple of other Visual C++-specific comments:
As long as you are not defining _CRT_DISABLE_PERFCRIT_LOCKS (which only works with the statically-linked runtime libraries, libcmt.lib and friends) or using the _nolock-suffixed functions, then most operations on a single stream are atomic.
If you require atomicity across multiple operations on a stream, you can acquire the lock for a file yourself by acquiring and releasing the stream lock yourself using _lock_file and _unlock_file.
Related
This question already has answers here:
Atomicity of `write(2)` to a local filesystem
(4 answers)
Closed 4 years ago.
I was reading the APUE(Advanced Programming in the UNIX Environment), and come across this question when I see $3.11:
if (lseek(fd, 0L, 2) < 0) /* position to EOF */
err_sys("lseek error");
if (write(fd, buf, 100) != 100) /* and write */
err_sys("write error")
APUE says:
This works fine for a single process, but problems arise if multiple processes use this technique to append to the same file. .......The problem here is that our logical operation of ‘‘position to the end of file and
write’’ requires two separate function calls (as we’ve shown it). Any operation that requires more than one function call cannot be atomic, as there is always the possibility that the kernel might temporarily suspend the process between the two function calls.
It just says cpu will switch between function calls between lseek and write, I want to know if it will also switch in half write operation? Or rather, is write atomic? If threadA writes "aaaaa", threadB writes "bbbbb", will the result be "aabbbbbaaa"?
What's more,after that APUE says pread and pwrite are all atomic operations, does that mean these functions use mutex or lock internally to be atomic?
To call the Posix semantics "atomic" is perhaps an oversimplification. Posix requires that reads and writes occur in some order:
Writes can be serialized with respect to other reads and writes. If a read() of file data can be proven (by any means) to occur after a write() of the data, it must reflect that write(), even if the calls are made by different processes. A similar requirement applies to multiple write operations to the same file position. This is needed to guarantee the propagation of data from write() calls to subsequent read() calls. (from the Rationale section of the Posix specification for pwrite and write)
The atomicity guarantee mentioned in APUE refers to the use of the O_APPEND flag, which forces writes to be performed at the end of the file:
If the O_APPEND flag of the file status flags is set, the file offset shall be set to the end of the file prior to each write and no intervening file modification operation shall occur between changing the file offset and the write operation.
With respect to pread and pwrite, APUE says (correctly, of course) that these interfaces allow the application to seek and perform I/O atomically; in other words, that the I/O operation will occur at the specified file position regardless of what any other process does. (Because the position is specified in the call itself, and does not affect the persistent file position.)
The Posix sequencing guarantee is as follows (from the Description of the write() and pwrite() functions):
After a write() to a regular file has successfully returned:
Any successful read() from each byte position in the file that was modified by that write shall return the data specified by the write() for that position until such byte positions are again modified.
Any subsequent successful write() to the same byte position in the file shall overwrite that file data.
As mentioned in the Rationale, this wording does guarantee that two simultaneous write calls (even in different unrelated processes) will not interleave data, because if data were interleaved during a write which will eventually succeed the second guarantee would be impossible to provide. How this is accomplished is up to the implementation.
It must be noted that not all filesystems conform to Posix, and modular OS design, which allows multiple filesystems to coexist in a single installation, make it impossible for the kernel itself to provide guarantees about write which apply to all available filesystems. Network filesystems are particularly prone to data races (and local mutexes won't help much either), as is mentioned as well by Posix (at the end of the paragraph quoted from the Rationale):
This requirement is particularly significant for networked file systems, where some caching schemes violate these semantics.
The first guarantee (about subsequent reads) requires some bookkeeping in the filesystem, because data which has been successfully "written" to a kernel buffer but not yet synched to disk must be made transparently available to processes reading from that file. This also requires some internal locking of kernel metadata.
Since writing to regular files is typically accomplished via kernel buffers and actually synching the data to the physical storage device is definitely not atomic, the locks necessary to provide these guarantee don't have to be very long-lasting. But they must be done inside the filesystem because nothing in the Posix wording limits the guarantees to simultaneous writes within a single threaded process.
Within a multithreaded process, Posix does require read(), write(), pread() and pwrite() to be atomic when they operate on regular files (or symbolic links). See Thread Interactions with Regular File Operations for a complete list of interfaces which must obey this requirement.
In Linux there are blocking and non-blocking system calls. The write is an example of blocking system call, which means the execution thread will be blocked until the write completes. So once the user process called write, it can not execute anything else until the system call is complete. So from user thread perspective it will behave like atomic [although at kernel level lot many things can happen and kernel execution of system call can be interrupted many times].
Are there any guarantees on when a memory write in one thread becomes visible in other threads using pthreads?
Comparing to Java, the Java language spec has a section that specifies the interaction of locks and memory that makes it possible to write portable multi-threaded Java code.
Is there a corresponding pthreads spec?
Sure, you can always go and make shared data volatile, but that is not what I'm after.
If this is platform dependent, is there a de facto standard? Or should another threading library be used?
POSIX specifies the memory model in 4.11 Memory Synchronization:
Applications shall ensure that access to any memory location by more than one thread of control (threads or processes) is restricted such that no thread of control can read or modify a memory location while another thread of control may be modifying it. Such access is restricted using functions that synchronize thread execution and also synchronize memory with respect to other threads. The following functions synchronize memory with respect to other threads:
fork()
pthread_barrier_wait()
pthread_cond_broadcast()
pthread_cond_signal()
pthread_cond_timedwait()
pthread_cond_wait()
pthread_create()
pthread_join()
pthread_mutex_lock()
pthread_mutex_timedlock()
pthread_mutex_trylock()
pthread_mutex_unlock()
pthread_spin_lock()
pthread_spin_trylock()
pthread_spin_unlock()
pthread_rwlock_rdlock()
pthread_rwlock_timedrdlock()
pthread_rwlock_timedwrlock()
pthread_rwlock_tryrdlock()
pthread_rwlock_trywrlock()
pthread_rwlock_unlock()
pthread_rwlock_wrlock()
sem_post()
sem_timedwait()
sem_trywait()
sem_wait()
semctl()
semop()
wait()
waitpid()
The pthread_once() function shall synchronize memory for the first call in each thread for a given pthread_once_t object.
The pthread_mutex_lock() function need not synchronize memory if the mutex type if PTHREAD_MUTEX_RECURSIVE and the calling thread already owns the mutex. The pthread_mutex_unlock() function need not synchronize memory if the mutex type is PTHREAD_MUTEX_RECURSIVE and the mutex has a lock count greater than one.
Unless explicitly stated otherwise, if one of the above functions returns an error, it is unspecified whether the invocation causes memory to be synchronized.
Applications may allow more than one thread of control to read a memory location simultaneously.
I am not aware that POSIX threads give such guarantees. They don't have a model for atomic access to thread-shared objects. If it'd be for POSIX threads, the only guarantees that you can have for visibility of modifications is using some kind of lock.
Modern C, C11, (and probably also C++11) has a model for this kind of questions. It has threads and atomics (fences and all that stuff) that give you exact rules when you may assume that a modification done by one thread is visible by another.
The thread interface of C11 is a cooked-down version of POSIX threads, with less functionality. Unfortunately, the specification for the semantics of that thread interface is yet much to loose, basically the semantics are missing in many places. But a combination of C11 interfaces and POSIX thread semantics can give you a good view of how things work in modern systems.
Edit: So if you want to have guarantees for memory synchronization use either the lock interfaces that POSIX provides or go for atomic operations. All modern compilers have extensions that provide these, gcc and family (icc, opencc, clang) have e.g the series of __sync... builtins. Clang it its newest version also already has support of the new C11 _Atomic feature. There are also wrappers available that give you interfaces for the other compilers that come close to _Atomic.
This question already has answers here:
Does guarding a variable with a pthread mutex guarantee it's also not cached?
(3 answers)
Closed 3 years ago.
Do pthread_mutex_lock and pthread_mutex_unlock functions call memory fence/barrier instructions? Or do the the lower level instructions like compare_and_swap implicity have memory barriers?
Do pthread_mutex_lock and pthread_mutex_unlock functions call memory fence/barrier instructions?
They do, as well as thread creation.
Note, however, there are two types of memory barriers: compiler and hardware.
Compiler barriers only prevent the compiler from reordering reads and writes and speculating variable values, but don't prevent the CPU from reordering.
The hardware barriers prevent the CPU from reordering reads and writes. Full memory fence is usually the slowest instruction, most of the time you only need operations with acquire and release semantics (to implement spinlocks and mutexes).
With multi-threading you need both barriers most of the time.
Any function whose definition is not available in this translation unit (and is not intrinsic) is a compiler memory barrier. pthread_mutex_lock, pthread_mutex_unlock, pthread_create also issue a hardware memory barrier to prevent the CPU from reordering reads and writes.
From Programming with POSIX Threads by David R. Butenhof:
Pthreads provides a few basic rules about memory visibility. You can count on all implementations of the standard to follow these rules:
Whatever memory values a thread can see when it calls pthread_create can also be seen by the new thread when it starts. Any data written to memory after the call to pthread_create may not necessarily be seen by the new thread, even if the write occurs before the thread starts.
Whatever memory values a thread can see when it unlocks a mutex, either directly or by waiting on a condition variable, can also be seen by any thread that later locks the same mutex. Again, data written after the mutex is unlocked may not necessarily be seen by the thread that locks the mutex, even if the write occurs before the lock.
Whatever memory values a thread can see when it terminates, either by cancellation, returning from its start function, or by calling pthread_exit, can also be seen by the thread that joins with the terminated thread bycalling pthread_join. And, of course, data written after the thread terminates may not necessarily be seen by the thread that joins, even if the write occurs before the join.
Whatever memory values a thread can see when it signals or broadcasts a condition variable can also be seen by any thread that is awakened by that signal or broadcast. And, one more time, data written after the signal or broadcast may not necessarily be seen by the thread that wakes up, even if the write occurs before it awakens.
Also see C++ and Beyond 2012: Herb Sutter - atomic<> Weapons for more details.
Please take a look at section 4.12 of the POSIX specification.
Applications shall ensure that access to any memory location by more than one thread of control (threads or processes) is restricted such that no thread of control can read or modify a memory location while another thread of control may be modifying it. Such access is restricted using functions that synchronize thread execution and also synchronize memory with respect to other threads. [emphasis mine]
Then a list of functions is given which synchronize memory, plus a few additional notes.
If that requires memory barrier instructions on some architecture, then those must be used.
About compare_and_swap: that isn't in POSIX; check the documentation for whatever you are using. For instance, IBM defines a compare_and_swap function for AIX 5.3. which doesn't have full memory barrier semantics The documentation note says:
If compare_and_swap is used as a locking primitive, insert an isync at the start of any critical sections.
From this documentation we can guess that IBM's compare_and_swap has release semantics: since the documentation does not require a barrier for the end of the critical section. The acquiring processor needs to issue an isync to make sure it is not reading stale data, but the publishing processor doesn't have to do anything.
At the instruction level, some processors have compare and swap with certain synchronizing guarantees, and some don't.
I'm writing a server web.
Each connection is served by a separate thread, so I don't know in advance the number of threads.
There are also a group of text files (don't know the number, too), and each thread can read/write on each file.
A file can be written by just one thread a time, but different threads can write on different files at the same time.
If a file is read by one or more threads (reads can be concurrent), no thread can write on THAT file.
Now, I noticed this (Thread safe multi-file writing) solution, but I'd like also to use functions as fgets(), for example.
So, can I flock() a file, and then use a fgets() or another stdio read/write library function?
First of all, use fcntl, not flock. The latter is a non-standard, deprecated BSD function and does not work with NFS and possibly other filesystems. fcntl locking on the other hand is POSIX standard and is intended to work everywhere.
Now if you want to use file-level reader-writer locking mixed with stdio, it will work, but you have to take some care to ensure that buffering does not break your assumptions about locks. The method I'm about to explain is not the only one, but I believe it's the clearest/simplest:
When you want to operate on one of your files with stdio, obtaining the correct type of lock (read or write, aka shared of exclusive) should be the first thing you do after fopen. Use fileno to get the file descriptor number and apply the lock to it. After that, perform your entire read or write operation. Do not make any attempt to unlock the file; instead, call fclose to close the file and let it be implicitly unlocked when it's closed. Otherwise you may release the lock while unbuffered data is still unwritten, or later read data that was buffered before the lock was released, that's no longer valid after the lock is released.
Is writing to stdout using printf thread-safe on Linux? What about using the lower-level write command?
It's not specified by the C standard -- it depends on your implementation of the C standard library. In fact, the C standard doesn't even mention threads at all, since certain systems (e.g. embedded systems) don't have multithreading.
In the GNU implementation (glibc), most of the higher-level functions in stdio that deal with FILE* objects are thread-safe. The ones that aren't usually have unlocked in their names (e.g. getc_unlocked(3)). However, the thread safety is at a per-function call level: if you make multiple calls to printf(3), for example, each of those calls is guaranteed to output atomically, but other threads might print things out between your calls to printf(). If you want to ensure that a sequence of I/O calls gets output atomically, you can surround them with a pair of flockfile(3)/funlockfile(3) calls to lock the FILE handle. Note that these functions are reentrant, so you can safely call printf() in between them, and that won't result in deadlock even thought printf() itself makes a call to flockfile().
The low-level I/O calls such as write(2) should be thread-safe, but I'm not 100% sure of that - write() makes a system call into the kernel to perform I/O. How exactly this happens depends on what kernel you're using. It might be the sysenter instruction, or the int (interrupt) instruction on older systems. Once inside the kernel, it's up to the kernel to make sure that the I/O is thread-safe. In a test I just did with the Darwin Kernel Version 8.11.1, write(2) appears to be thread-safe.
Whether you'd call it "thread-safe" depends on your definition of thread-safe. POSIX requires stdio functions to use locking, so your program will not crash, corrupt the FILE object states, etc. if you use printf simultaneously from multiple threads. However, all stdio operations are formally specified in terms of repeated calls to fgetc and fputc, so there is no larger-scale atomicity guaranteed. That is to say, if threads 1 and 2 try to print "Hello\n" and "Goodbye\n" at the same time, there's no guarantee that the output will be either "Hello\nGoodbye\n" or "Goodbye\nHello\n". It could just as well be "HGelolodboy\ne\n". In practice, most implementations will acquire a single lock for the entire higher-level write call simply because it's more efficient, but your program should not assume so. There may be corner cases where this is not done; for instance an implementation could probably entirely omit locking on unbuffered streams.
Edit: The above text about atomicity is incorrect. POSIX guarantees all stdio operations are atomic, but the guarantee is hidden in the documentation for flockfile: http://pubs.opengroup.org/onlinepubs/9699919799/functions/flockfile.html
All functions that reference ( FILE *) objects shall behave as if they use flockfile() and funlockfile() internally to obtain ownership of these ( FILE *) objects.
You can use the flockfile, ftrylockfile, and funlockfile functions yourself to achieve larger-than-single-function-call atomic writes.
They are both thread-safe to the point that your application won't crash if multiple threads call them on the same file descriptor. However, without some application-level locking, whatever is written could be interleaved.
C got a new standard since this question was asked (and last answered).
C11 now comes with multithreading support and addresses multithreaded behavior of streams:
§7.21.2 Streams
¶7 Each stream has an associated lock that is used to prevent data races when multiple threads of execution access a stream, and to restrict the interleaving of stream operations performed by multiple threads. Only one thread may hold this lock at a time. The lock is reentrant: a single thread may hold the lock multiple times at a given time.
¶8 All functions that read, write, position, or query the position of a stream lock the stream before accessing it. They release the lock associated with the stream when the access is complete.
So, an implementation with C11 threads must guarantee that using printf is thread-safe.
Whether atomicity (as in no interleaving1) is guaranteed, wasn't that clear to me at a first glance, because the standard spoke of restricting interleaving, as opposed to preventing, which it mandated for data races.
I lean towards it being guaranteed. The standard speaks of restricting interleaving, as some interleaving that doesn't change the outcome is still allowed to happen; e.g. fwrite some bytes, fseek back some more and fwrite till the original offset, so that both fwrites are back-to-back. The implementation is free to reorder these 2 fwrites and merge them into a single write.
1: See the strike-through text in R..'s answer for an example.
It's thread-safe; printf should be reentrant, and you won't cause any strangeness or corruption in your program.
You can't guarantee that your output from one thread won't start half way through the output from another thread. If you care about that you need to develop your own locked output code to prevent multiple access.