Does the C/++ memory model apply to the atomic operation itself? - c

Question

I'm left confused about when the C/++ memory model is relevant, even after reading the GCC wiki.
My code is an IO library that allows taking/returning a buffer from a pool and using it for async IO. However, even after the buffer is returned to the pool, it isn't free unless the actual IO operation has also completed.
Each buffer has a structure that has status flags:
#define IO_FLAG_IN_USE 1 // a consumer has taken ownership of the buffer
#define IO_FLAG_IN_FLIGHT 2 // the buffer is in use by the system for async IO
A consumer requests a buffer with io_getbuf and waits using sem_wait. There are two ways a buffer can become available:
When the consumer calls io_putbuf and the IO has already completed, or when IO completes and the buffer has already been returned. This can cause a race, of course. I want to solve it using atomics, like this:
void io_completion(struct bufinfo *buf) {
if(!__atomic_or_fetch(&buf->flags, ~IO_FLAG_IN_FLIGHT, ...))
sem_post(semaphore);
}
void io_putbuf(struct bufinfo *buf) {
if(!__atomic_or_fetch(&buf->flags, ~IO_FLAG_IN_USE, ...))
sem_post(semaphore);
}
But I'm not sure which memory model to specify - does it matter?
tl;dr
Does the memory model apply to the atomic operations themselves (the load->or->return) or only relevant for operations preceding/following the atomic built-ins?

Answer 1

I take you to be asking what memory order property(-ies) to use, and to be asking in particular about the GCC builtin __atomic_and_fetch() (note spelling), which atomically modifies the specified memory location via a bitwise and operation on a scalar having non-_Atomic type and returns the result (atomic read / modify / write). The memory-order alternatives and resulting behavior correspond to the C++ memory model.
Do note well that that's a GCC-ism. C has had atomic types and atomic operations on them since C11, with the same set of memory-order alternatives as C++, but the __atomic_or_fetch() builtin and its siblings are separate from that and GCC-specific.
I'm not sure which memory model to specify - does it matter?
Yes, of course, else there wouldn't be alternatives to choose among.
Does
the memory model apply to the atomic operations themselves (the
load->or->return) or only relevant for operations preceding/following
the atomic built-ins?
The memory order property describes the relationships, if any, between the read and write performed by the atomic operation on one hand and not-necessarily-atomic reads and writes of the same and other memory locations on the other. If ever you are uncertain what memory order to use then you should use __ATOMIC_SEQ_CST. That provides the strongest constraints, and it corresponds to the default memory order for C++ atomic operations.
Other alternatives relax memory-order constraints in various ways, which may afford performance improvements under some circumstances. However, those relaxations are likely to also cause your program to manifest intermittent misbehavior if in fact it requires the stronger constraints, and that analysis involves a holistic evaluation of your threads' use of shared variables and synchronization.
I am not confident that enough information has been presented to fully perform that analysis, but we can see at least that each of the atomic operations must observe the effect of the other, therefore each one needs both acquire and release semantics with respect to the affected memory location. That is, you need at least __ATOMIC_ACQ_REL ordering, which is only one step below __ATOMIC_SEQ_CST. Use the latter, because it's safer, especially given that you are uncertain about the intricacies involved. This is for an I/O subsystem, so even if the former would be sufficient, any performance gain you might see from using that is unlikely to be noticeable anyway.
UPDATE:
Since apparently the above is not clear, again:
Does the memory model apply to the atomic operations themselves (the
load->or->return) or only relevant for operations preceding/following
the atomic built-ins?
And again: the memory order property describes the relationships, if any, between the read and write performed by the atomic operation on one hand and not-necessarily-atomic reads and writes of the same and other memory locations on the other.
That is, the chosen memory order parameter affects
whether there are happens-before relationships between writes to the affected memory location and the atomic op's read of that location by other threads;
whether there are happens-before relationships between the atomic op's write to the affected storage location and other reads of that location by other threads;
whether there are happens-before relationships between the atomic op's read of the affected storage location and other actions performed by the same thread; and
whether there are happens-before relationships between the atomic op's write of the affected storage location and other actions performed by the same thread.
Note that I say "affects", not "determines". Those factors are also affected by the memory order of other atomic operations, by the use of synchronization objects and functions, by details of the other statements executed and expressions evaluated by all threads, and by the vagaries of thread scheduling during any particular run (at least).
All of that speaks to the guarantees upon which one can rely and the conclusions one can draw about relationships among the values of shared variables throughout the program observed by each thread.
The full scope of the guarantees you require is unclear, but you at least need happens-before relationships between the reads and writes of the affected location by one function and those by the other function, with no specific order imposed on the calls to those functions. That requires at minimum __ATOMIC_ACQ_REL ordering, but, again, one step up to __ATOMIC_SEQ_CST is probably a better choice, not least because it might be a necessary one in light of other program code not shown.

Answer 2

With regards to the atomic variable itself, the __ATOMIC_RELAXED memory-model is always sufficient between two atomic operations. It guarantees that there will be some fixed order between them, and that they won't used cached values.
That's why you can use __ATOMIC_RELAXED for the simple incrementing of a counter, see the Relaxed ordering entry at cppreference.com
In the example given, if the buffers involved in the example code are 'simple' buffers that carry no state (they are fixed-sized etc.) and the consuming thread (io_getbuf) does not care about the contents of the buffer (it is just using it for a new read) then there are no other dependencies and you can use __ATOMIC_RELAXED.
The need for memory models is when there are other dependencies involved - if the buffers contain metadata, like the buffer size, and that size might have been changed (e.g. realloc) by the releasing thread then __ATOMIC_RELAXED does not guarantee that the consuming thread will see the update to the size field. Similarly, if this wasn't just a thread pool, but a producer/consumer setup, then the consumer needs to be sure that the contents of the buffer are actually synchronized and have been written before consuming the buffer - that would require a different memory model.