Reference counting in C - c

Question

Trying to pass a struct between threads in plain C using reference counting. I have pthreads and gcc atomics available. I can get it to work, but I'm looking for bulletproof.
At first, I used a pthread mutex owned by the struct itself:
struct item {
int ref;
pthread_mutex_t mutex;
};
void ref(struct item *item) {
pthread_mutex_lock(&item->mutex);
item->ref++;
pthread_mutex_unlock(&item->mutex);
}
void unref(struct item *item) {
pthread_mutex_lock(&item->mutex);
item->ref--;
pthread_mutex_unlock(&item->mutex);
if (item->ref <= 0)
free(item);
}
struct item *alloc_item(void) {
struct item *item = calloc(1, sizeof(*item));
return item;
}
But, realized the mutex shouldn't be owned by the item:
static pthread_mutex_t mutex;
struct item {
int ref;
};
void ref(struct item *item) {
pthread_mutex_lock(&mutex);
item->ref++;
pthread_mutex_unlock(&mutex);
}
void unref(struct item *item) {
pthread_mutex_lock(&mutex);
item->ref--;
if (item->ref <= 0)
free(item);
pthread_mutex_unlock(&mutex);
}
struct item *alloc_item(void) {
struct item *item = calloc(1, sizeof(*item));
return item;
}
Then, further realized pointers are passed by value, so I now have:
static pthread_mutex_t mutex;
struct item {
int ref;
};
void ref(struct item **item) {
pthread_mutex_lock(&mutex);
if (item != NULL) {
if (*item != NULL) {
(*item)->ref++;
}
}
pthread_mutex_unlock(&mutex);
}
void unref(struct item **item) {
pthread_mutex_lock(&mutex);
if (item != NULL) {
if (*item != NULL) {
(*item)->ref--;
if ((*item)->ref == 0) {
free((*item));
*item = NULL;
}
}
}
pthread_mutex_unlock(&mutex);
}
struct item *alloc_item(void) {
struct item *item = calloc(1, sizeof(*item));
if (item != NULL)
item->ref = 1;
return item;
}
Are there any logical missteps here? Thanks!

Answer 1

I don't know of a general purpose solution.
It would be nice to be able to reduce this down to an atomic add/subtract of the reference count. Indeed, most of the time that is all that is required... so stepping through a mutex or whatever hurts.
But the real problem is managing the reference count and the pointer to the item, at the same time.
When a thread comes to ref() an item, how does it find it ? If it doesn't already exist, presumably it must create it. If it does already exist, it must avoid some other thread freeing it before the reference-count is incremented.
So... your void ref(struct item** item) works on the basis that the mutex protects the struct item** pointer... while you hold the mutex, no other thread can change the pointer -- so only one thread can create the item (and increment the count 0->1), and only one thread can destroy the item (after decrementing the count 1->0).
It is said that many problems in computer science can be solved by introducing a new level of indirection, and that is what is going on here. The problem is how do all the threads obtain the address of the item -- given that it may (softly and suddenly) vanish away ? Answer: invent a level of indirection.
BUT, now we are assuming that the pointer to the item cannot itself vanish. This can be trivially achieved if the pointer to the item can be held a process global (static storage duration). If the pointer to the item is (part of) an allocated storage duration object, then we must ensure that this higher level object is somehow locked -- so that the address of the pointer to the item is "stable" while it is in use. That is, the higher level object won't move around in memory and won't be destroyed while we are using it !
So, the checks if (item == NULL) after locking the mutex are suspect. If the mutex also protects the pointer to the item, then that mutex needs to have been locked before establishing the address of the pointer to the item -- and in this case checking after the lock is too late. Or the address of the pointer to the item is protected in some other way (perhaps by another mutex) -- and in this case the check can be done before the lock (and moving it there makes it clear what the mutex protects, and what it does not protect).
However, if the item is part of a larger data structure, and that structure is locked, you may (well) not need a lock to cover the pointer to the item at all. It depends... as I said, I'm not aware of a general solution.
I have some large, dynamic data structures (hash tables, queues, trees, etc.) which are shared by a number of threads. Mostly, threads look up and hold on to items for some time. When the system is busy, it is very busy, and the destruction of items can be deferred until things are quieter. So I use read/write locks on the large structures, atomic add/subtract for the reference counts, and a garbage collector to do the actual destruction of items. The point here is that the choice of mechanism for the (apparently simple and self contained) increment/decrement of the reference count, depends on how the creation and destruction of items is managed, and how threads come to be in possession of a pointer to an item (which is what the reference count counts, after all).
If you have 128-bit atomic operation to hand, you can put a 64-bit address and a 64-bit reference count together and do something along the lines of:
ref: bar = fetch_add(*foo, 1) ;
ptr = bar >> 64 ;
if (ptr == NULL)
{
if (bar & 0xF...F)
...create item etc.
else
...wait for item
} ;
unref: bar = fetch_sub(*foo, 1) ;
if ((bar & 0xF...F) == 0)
{
if (cmp_xchg(*foo, bar, (NULL << 64) | 0)))
...free(bar >> 64) ;
} ;
where foo is the 128-bit combined ptr/ref-count (whose existence is protected by some external means) -- assuming 64-bit ptr and 64-bit count -- and bar is a local variable of that form, and ptr is a void*.
If finding the pointer NULL triggers the item creation, then the first thread to move the count from 0->1 knows who they are, and any threads that arrive before the item is created, and the pointer set, also know who thet are and can wait. Setting the pointer requires a cmp_xchg(), and the creator then discovers how many threads are waiting for same.
This mechanism moves the reference count out of the item, and bundles it with the address of the item, which seems neat enough -- though you now need the address of the item when operating on that, and the address of the reference to the item when you are operating on its reference count.
This replaces the mutex in your ref and unref functions... but does NOT solve the problem of how the reference itself is protected.