Is there a library that implements lock-free algorithms(queue, linked list and others) written in C (not in C++)? I've taken a look at some libraries like Intel's, but I would like to use generic libraries, at least more generic than Intel's one.
See Practical lock-free data structures from the University of Cambridge
I've written my own, Rig, currently queue, stack and list are there, hash-table will soon follow. While I'm still working on it, it is intended for public consumption, and the API is mostly stable, just use the SVN trunk. :)
The only other such library in C that I know of is liblfds, though I've never used it.
liblfds
http://www.liblfds.org
Wiki with full API documentation, forum for questions, blog for reading the author rattle on :-)
Platform independent. Out of the box for Windows, Linux, Intel and ARM.
Release 7 should be out in a month or two. Will add run-time cache line alignment, backoff and SMR. (SMR also gives a ton of the other CPU types - basically, anything GCC compiles on which supports atomic ops, e.g. SPARC, MIPS, IA64, etc).
Also, there's no license - you can use the code however you want. Make money! It's not GPL.
I'm currently writing a lock-free lib but it's C++. Here's an STL-like Lock-Free Doubly-Linked List.
The memory manager it uses is quite powerful (32-bit CAS without ABA issues) so I'm using it to create a complete set of containers: a lock-free map/set (using skip lists), a lock-free bag (instead of queue/stack), and a lock-free unordered map (hash table using split-ordered lists).
For more info about the doubly-linked list check out my answer to a related question.
Related
I'm trying to find a lightweight cooperative threading solution to try implementing an actor model.
As far as I know, the only solution is setcontext/getcontext,
but the functionality is deprecated(?) by Apple. I'm confused by why they did this; however, I'm finding replacement for this.
Pthreads are not an option because I need cooperative model instead of preemptive model to control context switching timing precisely/manually without expensive locking.
-- edit --
Reason of avoiding pthreads:
Because pthreads are not cooperative/deterministic and too expensive. I need actor model for game logic code, so thousand of execution context are required at minimal. Hardware threading requires MB of memory and expense to create/destruct. And parallelism is not important. In fact, I just need concurrent execution of many functions. This can be implemented with many divided functions and some kind of object model, but my goal is reducing those overheads.
If I know something wrong, please correct me. It'll be very appreciated.
The obvious 'lightweight' solution is to avoid complex nested calling except for limited situations where the execution time will be tightly bounded, then store an explicit state structure for each "thread" and implement the main program logic as a state machine that's easily suspendable/resumable at most points. Then you can simply swap out the pointer to the state structure for 'context switch'. Basically this technique amounts to keeping all of your important state variables, including what would conventionally be local variables, in the state structure.
Whether this is worthwhile probably depends on your reason for avoiding pthreads. If your reason is to be portable to non-POSIX systems, or if you really need deterministic program flow, then it may be worthwhile. But if you're just worried about performance overhead and memory synchronization issues, I think you should use pthreads and manage these issues. If you avoid unnecessary locking, use fine-grained locks, and minimize the amount of time locks are held, performance should not suffer.
Edit: Based on your further details posted in the comments on the main question, I think the solution I've proposed is the right one. Each actor should have their own context in which you store the state of the actor's action/thinking/etc. You would have a run_actor function which would take an actor context and a number of "ticks" to advance the actor's state by, and a run_all_actors function which would iterate over a list of active actors and call run_actor for each with the specified number of ticks.
Further, note that this solution still allows you to use real threads to take advantage of SMP/multicore machines. You simply divide the actors up between threads. You may need some degree of locking if one actor needs to examine another's context (e.g. for collision detection).
I was researching this question as well, and I ran across GNU Pth (not to be confused with Pthreads). See http://www.gnu.org/software/pth/
It aims to be a portable solution for cooperative threads. It does mention it is implemented via setcontext/getcontext if available (so it may not be on Mac OSX). Otherwise it says it uses longjmp/setjmp, but it's not clear to me how that works.
Hope this is helpful to anyone who searches for this question.
I have discovered the some of required functionalities from setcontext/getcontext are implemented in libunwind.
Unfortunately the library won't be compiled on Mac OS X because of deprecation of the setcontext/getcontext. Anyway Apple has implemented their own libunwind which is compatible with GNU's implementation at source level. The library is exist on Mac OS X 10.6, 10.7, and iOS. (I don't know exact version in case of iOS)
This library is not documented, but I could find the headers from these locations.
/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS5.0.sdk/usr/include/libunwind.h
/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator4.3.sdk/usr/include/libunwind.h
/Developer/Platforms/iPhoneSimulator.platform/Developer/SDKs/iPhoneSimulator5.0.sdk/usr/include/libunwind.h
/Developer/SDKs/MacOSX10.6.sdk/usr/include/libunwind.h
/Developer/SDKs/MacOSX10.7.sdk/usr/include/libunwind.h
There was a note in the header file that to go GNU libunwind site for documentation.
I'll bet on the library.
Is there any reliable and simple priority queue (linked list preferred, not necessary) implementation for C?
More generally, what C standard libraries do you use?
PQLib (the current accepted answer) is incomplete and the functionality doesn't match the documentation as of this posting. E.g., the pq_dequeue documentation says it returns an entry. The implementation returns NULL. There are many "TO DO" comments in the code, such as, "remove node containing highest priority entry from its heap." Essential logic is missing.
To anyone looking for a priority queue: I recommend finding some code that has good, passing unit tests. I don't recommend PQLib unless it's updated and includes tests.
To the owner of PQLib or anyone recommending it: I assumed this code was complete and spent a fair bit of time debugging until I realized it wasn't, which was frustrating. Please don't recommend code you haven't tried or know to be a work in progress.
The source code accompanying Robert Sedgewick's Algorithms in C, Parts 1-4 (Fundamental Algorithms, Data Structures, Sorting, Searching) contains both a heap-based and a list-based implementation. See Chapter 9 - Priority Queues and Heapsort.
I have a priority queue written in C, hosted on google code. MIT license
https://code.google.com/p/pqueue-heap-c/source/browse/trunk/pqueue.cpp
The code has been used in a few projects so it's solid, but I wrote it in '98 so I don't remember how to use it. Don't be misled by the cpp extension. It's straight C.
Check out PQLib.
I use the standard C standard libraries. ;)
I'm looking for some C library that includes STM-style (Software Transactional Memory) hash maps, but I had no luck so far. It would be great if it was based on glib / gobject, but it's not that crucial. It also doesn't need proper transactions over many objects - single immutable hash support is all I really need.
Must haves: immutable snapshot read, lock-free write with auto-retry.
Wikipedia has a list of various STM implementations.
Well, I think (and there are a number of study) that current STM isn't faster than lock-free and mutex-based code. It is obvious: STM requires on-line data conflict checking. However, such conflict checking in pure software requires very huge time overheads. Currently, only Sun's ROCK processor supports a limited form of STM (Best-effort HTM with STM) by hardware. No x86 CPUs currently support TM in hardware. In short, STM is just slow.
In my opinion, you'd better just using a concurrent hash table. For example, you can find concurrent_hash_map in Intel TBB. Here is the link of TBB Manual. Oh, but it's C++, not C. But, I do believe you can (though it might take significant work) translate C++-based such hash table to C code. Intel TBB is open source.
Also, I think such highly concurrent (usually implemented as lock-free) data structures are not always useful. In some workload pattern, using such data structures is not good. To be sure, I recommend you to write a small micro-benchmark for two versions of hash tables: (1) lock-free and (2) lock-based. Also, please keep in mind the workload patterns for such micro-benchmark should be close to real one. An example could be found in here.
TBoost.STM has implemented a hash map in its example.
I am using C NOT C++!
I know the C++ collections, but I was wondering if Microsoft has a C based List structure of some type, like the linux kernel provides, that I can use in a user mode project?
I would prefer not rolling my own.
Only thing in the Windows API is interlocked singly linked lists, which are used via InterlockedPushEntrySList and InterlockedPopEntrySList .
For device drivers, there is LIST_ENTRY, but I am not sure if this can be pulled into user-mode.
Many algorithms books and websites contain implementations of linked lists that can easily be ported to C. Rolling your own is not too difficult.
reusable collections are tough in C, it just doesn't have the flexibility or metadata (how do you know when you are overflowing this array list and need to reallocate? How do will reallocate work if the rest of the code is using a custom alloc?
You can do it (you CAN do anything in c), but it gets abstract really fast.
On the other hand, creating a linked list in c yourself is downright fun. Arrays are already there, hashes are annoying but not impossible, trees are fun, ...
Also--people who think in c tend to be constantly optimizing. Setting every linked list operation behind a function call instead of just using this=this.next would probably disgust many of them (rightfully so).
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 9 years ago.
Could you recommend a good general purpose data container library for C (pure C, not C++)?
I just need basic structures, like dynamic arrays, hash tables, etc.
(Note: By "good" I mean fast + elegant interface).
Isn't Glib sufficient for your needs ? :-)
developer.gnome.org/glib
I think Dave Hanson's C Interfaces and Implementations qualifies as both fast and elegant. It includes many different kinds of containers, some string processing, multiprecision arithmetic, exceptions, a couple of different memory managers. Nice stuff.
There's a book, but you don't need to buy it to use the software.
Apache Portable Runtime
http://apr.apache.org/
Includes modules for these areas.
Platform Definitions
Internal Memory Allocation
Atomic Operations
Dynamic Object Handling
Functions for manipulating the environment
Error Codes
File Information
File I/O Handling Functions
Filename Matching Functions
Miscellaneous library routines
Command Argument Parsing
Global Locking Routines
Hash Tables
General Purpose Library Routines
MMAP (Memory Map) Routines
Network Routines
Poll Routines
Memory Pool Functions
Portability Routines
Process Locking Routines
Random Functions
Ring Macro Implementations
Shared Memory Routines
Signal Handling
String routines
Internal APR support functions
Table and Array Functions
Condition Variable Routines
Thread Mutex Routines
Threads and Process Functions
Reader/Writer Lock Routines
Time Routines
User and Group ID Services
I previously recommended MemSL, but the website and company seem to have gone extinct. I would now suggest glib, it covers much of the same functionality and is widely supported and used throughout the Linux community.
I used MemSL when implementing an x86 kernel, and I found it to be general, reliable, and bug free.
From the description:
C Overview of the Memory Structures Library (MemSL)
The Memory Structures Library, MemSL for short, is a library of useful routines allowing the efficient use of complex data structures in C. The MemSL contains routines for managing:
Multi-Dimensional Dynamically Allocated Arrays
Single Linked Lists
Double Linked Lists
Circular Linked Lists
Cut, Copy and Paste with Linked Lists
Multiple Positional Pointers to Linked Lists
Stacks
Queues
Dequeues
Sets
Bags
Tables
Dictionaries
Hash Tables with Separate Chaining
Hash Tables with User-Defined Paging
Hash Tables with Dynamic Paging
Binary Search Trees
Threaded Binary Search Trees
AVL Balanced Binary Search Trees
AVL Balanced Threaded Binary Search Trees
Priority Heaps
Fully Dynamic Priority Search Queues
http://home.gna.org/gdsl/
You might also want to check out the iMatix SFL, to quote their webpage:
The SFL (Standard Function Library) from iMatix is a portable function
library for C/C++ programs. The SFL is the result of many years' development,
and is provided as Open Source software for the benefit of the
Internet community.
The SFL is written in ANSI C and has been ported to MS-DOS, Windows, OS/2, Linux
and other UNIX systems (IBM AIX, SunOS, HP/UX, Solaris, NetBSD, FreeBSD,
SCO OpenServer,> Digital UNIX) and Digital OpenVMS. It comes with complete sources and
documentation in HTML.
The SFL provides about 450 functions that cover these areas:
Compression, encryption, and encoding;
Datatype conversion and formatting;
Dates, times, and calendars;
Directory and environment access;
User and process groups;
Inverted bitmap indices;
Symbol tables;
Error message files;
Configuration files;
String manipulation and searching;
File access;
Internet socket access;
Internet programming (MIME, CGI);
SMTP (e-mail) access;
Server (batch) programming;
Program tracing.
The SFL is free software that you may use and distribute for private or commercial
purposes according to the SFL License Agreement.
More details about the API itself, see here.
Sglib "is a generic library for C that was inspired by the Standard Template Library from C++"
Let me add:
http://libslack.org
(... and add to this: libcx)
You may interested in looking at qLibc.
http://en.wikipedia.org/wiki/Qlibc
http://www.qdecoder.org/qlibc/
It's a general purpose complete C/C++ library which includes all kinds of containers and general library routines.