C - Inaccurate write() position obtained with lseek() when using multiple threads - c

Question

I'm having a problem getting the correct file position at which I'm writing when simultaneously writing to different parts of the same file using multiple threads.
I have one global file descriptor to the file. In my writing function, I
first lock a mutex, then do lseek(global_fd, 0, SEEK_CUR) to get the current file
position. I next write 31 zero bytes (31 is my entry size) using write(), in effect to reserve space for later. I then unlock the mutex.
Later in the function, I declare a local fd variable to the same file, and open
it. I now do an lseek on that local fd to get to the position I learned from
earlier, where my space is reserved. Finally, I write() 31 data bytes there for
the entry, and close the local fd.
The issue seems to be that rarely, an entry doesn't get written to the expected location (it's not mangled data - it seems that either it is swapped with a different entry, or two entries were written to the same location). There are multiple threads running that
"writing function" I described.
I since learned that pwrite() can be used to write to a specific offset, which would be more efficient, and eliminate the lseek(). However, I first want to find out: what is wrong with my original algorithm? Is there any type of buffering that could be causing the discrepancy between the expected write location, and where the data actually ends up getting stored in the file?
The relevant code snippet is below. The reason this is an issue is that in a second data file, I record the location where the entry I'm writing will be stored. If that location, based on the lseek() before the write, is not accurate, my data doesn't match up properly -- which is what happens on occasion (it's hard to reproduce - it happens in maybe 1 in 100k writes). Thanks!
db_entry_add(...)
{
char dbrecord[DB_ENTRY_SIZE];
int retval;
pthread_mutex_lock(&db_mutex);
/* determine the EOF index, at which we will add the log entry */
off_t ndb_offset = lseek(cfg.curr_fd, 0, SEEK_CUR);
if (ndb_offset == -1)
{
fprintf(stderr, "Unable to determine ndb offset: %s\n", strerror_s(errno, ebuf, sizeof(ebuf)));
pthread_mutex_unlock(&db_mutex);
return 0;
}
/* reserve entry-size bytes at the location, at which we will
later add the log entry */
memset(dbrecord, 0, sizeof(dbrecord));
/* note: db_write() is a write() loop */
if (db_write(cfg.curr_fd, (char *) &dbrecord, DB_ENTRY_SIZE) < 0)
{
fprintf(stderr, "db_entry_add2db - db_write failed!");
close(curr_fd);
pthread_mutex_unlock(&db_mutex);
return 0;
}
pthread_mutex_unlock(&db_mutex);
/* in another data file, we now record that the entry we're going to write
will be at the specified location. if it's not (which is the problem,
on rare occasion), our data will be inconsistent */
advertise_entry_location(ndb_offset);
...
/* open the data file */
int write_fd = open(path, O_CREAT|O_LARGEFILE|O_WRONLY, 0644);
if (write_fd < 0)
{
fprintf(stderr, "%s: Unable to open file %s: %s\n", __func__, cfg.curr_silo_db_path, strerror_s(errno, ebuf, sizeof(ebuf)));
return 0;
}
pthread_mutex_lock(&db_mutex);
/* seek to our reserved write location */
if (lseek(write_fd, ndb_offset, SEEK_SET) == -1)
{
fprintf(stderr, "%s: lseek failed: %s\n", __func__, strerror_s(errno, ebuf, sizeof(ebuf)));
close(write_fd);
return 0;
}
pthread_mutex_unlock(&db_mutex);
/* write the entry */
/* note: db_write_with_mutex is a write() loop wrapped with db_mutex lock and unlock */
if (db_write_with_mutex(write_fd, (char *) &dbrecord, DB_ENTRY_SIZE) < 0)
{
fprintf(stderr, "db_entry_add2db - db_write failed!");
close(write_fd);
return 0;
}
/* close the data file */
close(write_fd);
return 1;
}
One more note, for completeness. I have a similar but simpler routine that could also be causing the problem. This one uses buffered output (FILE*, fopen, fwrite), but performs an fflush() at the end of each write. It writes to a different file than the earlier routine, but could cause the same symptom.
pthread_mutex_lock(&data_mutex);
/* determine the offset at which the data will be written. this has to be accurate,
otherwise it could be causing the problem */
offset = ftell(current_fp);
fwrite(data);
fflush(current_fp);
pthread_mutex_unlock(&data_mutex);

Answer 1

There seem to be several places where things could go wrong. I would make the following changes: (1) be consistent and use the same I/O library as per bdonlan's suggestion, (2) make the lseek() and the writes an atomic action guarded by a mutex so that only a single thread at a time can do those actions of adding to both files. SEEK_CUR does a seek based on the current location of the file offset pointer so would you not want SEEK_END to seek to the end of the file in order to append there? Then if you are modifying a particular section of the file you would use SEEK_SET to reposition to the location you want to write to. And you would want to do this in a mutex guarded section so as to allow only a single thread to do the file positioning and file update.

Answer 2

If you're using your 'simpler routine' at the same time, this could indeed be a problem. If these are separate file descriptors, there's nothing to ensure that they're both pointing at the end of the file at all times (unless you use append mode, however I'm not sure what the semantics around ftell for append mode are). If they're the same fd (ie, you have a raw fd and a FILE * pointing to the same place), you might have problems with the standard library getting confused about where you are in a file, when you use write() to bypass it.