How not to open a file twice in linux? - c

Question

I have a linked list with an fd and a string I used to open this file in each entry. I want to open and add files to this list only if this file is not already opened, because I open and parse this files and do not want to do it twice. My idea was to compare the filename with every single name in this list, but my program do it multiple times and one file in Linux can have multiple names (soft/hard links). I think it should not be so complicated, because its easy for the OS to check, whether I already used a inode or not, r?
I already tried to open the same file with and without flock, but I always get a new fd.

Answer 1

When you successfully open a file use fstat on the file. Check to see if the st_ino and st_dev of the struct stat filed in by fstat have already been recorded in your linked list. If so then close the file descriptor and move on to the next file. Otherwise add the file descriptor, the file name and st_ino and st_dev values to the list.
You can instead use stat to check before opening the file, but using fstat after will be slightly faster if the usual case is that file hasn't already been opened.

Answer 2

In situations like this, it's often useful to consider your data structures. Change to a data structure which does not allow duplicates, such as a hash table.
Maintain a set of which data you've seen before. I've used a hash table for this set. As per #RossRidge's answer, use the inode and device as the key. This allows duplicates to be discovered in O(1) time.
Here is an example implementation.
#include <string.h>
#include <stdio.h>
#include <stdlib.h>
#include <glib.h>
#include <sys/stat.h>
#include <errno.h>
#include <fcntl.h>
static int get_fd(GHashTable *fds, const char *filename, int mode) {
int fd;
struct stat stat;
int keysize = 33;
char key[keysize]; /* Two 64 bit numbers as hex and a separator */
/* Resolve any symlinks */
char *real_filename = realpath(filename, NULL);
if( real_filename == NULL ) {
printf("%s could not be resolved.\n", filename);
return -1;
}
/* Open and stat */
fd = open( real_filename, mode );
if( fd < 0 ) {
printf("Could not open %s: %s.\n", real_filename, strerror(errno));
return -1;
}
if( fstat(fd, &stat) != 0 ) {
printf("Could not stat %s: %s.\n", real_filename, strerror(errno));
return -1;
}
/* Make a key for tracking which data we've processed.
This uses both the inode and the device it's on.
It could be done more efficiently as a bit field.
*/
snprintf(key, keysize, "%lx|%lx", (long int)stat.st_ino, (long int)stat.st_dev);
/* See if we've already processed that */
if( g_hash_table_contains(fds, key) ) {
return 0;
}
else {
/* Note that we've processed it */
g_hash_table_add(fds, key);
return fd;
}
}
int main(int argc, char** argv) {
int mode = O_RDONLY;
int fd;
GHashTable *fds = g_hash_table_new(&g_str_hash, &g_str_equal);
for(int i = 1; i < argc; i++) {
char *filename = argv[i];
fd = get_fd(fds, filename, mode);
if( fd == 0 ) {
printf("%s has already been processed.\n", filename);
}
else if( fd < 0 ) {
printf("%s could not be processed.\n", filename);
}
else {
printf("%s: %d\n", filename, fd);
}
}
}
And here's a sample result.
$ touch one two three
$ ln one one_link
$ ln -s two two_sym
$ ./test one* two* three*
one: 3
one_link has already been processed.
two: 5
two_sym has already been processed.
three: 7

Answer 3

As long as you don't close the successfully and intentionally opened files, you can use nonblocking flock to prevent another lock on the same file:
#include <unistd.h>
#include <sys/file.h>
#include <stdio.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <fcntl.h>
#include <assert.h>
int openAndLock(const char* fn){
int fd = -1;
if(((fd = open(fn, O_RDONLY)) >= 0) && (flock(fd, LOCK_EX|LOCK_NB) == 0)){
fprintf(stderr, "Successfully opened and locked %s\n", fn);
return fd;
}else{
fprintf(stderr, "Failed to open or lock %s\n", fn);
close(fd);
return -1;
}
}
int main(int argc, char** argv){
for(int i=1; i<argc; i++){
openAndLock(argv[i]);
}
return 0;
}
Example:
$ touch foo
$ ln foo bar
$ ./a.out foo foo
Successfully opened and locked foo
Failed to open or lock foo
$ ./a.out foo bar
Successfully opened and locked foo
Failed to open or lock bar