I was wondering how the memcached source code is multithread safe in a multicore enviornement, when the manipulation of many of the _stritem or item structs variables don't seem to utilize a lock when manipulated (instead using refcount to indicate how many threads are using the item).
Line: 519 in memcached.h
typedef struct _stritem {
/* Protected by LRU locks */
struct _stritem *next;
struct _stritem *prev;
/* Rest are protected by an item lock */
struct _stritem *h_next; /* hash chain next */
rel_time_t time; /* least recent access */
rel_time_t exptime; /* expire time */
int nbytes; /* size of data */
unsigned short refcount;
uint16_t it_flags; /* ITEM_* above */
uint8_t slabs_clsid;/* which slab class we're in */
uint8_t nkey; /* key length, w/terminating null and padding */
/* this odd type prevents type-punning issues when we do
* the little shuffle to save space when not using CAS. */
union {
uint64_t cas;
char end;
} data[];
/* if it_flags & ITEM_CAS we have 8 bytes CAS */
/* then null-terminated key */
/* then " flags length\r\n" (no terminating null) */
/* then data with terminating \r\n (no terminating null; it's binary!) */
} item;
Line 851 in memcached.h
#define refcount_incr(it) ++(it->refcount)
#define refcount_decr(it) --(it->refcount)
Wouldn't a lack of locking on the item locks when incrementing/decrementing the variable structure in a multithreaded multicore environment potentially create a race condition?
For example: the variable could be fetched by both threads and placed in separate registers, then incremented in both registered, then replaced back in their memory location. This would cause a increment of only 1. Then if 1 thread decremented, couldn't the object be freed and then cause a problem for the other thread?
Edit: To clarify, given memcached has existed for 17 + years the implementation is most likely bug-free, but I am curious as to how memcached handles this.
I would like to add an attribute to the buffer description in the PostgreSQL source code, but when I try to initialize it I get an error of: PANIC: stuck spinlock (0x7fc1cddd0cd0) detected at freelist.c:206
The struct is described in buf_internals.h as:
typedef struct sbufdesc
{
BufferTag tag; /* ID of page contained in buffer */
BufFlags flags; /* see bit definitions above */
uint16 usage_count; /* usage counter for clock sweep code */
unsigned refcount; /* # of backends holding pins on buffer */
int wait_backend_pid; /* backend PID of pin-count waiter */
int buf_age; //<<<<<<<<<<< The age of the buffer
slock_t buf_hdr_lock; /* protects the above fields */
int buf_id; /* buffer's index number (from 0) */
int freeNext; /* link in freelist chain */
LWLockId io_in_progress_lock; /* to wait for I/O to complete */
LWLockId content_lock; /* to lock access to buffer contents */
} BufferDesc;
but it gets stuck at line 206 of freelist.c, which is just:
LockBufHdr(buf);
All I've added was an int to the struct and set it to zero in the same place all the other buffers are initialized. How could this cause a spinlock error?
It looks like running make clean first has corrected the issue.
I'm confused about how the OS actually "opens" a file in C when you do an fopen in the code. To elaborate, suppose I had a 100 binary files (say of size 1 MB) which I open in C
FILE **fptr;
fptr = calloc(100, sizeof(FILE *));
for (ii = 0; ii < 100; ii++)
fptr[ii] = fopen(filename[ii], "rb+");
Assume that filename and ii were already defined appropriately.
Will the OS load 100 MB into memory, or does the above code just tell the program to keep these files ready for access?
The latter, no data is read from the file until needed, i.e. when you call fread() or some other I/O function.
Of course the underlying operating system might decide to speculatively read data when the file is opened, to save time later, but that's outside your control so in effect it doesn't matter. I mean that it doesn't matter because any memory used by such speculative buffering will need to be immediately made available to applications on demand.
That said, it's not as if any practical system will let the fopen() spend the time needed to read 100 MB though, that would be very bad engineering.
Also note that there might be limits on how many files a single process can open in parallel. 100 should be fine for most modern systems, though.
The file is not loaded into memory upon opening it. Instead, parts are loaded in for each read.A call to fopen should not cause reading the file content from media now fread will cause partial read (or complete read for small files) from the media. Partial read usually is equal to cache line size in cache manager.
There are few things we need to look at first
FILE
this is typedefed as below
stdio.h:
typedef struct _IO_FILE FILE;
next _IO_FILE
_IO_FILE looks like below :
libio.h:
struct _IO_FILE {
int _flags; /* High-order word is _IO_MAGIC; rest is flags. */
#define _IO_file_flags _flags
/* The following pointers correspond to the C++ streambuf protocol. */
/* Note: Tk uses the _IO_read_ptr and _IO_read_end fields directly. */
char* _IO_read_ptr; /* Current read pointer */
char* _IO_read_end; /* End of get area. */
char* _IO_read_base; /* Start of putback+get area. */
char* _IO_write_base; /* Start of put area. */
char* _IO_write_ptr; /* Current put pointer. */
char* _IO_write_end; /* End of put area. */
char* _IO_buf_base; /* Start of reserve area. */
char* _IO_buf_end; /* End of reserve area. */
/* The following fields are used to support backing up and undo. */
char *_IO_save_base; /* Pointer to start of non-current get area. */
char *_IO_backup_base; /* Pointer to first valid character of backup area */
char *_IO_save_end; /* Pointer to end of non-current get area. */
struct _IO_marker *_markers;
struct _IO_FILE *_chain;
int _fileno;
#if 0
int _blksize;
#else
int _flags2;
#endif
_IO_off_t _old_offset; /* This used to be _offset but it's too small. */
#define __HAVE_COLUMN /* temporary */
/* 1+column number of pbase(); 0 is unknown. */
unsigned short _cur_column;
signed char _vtable_offset;
char _shortbuf[1];
/* char* _save_gptr; char* _save_egptr; */
_IO_lock_t *_lock;
#ifdef _IO_USE_OLD_IO_FILE
};
and the chain of internal pointers go on, however this file structure by no means keeps the data with itself what this maintains is a snapshot of record and presents to userspace. incase when user requests to read data from file the same FILE * will be used however the data will be retrieved by a syscall (in linux case read call), the offset state however will persisted here itself to go forward , backward. most importantly this provides a very good abstraction to internal system calls.
It's implementation defined. You're not required to read the entire file in memory on fopen(). Most implementations these days use some form of mmap() to actually read the data into memory.
Hi I'm working with OS161 and I'm trying to extend my inode structures so that when I write to disk, I write the first chunk of file data into the actual inode structure, because currently it's set up such that a large chunk of the structure is wasted.
struct sfs_inode {
u_int32_t sfi_size; /* Size of this file (bytes) */
u_int16_t sfi_type; /* One of SFS_TYPE_* above */
u_int16_t sfi_linkcount; /* Number of hard links to this file */
u_int32_t sfi_direct[SFS_NDIRECT]; /* Direct blocks */
u_int32_t sfi_indirect; /* Indirect block */
u_int32_t sfi_waste[128-3-SFS_NDIRECT]; /* unused space */
};
I'd like to replace the sfi_waste above with a char sfi_inlinedata[INLINE_SIZE]; so that
my disk io will always write/read the first INLINE_SIZE of data to the sfs_inode struct.
The following is the source I'm working from for the relevant io functions. I know I need to change the way that the offset maps to actual addresses in the io functions here, but I'm having trouble coming up with a concrete solution.
https://github.com/rbui/projectJailBait/blob/master/os161-1.11/kern/fs/sfs/sfs_vnode.c
Any help would be much appreciated!
How can I get exact total space of a drive using C language program running on Linux? I dont want to use shell script. Any suggestions?
statfs/statfs64
#include <sys/vfs.h> /* or <sys/statfs.h> */
int statfs(const char *path, struct statfs *buf);
int fstatfs(int fd, struct statfs *buf);
From the man page:
The function statfs() returns information about a mounted file system.
path is the pathname of any file within the mounted file system.
buf is a pointer to a statfs structure defined approximately as follows:
struct statfs {
long f_type; /* type of file system (see below) */
long f_bsize; /* optimal transfer block size */
long f_blocks; /* total data blocks in file system */
long f_bfree; /* free blocks in fs */
long f_bavail; /* free blocks avail to non-superuser */
long f_files; /* total file nodes in file system */
long f_ffree; /* free file nodes in fs */
fsid_t f_fsid; /* file system id */
long f_namelen; /* maximum length of filenames */
};
You can use it like this:
struct statfs buf;
statfs("/", &buf);