I wanted to test inotify, so took a couple of examples from internet, modified it to learn various aspects, but failed as it didn't work like i wanted to. First i tried to watch over a directory where it worked pretty nicely.
So i extended that example for file with some modification, but it works for one time only and gets blocked on read function
#include <sys/inotify.h>
#include <unistd.h>
#include <stdio.h>
#define EVENT_SIZE (sizeof (struct inotify_event))
#define BUF_LEN (16 * (EVENT_SIZE + 16))
int main()
{
int fd;
fd = inotify_init();
if (fd < 0)
perror("inotify_init()");
int wd;
wd = inotify_add_watch(fd, "target", IN_CLOSE_WRITE);
if (wd < 0)
perror("inotify_add_watch");
char buf[BUF_LEN];
int len;
while(1) {
len = read(fd, buf, BUF_LEN);
printf("after read\n");
if (len > 0)
{
int i = 0;
while (i < len)
{
struct inotify_event *event;
event = (struct inotify_event *) &buf[i];
printf("wd=%d mask=%x cookie=%u len=%u\n",
event->wd, event->mask,
event->cookie, event->len);
if (event->mask & IN_MODIFY)
printf("file modified %s", event->name);
if (event->len)
printf("name=%s\n", event->name);
i += EVENT_SIZE + event->len;
}
}
}
return 0;
}
So, i shifted to select(), but here also, it works once,reports twice then stop reporting the changes.
#include <stdio.h>
#include <stdlib.h>
#include <errno.h>
#include <sys/types.h>
#include <sys/inotify.h>
#define EVENT_SIZE ( sizeof (struct inotify_event) )
#define BUF_LEN ( 1024 * ( EVENT_SIZE + 16 ) )
int main( int argc, char **argv )
{
int length, i = 0;
int fd;
int wd;
char buffer[BUF_LEN];
struct timeval timeout;
fd = inotify_init();
if ( fd < 0 ) {
perror( "inotify_init" );
}
wd = inotify_add_watch( fd, "target",
IN_CLOSE_WRITE );
fd_set rfds,rfdss;
int ret;
/* zero-out the fd_set */
FD_ZERO (&rfds);
FD_ZERO (&rfdss);
FD_SET (fd, &rfds);
timeout.tv_sec = 5;
timeout.tv_usec = 0;
while(1){
printf("Before select\n");
//rfds = rfdss;
ret = select (fd + 1, &rfds, NULL, NULL, NULL);
printf("After Select\n");
timeout.tv_sec = 5;
timeout.tv_usec = 0;
if (ret < 0)
perror ("select");
else if (!ret){
}
/* timed out! */
else if (FD_ISSET (fd, &rfds)){
printf("file changed============\n");
length = read( fd, buffer, BUF_LEN );
}
}
( void ) inotify_rm_watch( fd, wd );
( void ) close( fd );
exit( 0 );
}
A bit of research showed that that the popular editors save it in different way.
Instead of overwriting that file directly, they actually make a temporary file and then replace original file with new temporary one. So what actually happens is the file which you were actually watching no longer exists and therefore any changes that would be making won't be reflected back.
Editors that actually follow this method are(many more may exist)
Gedit, Geany, vi
Editors that directly overwrite the file is(many more may exist)
nano
Hence even though code was correct, anomalous behavior of editor can be problematic
For the blocking read call, see: https://stackoverflow.com/a/914520/149111
You may also want to switch from IN_CLOSE_WRITE to IN_ALL_EVENTS to make sure you're not missing something; likely a delete:
It is a best practice to watch the dir that contains the file(s) of interest rather than the individual files, as this will consume fewer resources in the kernel. It will also allow you to observe "atomic" file replacement operations where the writer creates a temporary file on the file system (perhaps in the same dir), write to the temporary file and then at the end rename(2) it over the top of the original file.
This style of replacement ensures that observers of the file will only ever notice the complete content of the file and not a half-written version of it.
Related
This hashing function I've created (that scans for new files, and computes their hashes) seemingly functions, however once removing a file, for example test.c, and then replacing it with the exact same file it returns 2 different hash values. By this I mean that whilst the program is running the first calculation might return a hash of 1234, for example, and once deleting and placing the same file within the folder the it then returns 2345.
There seems to be no order, as 1234 could be the result 5 times in a row. I wondered whether there's any strikingly obvious reason in this code?
#include <stdio.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <string.h>
#include <errno.h>
#include <sys/inotify.h>
#include <openssl/sha.h>
int main (int argc, char *argv[])
{
int fd;
unsigned char c[SHA512_DIGEST_LENGTH];
int i;
SHA512_CTX mdContext;
int bytes;
unsigned char data[1024];
const int event_size = sizeof(struct inotify_event);
const int buf_len = 1024 * (event_size + FILENAME_MAX);
char *directory = "/home/joe/Documents/";
char *hashDirectory = "/home/joe/Documents/_Hash/";
char hashInBuf[100];
char hashOutBuf[100];
fd = inotify_init();
if (fd < 0) {
perror("inotify_init");
}
while (1) {
char buff[buf_len];
int no_of_events, count = 0;
//SEARCH FOR NEW FILES WITHIN DIRECTORY
no_of_events = read (fd, buff, buf_len);
while (count < no_of_events) {
struct inotify_event *event = (struct inotify_event *)&buff[count];
if (event->len) {
if ((event->mask & IN_CREATE))
if(!(event->mask & IN_ISDIR)) {
printf("\n%s has been created\n", event->name);
//CONJOIN DIRECTORY AND FILENAME / EXTENSION
snprintf(hashInBuf, sizeof(hashInBuf), "%s/%s", directory, event->name);
snprintf(hashOutBuf, sizeof(hashOutBuf), "%s/%s.txt", hashDirectory, event->name);
FILE *ftest=fopen(hashInBuf, "rb");
FILE *ftest2=fopen(hashOutBuf, "wt");
//HASH FUNCTION
SHA512_Init (&mdContext);
while ((bytes = fread (data, 1, 1024, ftest)) != 0)
SHA512_Update (&mdContext, data, bytes);
SHA512_Final (c,&mdContext);
for(i = 0; i < SHA512_DIGEST_LENGTH; i++){
fprintf(ftest2, "%02x", c[i]);
printf("%02x", c[i]);
}
fclose (ftest);
fclose (ftest2);
fflush (stdout);
}
}
count += event_size + event->len;
}
}
return 0;
}
Thank you in advance!
In this line
if ((event->mask & IN_CREATE))
you wait for the event that a file is created. Then, your hashing function immediately starts running!
This may lead to the situation that the file is not fully written yet, so you only hashed a part of the file.
You should use the event IN_CLOSE_WRITE to make sure, that the file has already been completely written.
Another option is to not create the files in this directory, but creating them in a temporary directory and subsequently moving them into the target directory. The corresponding event is IN_MOVED_TO then.
I searched for questions related to inotify, and this one is somewhat different...
I use the following code to monitor change of one file (not directory). In testing, the read() does return when I save the target file which means it works. But event->mask is 32768 which is not IN_MODIFY and name is empty. Another issue: it cannot monitor continuously. When I change the file the second time, it has no response. Thank you for the help!
#include <sys/inotify.h>
#include <unistd.h>
#include <stdio.h>
#define EVENT_SIZE (sizeof (struct inotify_event))
#define BUF_LEN (16 * (EVENT_SIZE + 16))
int main()
{
int fd;
fd = inotify_init();
if (fd < 0)
perror("inotify_init()");
int wd;
wd = inotify_add_watch(fd, "target.txt", IN_MODIFY);
if (wd < 0)
perror("inotify_add_watch");
char buf[BUF_LEN];
int len;
start:
len = read(fd, buf, BUF_LEN);
if (len > 0)
{
int i = 0;
while (i < len)
{
struct inotify_event *event;
event = (struct inotify_event *) &buf[i];
printf("wd=%d mask=%x cookie=%u len=%u\n",
event->wd, event->mask,
event->cookie, event->len);
if (event->mask & IN_MODIFY)
printf("file modified %s", event->name);
if (event->len)
printf("name=%s\n", event->name);
i += EVENT_SIZE + event->len;
}
}
goto start;
return 0;
}
The 0x8000 corresponds to IN_IGNORED. Its presence in the mask indicates that the inotify watch had been removed because the file had been removed. Your editor probably removed the old file and put a new file in its place. Changing the file a second time had no effect because the watch had been removed.
The name is not being returned because you are not watching a directory.
From the inotify man page.
The name field is only present when an event is returned for a file inside a watched directory; it identifies the file pathname relative to the watched directory.
...
IN_IGNORED -- Watch was removed explicitly (inotify_rm_watch(2)) or automatically (file was deleted, or file system was unmounted).
event->mask 32768 is equivalent to 0x8000 that is IN_IGNORED
For more information : "/usr/include/linux/inotify.h"
if (event->mask & IN_IGNORED) {
/*Remove watch*/ inotify_rm_watch(fileDescriptor,watchDescriptor)
/*Add watch again*/ inotify_add_watch
}
I am trying to run an example of inotify in C..but it's not working.
I want to monitor modifications to a file (the file is tmp.cfg), but it doesn't work..I don't know if i'm running it correctly, because I understand how to monitor directories, but not a single file
HereĀ“s the example:
#include <stdio.h>
#include <stdlib.h>
#include <errno.h>
#include <sys/types.h>
#include <sys/inotify.h>
#include <unistd.h>
#define EVENT_SIZE ( sizeof (struct inotify_event) )
#define BUF_LEN ( 1024 * ( EVENT_SIZE + 16 ) )
int main( int argc, char **argv )
{
int length, i = 0;
int fd;
int wd;
char buffer[BUF_LEN];
fd = inotify_init();
if ( fd < 0 ) {
perror( "inotify_init" );
}
wd = inotify_add_watch( fd, "/home/name/tmp.cfg",
IN_MODIFY | IN_CREATE | IN_DELETE );
length = read( fd, buffer, BUF_LEN );
if ( length < 0 ) {
perror( "read" );
}
while ( i < length ) {
struct inotify_event *event = ( struct inotify_event * ) &buffer[ i ];
if ( event->mask & IN_CREATE ) {
printf( "The file %s was created.\n", event->name );
}
else if ( event->mask & IN_DELETE ) {
printf( "The file %s was deleted.\n", event->name );
}
else if ( event->mask & IN_MODIFY ) {
printf( "The file %s was modified.\n", event->name );
}
i += EVENT_SIZE + event->len;
}
( void ) inotify_rm_watch( fd, wd );
( void ) close( fd );
return 0;
}
Once i run it, if i write something on the file and then save it, nothing happens.
i've tryed debugging it..the problem seems to be the if ( event->mask & IN_MODIFY ), as it doesn't recognize it as a modification
You have 2 issues. First, as far as I can tell, inotify does not really work on files - it needs directory name to watch.
Second, you missed if (event->len) { inside while loop.
This code works for me for creating, deleting and modifying files in current directory:
#include <stdio.h>
#include <stdlib.h>
#include <errno.h>
#include <sys/types.h>
#include <sys/inotify.h>
#include <unistd.h>
#define EVENT_SIZE (sizeof(struct inotify_event))
#define BUF_LEN (1024 * (EVENT_SIZE + 16))
int main(int argc, char **argv) {
int length, i = 0;
int fd;
int wd;
char buffer[BUF_LEN];
fd = inotify_init();
if (fd < 0) {
perror("inotify_init");
}
wd = inotify_add_watch(fd, ".",
IN_MODIFY | IN_CREATE | IN_DELETE);
length = read(fd, buffer, BUF_LEN);
if (length < 0) {
perror("read");
}
while (i < length) {
struct inotify_event *event =
(struct inotify_event *) &buffer[i];
if (event->len) {
if (event->mask & IN_CREATE) {
printf("The file %s was created.\n", event->name);
} else if (event->mask & IN_DELETE) {
printf("The file %s was deleted.\n", event->name);
} else if (event->mask & IN_MODIFY) {
printf("The file %s was modified.\n", event->name);
}
}
i += EVENT_SIZE + event->len;
}
(void) inotify_rm_watch(fd, wd);
(void) close(fd);
return 0;
}
It doesn't work on a single file because, when we use a editor to modify file, the editor opens a copy of the file and when we save the edited version from the text editor, the existing file is deleted and a new file of the same name is created with the modifications.
When the old file is deleted, the watch created on that file becomes invalid and it is deleted automatically.
You can see the old file being replaced by the new file if you monitor the parent directory.
There are two ways to solve it, monitor the parent directory and print the message when modifications is done to the particular that you want to watch.
Else create a new watch on the file whenever modifications are made. When the old file is deleted, IN_DELETE_SELF event is triggered.
event->name will be non-empty only when you watch a directory, as it will contain the name of the file on which the event has occurred in the watched directory.
I think you're not using your user name, which is your home directory, and you're not checking the return of inotify_add_watch which probably fails:
"/home/name/tmp.cfg"
Edit: okay second problem, you shouldn't print name because
The name field is only present when an event is returned for a file
inside a watched directory;
Edit2: third problem, the file must exist before you run the program because you add a watch on the file, I suggest you check the error from inotify_add_watch
In watching a file, if the file is manipulated by an editor which you might do to edit it and create a change, it is likely to be doing some operations that results in the original file you asked to watch being deleted. Hence the notifications will stop if you only watch the one file.
I am testing kernel asynchronous io functions (not posix aio) and am trying to figure out how it works. The code below is a complete program where I simply write an array repeatedly to a file opened using O_DIRECT. I get an error in the callback function "write missed bytes expect 1024 got 0" (see the fprintf statement in work_done()).
For those not familiar with kernel aio, the code below does the following:
Init some structs
Prepare aio (io_prep_pwrite)
Submit io requests (io_submit)
Check for event completion (io_getevents)
Call a callback function to see if everything went ok.
I get an error at step 5. If I do not open the file using O_DIRECT, things work fine, but it beats the purpose of having async writes.
Can someone tell me what I am doing wrong? Is this the correct usage of kernel aio, for example, is my use of callbacks correct? Are there any restrictions on the usage of O_DIRECT?
I compile using 'gcc -Wall test.c -laio'
Thanks in advance.
/*
* File: myaiocp.c
* Author: kmehta
*
* Created on July 11, 2011, 12:50 PM
*
*
* Testing kernel aio.
* Program creates a 2D matrix and writes it multiple times to create a file of desired size.
* Writes are performed using kernel aio functions (io_prep_pwrite, io_submit, etc.)
*/
#define _GNU_SOURCE
#define _XOPEN_SOURCE 600
#include <stdio.h>
#include <stdlib.h>
#include <getopt.h>
#include <pthread.h>
#include <fcntl.h>
#include <string.h>
#include <sys/uio.h>
#include <sys/time.h>
#include <omp.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <errno.h>
#include <libaio.h>
char ** buf;
long seg_size;
int seg_rows;
double total_size;
char * filename;
static int wait_count = 0;
void io_task();
void cleanup();
void allocate_2D_matrix(int[]);
int file_open(char *);
void wr_done(io_context_t ctx, struct iocb* iocb, long res, long res2);
int main(int argc, char **argv) {
total_size = 1048576; //1MB
seg_size = 1024; //1kB
seg_rows = 1024;
filename = "aio.out";
int dims[] = {seg_rows, seg_size};
allocate_2D_matrix(dims); //Creates 2D matrix
io_task();
cleanup();
return 0;
}
/*
* Create a 2D matrix
*/
void allocate_2D_matrix(int dims[2]) {
int i;
char *data;
//create the matrix
data = (char *) calloc(1, dims[0] * dims[1] * sizeof (char));
if (data == NULL) {
printf("\nCould not allocate memory for matrix.\n");
exit(1);
}
buf = (char **) malloc(dims[0] * sizeof (char *));
if (buf == NULL) {
printf("\nCould not allocate memory for matrix.\n");
exit(1);
}
for (i = 0; i < dims[0]; i++) {
buf[i] = &(data[i * dims[1]]);
}
}
static void io_error(const char *func, int rc)
{
if (rc == -ENOSYS)
fprintf(stderr, "AIO not in this kernel\n");
else if (rc < 0)
fprintf(stderr, "%s: %s\n", func, strerror(-rc));
else
fprintf(stderr, "%s: error %d\n", func, rc);
exit(1);
}
/*
* Callback function
*/
static void work_done(io_context_t ctx, struct iocb *iocb, long res, long res2)
{
if (res2 != 0) {
io_error("aio write", res2);
}
if (res != iocb->u.c.nbytes) {
fprintf(stderr, "write missed bytes expect %lu got %ld\n",
iocb->u.c.nbytes, res2);
exit(1);
}
wait_count --;
printf("%d ", wait_count);
}
/*
* Wait routine. Get events and call the callback function work_done()
*/
int io_wait_run(io_context_t ctx, long iter)
{
struct io_event events[iter];
struct io_event *ep;
int ret, n;
/*
* get up to aio_maxio events at a time.
*/
ret = n = io_getevents(ctx, iter, iter, events, NULL);
printf("got %d events\n", n);
/*
* Call the callback functions for each event.
*/
for (ep = events ; n-- > 0 ; ep++) {
io_callback_t cb = (io_callback_t)ep->data ; struct iocb *iocb = ep->obj ; cb(ctx, iocb, ep->res, ep->res2);
}
return ret;
}
void io_task() {
long offset = 0;
int bufIndex = 0;
//Open file
int fd = file_open(filename);
//Initialize structures
long i;
long iter = total_size / seg_size; //No. of iterations to reach desired file size (total_size)
io_context_t myctx;
if(0 != io_queue_init(iter, &myctx))
{
perror("Could not initialize io queue");
exit(EXIT_FAILURE);
}
struct iocb * ioq[iter];
//loop through iter times to reach desired file size
for (i = 0; i < iter; i++) {
struct iocb *io = (struct iocb*) malloc(sizeof (struct iocb));
io_prep_pwrite(io, fd, buf[bufIndex], seg_size, offset);
io_set_callback(io, work_done);
ioq[i] = io;
offset += seg_size;
bufIndex ++;
if (bufIndex > seg_rows - 1) //If entire matrix written, start again from index 0
bufIndex = 0;
}
printf("done preparing. Now submitting..\n");
if(iter != io_submit(myctx, iter, ioq))
{
perror("Failure on submit");
exit(EXIT_FAILURE);
}
printf("now awaiting completion..\n");
wait_count = iter;
int res;
while (wait_count) {
res = io_wait_run(myctx, iter);
if (res < 0)
io_error("io_wait_run", res);
}
close(fd);
}
void cleanup() {
free(buf[0]);
free(buf);
}
int file_open(char *filename) {
int fd;
if (-1 == (fd = open(filename, O_DIRECT | O_CREAT | O_WRONLY | O_TRUNC, 0666))) {
printf("\nError opening file. \n");
exit(-1);
}
return fd;
}
First of all, good job using libaio instead of POSIX aio.
Are there any restrictions on the usage of O_DIRECT ?
I'm not 100% sure this is the real problem, but O_DIRECT has some requirements (quoting mostly from TLPI):
The data buffer being transferred must be aligned on a memory boundary that is a multiple of the block size (use posix_memalign)
The file or device offset at which data transfer commences must be a multiple of the block size
The length of the data to be transferred must be a multiple of the block size
At a glance, I can see you are not taking aby precautions to align memory in allocate_2D_matrix.
If I do not open the file using O_DIRECT, things work fine, but it
beats the purpose of having async writes.
This happens not to be the case. Asynchronous I/O works well without O_DIRECT (for instance think of the number of system calls slashed).
I write a program by inotify to check file change in loop for grab any changes to that.
but i think this can be done better.
can anybody write this code better?
#include <stdio.h>
#include <stdlib.h>
#include <errno.h>
#include <sys/types.h>
#include <linux/inotify.h>
#include <sys/select.h>
#define EVENT_SIZE ( sizeof (struct inotify_event) )
#define EVENT_BUF_LEN ( 1024 * ( EVENT_SIZE + 16 ) )
int event_check (int fd)
{
fd_set rfds;
FD_ZERO (&rfds);
FD_SET (fd, &rfds);
/* Wait until an event happens or we get interrupted
by a signal that we catch */
return select (FD_SETSIZE, &rfds, NULL, NULL, NULL);
}
int main( )
{
int length, i = 0;
int fd;
int wd;
while(1){
i=0;
fd = inotify_init();
if ( fd < 0 ) {
perror( "inotify_init" );
}
wd = inotify_add_watch( fd, "/tmp/test", IN_CLOSE_WRITE);
if (event_check (fd) > 0)
{
char buffer[EVENT_BUF_LEN];
int count = 0;
length = read( fd, buffer, EVENT_BUF_LEN );
if ( length < 0 ) {
perror( "read" );
}
while ( i < length ) { struct inotify_event *event = ( struct inotify_event * ) &buffer[ i ];
printf( "New file %s Editted.\n", event->name );
i += EVENT_SIZE + event->len;
}
}
}
inotify_rm_watch( fd, wd );
close( fd );
}
I think you only need one inotify_init for one program and only one inotify_add_watch for one directory.
Can you please paste the "init and add_watch outside the loop" version which you said doesn't work?
I've never used inotify, so its possible that this might be wrong but here's a couple of changes that I think might "improve" your code.
I don't think there's any reason to put either inotify_init or inotify_add_watch inside the loop. Just do this initialisation work once, before the loop.
I'm not sure why you've created the event_check function. You aren't specifying a timeout and you're only using a single file descriptor, so I think read will give you the same funcionality.