I am currently thinking of implementing a syscall in some BSD flavours in order to close a given file descriptor.
The file descriptor would be defined as a pair of PID and file descriptor number.
It will be useful in order to test/debug a program or others strange purposes.
I think that I will do it anyway, you know, for learning purpose.
What I'm asking here is: can it be useful to someone somehow ? Can I publish my work and maintain it ?
I don't think any operating system will accept my code if there's no need for the end users or programmers.
Thanks for your advices.
Come on, that would be useless, it's like taking back memory from a program while it's running. This will never happen in reality.
Code it just for the fun if you really want to, no one will ever need that.
Or maybe I'm missing something ?
You could post it as a tutorial for kernel programming stuff. There aren't so much of them out there, and it's not as documented as one could expect.
Related
I'm trying to write a C program that is able to test the performance of other programs by passing in input and testing the output without having to restart the program every time it runs. Co-workers and I are writing sudoku solvers, and I'm writing the program to test how fast each one runs by solving numerous puzzles, which could all be in different languages, and I don't want to penalize people for using languages, like Java, that are really slow to start up. Ideally, this program will start the sudoku solver program, keep it running, and continually pass in a new puzzle via stdin and test the output in stdout.
Here's pseudocode of what I want to do:
start a sudoku solver in another process
once process is running
pass puzzle string into child stdin
wait until output comes into stdout
repeat until end time limit ends
close process
I've messed around with popen, but I couldn't figure out how to write to the child process stdin. I've done a bunch of poking around the internet, and I haven't been able to figure it out.
Any suggestions on how to accomplish this? I'm running this on a Linux box. It doesn't have to be stdin and stdout for communication, but that would be the easiest for everyone else.
This is more a long comment than an answer, but your question is really too broad and ill-defined, and I'm just giving some hints.
You first need to understand how to start, manage, and communicate with child processes. An entire Unix programming book is needed to explain that. You could read ALP or some newer book. You need to be able to write a Unix shell-like program. Become familiar with many syscalls(2) including fork(2), pipe(2), execve(2), dup2(2), poll(2), waitpid(2) and a dozen others. See also signal(7) & time(7).
You also need to discuss with your colleagues some conventions and protocol about these sudoku programs and how your controlling program would communicate with them (and the evil is in the details). For example, your pseudo-code is mentioning "pass puzzle string" but you don't define what that exactly means (what if the string contains newlines, or weird characters?). Read also about inter-process communication.
(You might want to have more than one sudoku process running. You probably don't want a buggy sudoku client to break your controlling program. This is unclear in your question)
You could want to define a text-based protocol (they are simpler to debug and use than binary protocols). Details matter a lot, so document it precisely (probably using some EBNF notation). You might want to use textual formats like JSON, YAML, S-expressions. You could take inspiration from SMTP, HTTP, JSONRPC etc (or perhaps choose to use one of them).
Remember that pipe(7)-s, fifo(7)-s and tcp(7)-s socket(7)-s are just a stream of bytes without any message boundaries. Any message organization above these should be a documented convention (and it might happen that the message would be fragmented, so you need careful buffering). See also this.
(I recommend making some free software sample implementation of your protocol)
Look also into similar work, perhaps SAT competition (or chess contests programs, I don't know the details).
Read also something about OSes, like Operating Systems: Three Easy Pieces
I want to implement a C program in Linux (Ubuntu distro) that mimics tail -f. Note that I do not want to actually call tail -f from my C code, rather implement its behaviour. At the moment I can think of two ways to implement it.
When the program is called, I seek to the end of file. Afterwards, I would read to the end of file periodically and print whatever I read if it is not empty.
The second method which can potentially be more efficient is to again, seek to the end of file. But, this time I "somehow" listen for changes to that file and read to the end of file, only if I it is changed.
With that being said, my question is how to implement the second approach and if someone can share if it is worth the effort. Also, are these the only two options?
NOTE: Thanks for the comments, the question is changed based on them.
There is no standardized mechanism for monitoring changes to a file, so you'll need to implement a "polling" solution anyway (that is, when you hit the end of file, wait a short amount of time and try again.)
On Linux, you can use the inotify family of system calls, but be aware that it won't always work. It doesn't work for special files or remote filesystems, for example, and it may not work for some local filesystems. It is complicated in the case of symlinks. And so on. There is a Windows equivalent, but I believe it suffers from some of the same issues.
So even if you use a notification system, you'll need the polling solution as a backup, and since OS notifications are not guaranteed to be reliable (that is, if the system is under load, notifications might be dropped), you'll need to poll on timeout even if you are using a notification system.
You might want to take a look at the implementation of the GNU tail utility (http://git.savannah.gnu.org/cgit/coreutils.git/tree/src/tail.c) to see how the special cases are handled.
You can implement the requirement by following steps:
1) fopen with 'a+' mode;
2) select the file discriptor opened (need do convert from FILE * to file descriptor) and do the read.
I'm trying to make a simple reader/writer program using POSIX named semaphores, its working, but on some systems, it halts immediately on the first semaphore and thats it ... I'm really desperate by now. Can anyone help please? Its working fine on my system, so i can't track the problem by ltrace. (sorry for the comments, I'm from czech republic)
https://www.dropbox.com/s/hfcp44u2r0jd7fy/readerWriter.c
POSIX semaphores are not well suited for application code since they are interruptible. Basically any sort of IO to your processes will mess up your signalling. Please have a look at this post.
So you'd have to be really careful to interpret all error returns from the sem_ functions properly. In the code that you posted there is no such thing.
If your implementation of POSIX supports them, just use rwlocks, they are made for this, are much higher level and don't encounter that difficulty.
In computer science, the readers-writers problems are examples of a common computing problem in concurrency. There are at least three variations of the problems, which deal with situations in which many threads try to access the same shared memory at one time. Some threads may read and some may write, with the constraint that no process may access the share for either reading or writing, while another process is in the act of writing to it. (In particular, it is allowed for two or more readers to access the share at the same time.) A readers-writer lock is a data structure that solves one or more of the readers-writers problems.
I have to work with two C programs that communicate via a file-based interface. That is, each of them has a main loop where it polls three or four files (fopen, fscanf), reacts to what it reads and eventually makes its own changes to the files (fprintf) for the other process to read.
Now I have to condense these two programs into a single program, with minimal changes to the program logic and the code in general. However, mainly for aesthetic reasons I'm supposed to replace the file-based communication with something in-memory.
I can imagine a few hacky ways to accomplish this, but I'm sure that stackoverflow will give me a hint at a beautiful solution :)
Since you tagged this Linux, I'm going to suggest open_memstream. It was added to POSIX with POSIX 2008, but it's been available on glibc-based Linux systems for a long time. Basically it lets you open a FILE * that's actually a dynamically-growing buffer in memory, so you wouldn't have to change much code. This "file" is write-only, but you could simply use sscanf instead of fscanf on the buffer to read it, or use fmemopen (which doesn't have the dynamic-growth semantics but which is very convenient for reading from in-memory buffers).
RabbitMQ is a really robust/elegant solution for event processing. After mucking with state machines for the past few years this has been a breath of fresh air. There are other messaging servers with C libs like OPenAMQ.
Since you tagged this Linux, I'd suggest putting the communication files on /dev/shm. That way you sort-of replace the file-based communication with an in-memory one, without actually altering any of the application logic :-)
You say that you have condensed the reader / Writer Processes into a single Program.
So, now you have different threads for the purpose?
If so, i think a mutex-guarded global buffer should serve the purpose well enough.
Use a global string with sscanf and sprintf instead of a file.
My application uses lseek() to seek the desired position to write data.
The file is successfully opened using open() and my application was able to use lseek() and write() lots of times.
At a given time, for some users and not easily reproducable, lseek() returns -1 with an errno of 9. File is not closed before this and the filehandle (int) isn't reset.
After this, another file is created; open() is okay again and lseek() and write() works again.
To make it even worse, this user tried the complete sequence again and all was well.
So my question is, can the OS close the file handle for me for some reason?
What could cause this? A file indexer or file scanner of some sort?
What is the best way to solve this; is this pseudo code the best solution?
(never mind the code layout, will create functions for it)
int fd=open(...);
if (fd>-1) {
long result = lseek(fd,....);
if (result == -1 && errno==9) {
close(fd..); //make sure we try to close nicely
fd=open(...);
result = lseek(fd,....);
}
}
Anybody experience with something similar?
Summary: file seek and write works okay for a given fd and suddenly gives back errno=9 without a reason.
So my question is, can the OS close the file handle for me for some reason? What could cause > this? A file indexer or file scanner of some sort?
No, this will not happen.
What is the best way to solve this; is
this pseudo code the best solution?
(never mind the code layout, will
create functions for it)
No, the best way is to find the bug and fix it.
Anybody experience with something similar?
I've seen fds getting messed up many times, resulting in EBADF in the some of the cases,
and blowing up spectacularly in others, it's been:
buffer overflows - overflowing something and writing a nonsense value into a 'int fd;' variable.
silly bugs that happen because some corner case someone did
if(fd = foo[i].fd) when they meant if(fd == foo[i].fd)
Raceconditions between threads, some thread closes the wrong file descriptor that some other thread wants to use.
If you can find a way to reproduce this problem, run your program under 'strace', so you can see whats going on.
The OS shall not close file handles randomly (I am assuming a Unix-like system). If your file handle is closed, then there is something wrong with your code, most probably elsewhere (thanks to the C language and the Unix API, this can be really anywhere in the code, and may be due to, e.g., a slight buffer overflow in some piece of code which really looks like to be unrelated).
Your pseudo-code is the worst solution, since it will give you the impression of having fixed the problem, while the bug still lurks.
I suggest that you add debug prints (i.e. printf() calls) wherever you open and close a file or socket. Also, try Valgrind.
(I just had yesterday a spooky off-by-1 buffer overflow, which damaged the least significant byte of a temporary slot generated by the compiler to save a CPU register; the indirect effect was that a structure in another function appeared to be shifted by a few bytes. It took me quite some time to understand what was going on, including some thorough reading of Mips assembly code).
I don't know what type of setup you have, but the following scenario, could I think produce such an effect (or else one similar to it). I have not tested this to verify, so please take it with a grain of salt.
If the file/device you are opening implemented as a server application (eg NFS), consider what could happen if the server application goes down / restarts / reboots. The file descriptor though originally valid at the client end might no longer map to a valid file handle at the server end. This can conceivably lead to a sequence of events wherein the client will get EBADF.
Hope this helps.
No, the OS should not close file handles just like that, and other applications (file scanners etc.) should not be able to do it.
Do not work around the problem, find it's source. If you don't know what the reason for your problem was, you will never know if your workaround actually does work.
Check your assumptions. Is errno set to 0 before the call? Is fd really valid at the point the call is being made? (I know you said it is, but did you check it?)
What is the output of puts( strerror( 9 ) ); on your platform?