I was looking at an example in K&R 2 (8.6 Example - Listing Directories). It is a stripped down version of Linux command ls or Windows' dir. The example shows an implementation of functions like opendir, readdir. I've tried and typed the code word-by-word but it still doesn't work. All it does is that it prints the a dot (for the current directory) and exits.
One interesting thing I found in the code (in the implementation of readdir) was that it was calling the system calls like open and read on directory. Something like -
int fd, n;
char buf[1000], *bufp;
bufp = buf;
fd = open("dirname", O_RDONLY, 0);
n = read(fd, bufp, 1000);
write(fd, bufp, n);
When I run this code I get no output even when the folder name "dirname" has some files in it.
Also, the book says, that the implementation is for Version 7 and System V UNIX systems. Is that the reason why it is not working on Linux?
Here is the code- http://ideone.com/tw8ouX.
So does Linux not allow read system calls on directories? Or something else is causing this?
In Version 7 UNIX, there was only one unix filesystem, and its directories had a simple on-disk format: array of struct direct. Reading it and interpreting the result was trivial. A syscall would have been redundant.
In modern times there are many kinds of filesystems that can be mounted by Linux and other unix-like systems (ext4, ZFS, NTFS!), some of which have complex directory formats. You can't do anything sensible with the raw bytes of an arbitrary directory. So the kernel has taken on the responsibility of providing a generic interface to directories as abstract objects. readdir is the central piece of that interface.
Some modern unices still allow read() on a directory, because it's part of their history. Linux history began in the 90's, when it was already obvious that read() on a directory was never going to be useful, so Linux has never allowed it.
Linux does provide a readdir syscall, but it's not used very much anymore, because something better has come along: getdents. readdir only returns one directory entry at a time, so if you use the readdir syscall in a loop to get a list of files in a directory, you enter the kernel on every loop iteration. getdents returns multiple entries into a buffer.
readdir is, however, the standard interface, so glibc provides a readdir function that calls the getdents syscall instead of the readdir syscall. In an ordinary program you'll see readdir in the source code, but getdents in the strace. The C library is helping performance by buffering, just like it does in stdio for regular files when you call getchar() and it does a read() of a few kilobytes at a time instead of a bunch of single-byte read()s.
You'll never use the original unbuffered readdir syscall on a modern Linux system unless you run an executable that was compiled a long time ago, or go out of your way to bypass the C library.
In fact Linux dosn't allow read for directories. See man page and search for errno EISDIR. You will find
The read() and pread() functions shall fail if ...
The fildes argument refers to a directory and the implementation does not allow the directory to be read using read() or pread(). The readdir() function should be used instead.
. Other UNIXes allow it nevertheless.
Related
This question already has answers here:
Where can I find the source code for all the C standard libraries?
(4 answers)
Closed 11 months ago.
I am trying to learn some C, but I am finding some of the standard functions a bit opaque.
Take putc or putchar as an example. I am trying to work out what drives this at the most basic level. I have tried to follow their definitions back through the GNU compiler source but it just ends up in this enormous tree of source files.
Is there a primitive "print this character" function that all the others are built from? I had assumed that it was just a write() system call, but an answer to a related question said that this is completely implementation specific. So how else can it actually produce the output if not a system call?
So how else can it actually produce the output if not a system call?
It does use a system call, but the specific system call is implementation-dependent.
Unix implementations use the write() system call. Implementations for other operating systems will use whatever is analogous to this.
There could also be standalone implementations that run directly on hardware without an operating system. These "unhosted" implementations might omit the stdio library, or they could implement its features by accessing the hardware directly. In this case there's no system call, the I/O is done by the stdio library itself.
For Unix based systems for which Linux is part, most functions in stdio library are wrappers that are one layer above the standard I/O system calls. You see, the operating system provides a set of APIs called system calls. Applications cannot directly access hardware resources and hence they usually call these "system calls" whenever they need to do any sort of privileged thing like writing to the screen or reading from the keyboard.
In Unix, everything is abstracted as a file so whenever you need to write characters to a screen, all you need to do is open some file that represents the "screen" and write those characters there. The kernel will take care of the rest. Quite simply, this is how you'd do this in C:
#include <unistd.h>
#include <fcntl.h>
#include <stdlib.h>
#define BUFF_SIZE 2
int main()
{
int terminal;
char buffer[BUFF_SIZE] = "C\n"; // This will store the character to print + new line
terminal = open("/dev/tty", O_WRONLY); // systemcall to open terminal
if (terminal < 0)
exit(1); // some error happened
dup2(terminal, STDOUT_FILENO); // use our newly opened terminal as default Standard output
if (write(terminal, buffer, BUFF_SIZE) != BUFF_SIZE) // systemcall to write to terminal
exit(1); // We couldn't write anything
}
This just goes to show you that everything in stdio is layered on top of the basic I/O system calls. These system calls are read, write, open, etc. If you want to learn more about system calls and some OS internals, read the book "Three Easy Pieces" by Andrea Arpaci-Dusseau
How exactly _fsopen() works? Does Linux also has similar way of opening files which prepares the file for subsequent shared reading or writing based on shflag?
Referred article here.
How exactly _fsopen() works?
You've linked to the docs. It does what they say it does. If you're asking how it is implemented then we cannot answer because that information is proprietary.
and Does linux also has similar way of opening files which prepares the file for subsequent shared reading or writing based on shflg?
Linux does not have share modes. That's a Windows quirk. Under Linux or other Unix-like operating systems such as macOS, you don't need special flags or modes to share files between processes.
Overall, _fsopen() is an MS-specific variant of the C standard library's fopen() function. In addition to the share-mode flag, which is not relevant to other operating systems, it performs parameter validation in the manner of various other MS extension functions. On Linux, one takes responsibility for validating one's own arguments and simply uses fopen().
On Windows files are opened using the CreateFileW function which uses the NtCreateFile system call.
Argument dwShareMode is used to specify file sharing policy and contains combination of flags FILE_SHARE_DELETE, FILE_SHARE_READ and FILE_SHARE_WRITE which are mapped to shflag argument of _fsopen.
If you want to know how possible implementation of the function can look like, then first you should keep in mind that MSVCRT tries to support to some equivalent of POSIX file descriptor API. Then check the following functions:
_open_osfhandle allows you to convert NT HANDLE to POSIX-like file descriptor
_fdopen allows you to get a FILE * from a file descriptor (equivalent of POSIX fdopen function).
So the possible implementation can look like this (in pseudo code):
FILE *_fsopen(...)
{
HANDLE hFile = CreateFileW(...);
int fd = _open_osfhandle(hFile, ...);
return _fdopen(fd, ...);
}
Linux doesn't provide an equivalent of file sharing policy, so there is no equivalent.
PS: Another related function is _wsopen - combines CreateFileW and _open_osfhandle.
I was looking to implement the behavior of Linux command cp -Rf <src_dir>/* <dst_dir> that copies everything inside the 'src_dir' into 'dst_dir' recursively
I looked online for help and got a few solutions but they did either of the following that I do not want to do:
using rename(src_dir, dst_dir), which essentially 'moves' the contents and not copies.
I need to keep the contents of the 'src_dir' intact.
Opening to read each file int the 'src_dir'.
I would like to do this w/o opening the files and reading the content.
Can the above be achieved with C without using system("cp -Rf <src_dir>/* <dst_dir>")?
(Edited the original question after realising from a few initial comments that the original lang was confusing)
I was looking to implement the behavior of linux system commands
1) cp that copies everything inside the 'src_dir' into 'dst_dir' recursively,
Can the above be achieved using C (and no linux system calls)?
Of course not; remember that a system call is the main way an application can interact with the Linux kernel: look on syscalls(2) for an exhaustive list of system calls (you are likely to need stat(2)...). Do not confuse system calls with calls to the C library system(3) function (which uses fork(2), execve(2), waitpid(2) etc....)
However, you might be interested in nftw(3) (which of course is implemented above several system calls). Look also into opendir(3) & readdir(3)
You cannot change anything on the file systems without going thru system calls.
addenda
(the question has changed to)
Can the above be achieved with C without using system("cp -Rf /* ")?
Yes; you could use nftw(3) first to scan the file tree (or play appropriately with readdir(3) etc...), then detect the directories to be made (using mkdir(2)...); you'll make appropriately the directories and you'll explicitly copy file contents (e.g. using stdio(3) functions or open(2), read(2), write(2), close(2) system calls....). Notice that open(2) (called by fopen(3)...) is required to read or write any content from a file (even using mmap(2)...) Of course, to copy a file, you need to open it, open its fresh copy, read it and write to the copy in a loop, and close both source and destination files. But you should close (both source & destination files) once the copy is done. BTW, a given process could have several hundreds (or thousands of) open file descriptors (see setrlimit(2)...) and opendir(3) also consume one.
(your number of opened file descriptors (most of them being directories, with opendir(3)) is bounded by the file tree depth so is practically likely to be small, less than a hundred; and using clever caching with nftw(3) you could even avoid that)
BTW, cp is part of GNU coreutils which is free software, so you can study its source code. And you might also use strace(1) to understand what cp is doing.
Read Advanced Linux Programming and also Operating System : Three Easy Pieces (both are freely downloadable, as chapters in PDF)
There is no way to copy the content of one file with a single system call. You need a loop. But look also into the Linux specific copy_file_range(2) which I don't recommend, better loop on POSIX read(2) and write(2)
For example, in C, we have fopen, and in Unix, we have open. There are some subtle differences between them, but they are doing the same thing.
There are also many other functions that both existing in C and Unix, what is the relationship between them? Which one should I prefer?
open is a system call from Unix systems.
fopen is the standard c function to open a file.
There's some advantages of using fopen rather than open.
It's mult-platform, as it's C standard, you can port your program to any platform with a C compiler.
It supports use of C standard functions, (i.e: fprintf, fscanf)
If you are handling with text files, those functions can deal with different new lines characters (Unix/Windows)
fopen(3) is returning a FILE* on success, but open(2) is returning a file descriptor on success, so they are not doing the same (since not giving the same type).
However, on Linux, fopen is internally using the open system call (and some others too...).
<stdio.h> file handles are handling buffering. With system calls like open and read you'll better do your own buffering.
See also this & that and read Advanced Linux Programming & syscalls(2). Be aware that on Linux, from the user-land application point of view, a system call is essentially an atomic elementary operation (e.g. the SYSCALL or SYSENTER machine instruction).
Use strace(1) to find out which system calls are executed (by a given process or command).
On Linux, the libc is implementing standard functions (like fprintf ....) above system calls.
Many system calls don't have any libc counterpart (except their wrapper), e.g. poll(2)
I am working on an embedded system with no filesystem and I need to execute programs that take input data from files specified via command like arguments or directly from stdin.
I know it is possible to bake-in the file data with the binary using the method from this answer: C/C++ with GCC: Statically add resource files to executable/library but currently I would need to rewrite all the programs to access the data in a new way.
Is it possible to bake-in a text file, for example, and access it using a fake file pointer to stdin when running the program?
If your system is an OS-less bare-metal system, then your C library will have "retargetting" stubs or hooks that you need to implement to hook the library into the platform. This will typically include low-level I/O functions such as open(), read(), write(), seek() etc. You can implement these as you wish to implement the basic stdin, stdout, stderr streams (in POSIX and most other implementations they will have fixed file descriptors 0, 1 and 2 respectively, and do not need to be explicitly opened), file I/O and in this case for managing an arbitrary memory block.
open() for example will be passed a file or device name (the string may be interpreted any way you wish), and will return a file descriptor. You might perhaps recognise "cfgdata:" as a device name to access your "memory file", and you would return a unique descriptor that is then passed into read(). You use the descriptor to reference data for managing the stream; probably little more that an index that is incremented by the number if characters read. The same index may be set directly by the seek() implementation.
Once you have implemented these functions, the higher level stdio functions or even C++ iostreams will work normally for the devices or filesystems you have supported in your low level implementation.
As commented, you could use the POSIX fmemopen function. You'll need a libc providing it, e.g. musl-libc or possibly glibc. BTW for benchmarking purposes you might install some tiny Linux-like OS on your hardware, e.g. uclinux