I've wanted to write a kernel for some time now. I already have a sufficient knowledge of C and I've dabbled in x86 Assembler. You see, I've wanted to write a kernel that is POSIX-compliant in C so that *NIX applications can be potentially ported to my OS, but I haven't found many resources on standard POSIX kernel functions. I have found resources on the filesystem structure, environment variables, and more on the Open Group's POSIX page.
Unfortunately, I haven't found anything explaining what calls and kernel functions a POSIX-compliant kernel must have (in other words, what kind of internal structure must a kernel have to comply with POSIX). If anyone could find that information, please tell me.
POSIX doesn't define the internal structure of the kernel, the kernel-to-userspace interface, or even libc, at all. Indeed, even Windows has a POSIX-compliant subsystem. Just make sure the POSIX interfaces defined at your link there work somehow. Note, however, that POSIX does not require anything to be implemented specifically in the kernel - you can implement things in the C library using simpler kernel interfaces of your own design where possible, if you prefer.
It just so happens that a lot of the POSIX compliant OSes (BSD, Linux, etc) have a fairly close relationship between many of those calls and the kernel layer, but there are exceptions. For example, on Linux, a write() call is a direct syscall, invoking a sys_write() function in the kernel. However on Windows, write() is implemented in a POSIX support DLL, which translates the file descriptor to a NT handle and calls NtWriteFile() to service it, which in turn invokes a corresponding system call in ntoskrnl.exe. So you have a lot of freedom in how to do things - which makes things harder, if anything :)
The opengroup.org leaves the decisions about kernel syscalls to each implmentation.
write(), for example has to look and behave as stated, but what it calls underneath is not defined. A lot of calls like write, read, lseek are free to call whatever entrypoint they want inside the kernel.
So, no, there really is nothing that says you have to have a certain function name with a defined set of semantics available in the kernel. It just has to available in the C runtime library.
Related
We know that Linux kernel is written in C. But does it also call standard C functions like malloc() or extra functions like mmap() which are provided by GNU C library (glibc)? In that case, it's strange, because direct low-level interaction with hardware, e.g. memory management, is supposed to be almost always the task of a kernel. So, which is dependent on the other? Which is more fundamental/low-level?
We know that Linux kernel is written in C. But does it also call standard C functions like malloc()
No. However, the kernel defines similar functions like kmalloc. Note this is not part of a library; it's part of the kernel itself.
or extra functions like mmap()
Not mmap, but there are a lot of memory management functions in the kernel.
which are provided by GNU C library (glibc)?
Definitely not. The kernel does not use glibc ever.
So, which is dependent on the other?
Some parts of glibc depends on the kernel. Other parts (like strcpy) have nothing to do with the kernel and don't depend on it. The kernel never depends on glibc. You can run programs on Linux that use a different libc (like "musl") or that don't use a libc at all.
I know the function to create socket:int socket(int domain, int type, int protocol); is located in #include <sys/socket.h> and I can find it on my linux file system.
But where I can find the implementation of this function? I couldn't find a matching one in the kernel source.
Look on kernel.org for the authentic kernel source. Understand that socket(2) is one of the many syscalls(2) (you need to understand precisely what system calls are) so it is implemented inside the kernel as sys_socket and/or sys_socketcall and/or do_socket; sockets and network code are an entire subsystem (net/) of the kernel, so see its net/socket.c, etc etc... See also socketcall(2)
Application user-side code are simply issuing a syscall, so socket(2) is a thin wrapper around a system call (in GNU libc or musl-libc or whatever implementation of the C standard library you are using). See also this.
sys/socket.h should be in /usr/include. Typically, it's a part of the "GNU C Library", but depending on your system, that might also be a different C library (ie. on systems you can't call GNU/Linux, like Android, there might be different libc than glibc).
However, that's just the header, not the implementation of the syscalls beneath! You will have to look through glibc's source code (which usually is not installed, only the headers), and then match what you find there to the system calls implementation in linux.
if you want to check the linux implementation of socket creation, you. can look here
I have wrapped a number of system call function like write(), open() etc and LD-PRELOAD is used to override the original system calls. Moreover I have defined a few more functions and made this too a shred library.
I would like to catch all system calls from different application processes to these shared libraries before they enter the shared library. How can i do that?
Thanks
LD_PRELOAD is not necessarily a good way to interpose system calls, because a) it only allows you to intercept library calls and b) it only allows you to intercept library calls. ;)
A) While in general, system calls are wrapped by the shared libC in your system, no one prevents you from calling a system call yourself, e.g., but setting up the right register content and then issuing INT 0x80 on an x86 system. If the program you're interested in does so, you'll never catch those with LD_PRELOAD-based libc-interposition.
B) While in general, most programs use the shared libC in your system to make system calls, sometimes applications are linked statically, which means the libC code is part of the application and does not come from the shared lib. In such cases, LD_PRELOAD also does not help.
A comment already suggested to use strace/ltrace -- my generalized advice would be to have a look at ptrace() which both of these tools use and which should give you what you want without the need of modifying the kernel.
Patch-free User-level Link-time intercepting of system calls and interposing on library functions may do the trick but I have not tested it.
I'm pretty sure the only way you can do this is by modifying the system call table. HIDS systems (such as Samhain) will report this as an intrusion and Linux kernel developers frown upon this, heavily. The implementation details are very specific to the OS (i.e. what works on FreeBSD won't necessarily work on Linux), but the general implementation details are going to be the same. A kernel module might be a better way to go with cleaner, more standardized APIs.
A simple question:
I need to add some logging to my program.
If two processes use "fwrite" on the same file but not the same file descriptor will the written log messages be atomic or mixed. Is there a length limit?
Is it defined ANSI-C behaviour or implementation defined?
If the later what is on MacOSX, Linux and Windows MSVC?
After doing some research and I've found the following in this link:
POSIX standard requires that C stdio
FILE* operations are atomic.
POSIX-conforming C libraries (e.g, on
Solaris and GNU/Linux) have an
internal mutex to serialize operations
on FILE*s.
It looks like that calls should be atomic, but it depends on your platform. In same link, there is also another paragraph that lets you think that the programmer should take care:
So, for 3.0, the question of "is
multithreading safe for I/O" must be
answered with, "is your platform's C
library threadsafe for I/O?" Some are
by default, some are not; many offer
multiple implementations of the C
library with varying tradeoffs of
threadsafety and efficiency. You, the
programmer, are always required to
take care with multiple threads.
Also, as you have two different FILE* in two different processes, I think you have no choice.
It can be mixed.
If you have more than one thread/process writing to the same file, you need to use locking.
An alternative is to send log messages to a dedicated service/thread. An excellent tool to adopt is syslog, which is surely installed on all unixes and can be run on Windows.
From "man flockfile" on Debian lenny, the stdio functions are thread-safe.
There're thread-unsafe stdio functions, "man unlocked_stdio" for more details.
You can get more information from the man page.
fwrite for visual studio locks the calling thread and is therefore thread-safe
There are multiple sections in the manpages. Two of them are:
2 Unix and C system calls
3 C Library routines for C programs
For example there is getmntinfo(3) and getfsstat(2), both look like they do the same thing. When should one use which and what is the difference?
System calls are operating system functions, like on UNIX, the malloc() function is built on top of the sbrk() system call (for resizing process memory space).
Libraries are just application code that's not part of the operating system and will often be available on more than one OS. They're basically the same as function calls within your own program.
The line can be a little blurry but just view system calls as kernel-level functionality.
Libraries of common functions are built on top of the system call interface, but applications are free to use both.
System calls are like authentication keys which have the access to use kernel resources.
Above image is from Advanced Linux programming and helps to understand how the user apps interact with kernel.
System calls are the interface between user-level code and the kernel. C Library routines are library calls like any other, they just happen to be really commonly provided (pretty much universally). A lot of standard library routines are wrappers (thin or otherwise) around system calls, which does tend to blur the line a bit.
As to which one to use, as a general rule, use the one that best suits your needs.
The calls described in section 2 of the manual are all relatively thin wrappers around actual calls to system services that trap to the kernel. The C standard library routines described in section 3 of the manual are client-side library functions that may or may not actually use system calls.
This posting has a description of system calls and trapping to the kernel (in a slightly different context) and explains the underlying mechanism behind system calls with some references.
As a general rule, you should always use the C library version. They often have wrappers that handle esoteric things like restarts on a signal (if you have requested that). This is especially true if you have already linked with the library. All rules have reasons to be broken. Reasons to use the direct calls,
You want to be libc agnostic; Maybe with an installer. Such code could run on Android (bionic), uClibc, and more traditional glibc/eglibc systems, regardless of the library used. Also, dynamic loading with wrappers to make a run-time glibc/bionic layer allowing a dual Android/Linux binary.
You need extreme performance. Although this is probably rare and most likely misguided. Probably rethinking the problem will give better performance benefits and not calling the system is often a performance win, which the libc can occasionally do.
You are writing some initramfs or init code without a library; to create a smaller image or boot faster.
You are testing a new kernel/platform and don't want to complicate life with a full blown file system; very similar to the initramfs.
You wish to do something very quickly on program startup, but eventually want to use the libc routines.
To avoid a known bug in the libc.
The functionality is not available through libc.
Sorry, most of the examples are Linux specific, but the rationals should apply to other Unix variants. The last item is quite common when new features are introduced into a kernel. For example when kqueue or epoll where first introduced, there was no libc to support them. This may also happen if the system has an older library, but a newer kernel and you wish to use this functionality.
If your process hasn't used the libc, then most likely something in the system will have. By coding your own variants, you can negate the cache by providing two paths to the same end goal. Also, Unix
will share the code pages between processes. Generally there is no reason not to use the libc version.
Other answers have already done a stellar job on the difference between libc and system calls.