I know that OS kernels are made up of drivers, but how does the driver become a part of the os?, does the kernel decompile itself, and then add the driver and recompile itself?, or are the drivers plug-ins for the kernel?, someone told me that for most operating systems, the drivers actually become a part of the kernel, but whenever I compile a c program, it turns into an ordinary executable
The driver architecture depends entirely on your operating system. For most operating systems running on computers (as opposed to embedded devices), thinking of drivers as 'plug-ins' for the kernel is pretty much accurate. That said, there are plenty of older, smaller, and less sophisticated operating systems which require you to build the driver in as part of the kernel - no dynamic loading possible. These days, several operating systems have support for "user-mode" drivers, which are device drivers that don't ever run in the kernel memory space at all.
It depends on the o/s.
Classically, the kernel was a monolithic executable that contained all the drivers - and was rebuilt when a new driver needed to be added, including the code for the new driver along with all the old ones.
In modern Linux, and probably other o/s too, the drivers are dynamically loaded by the kernel when needed. The driver is created in a form that allows the kernel to do that loading; typically, that means in a shared object or dynamic link library format.
In operating systems like Linux drivers can be actually compiled into the kernel image. Although even if statically linked, they may well exhibit a plug-in type architecture that allows one to easily only include the drivers one needs.
Alternatively, they are dynamically linked and loaded either at boot time or on demand when required by some system level software.
Related
Can the C library libc be moved from one system to another by just re-compiling it?
No, you cannot move libc by just recompiling.
The core part of the operating system (called the kernel), manages things like threads, processes, memory management, drivers, thermal sensors, and other vital things. glibc uses the Linux kernel for parts of the standard that require system calls.
You'd need to modify it for your system before compiling it. Or, if you are using the Linux kernel, you'd just need to cross-compile the Linux kernel and glibc for your target.
For Embedded systems where the program runs independently on a micro-controller :
Are the programs always static linked ? or in certain occasions it may be dynamic linked ?
From Wikipedia:
a dynamic linker is the part of an operating system that loads and
links the shared libraries needed by an executable when it is executed
(at "run time"), by copying the content of libraries from persistent
storage to RAM, and filling jump tables and relocating pointers.
So it implies that dynamic linking is possible only if:
1) You have a some kind of OS
2) You have some kind of persistent storage / file system.
On a bare-metal micros it is usually not the case.
Simply put: if there is a full-grown operation system like Linux running on the microcontroller, then dynamic linking is possible (and common).
Without such an OS, you very, very likely use static linking. For this the linker will (basically) not only link the modules and libraries, but also includes the functions which are done by the OS program loader.
Lets stay at these (smaller) embedded systems for now.
Apart from static or dynamic linking, the linker also does relocation. This does - simply put - change internal (relative) addresses of the program to the absolute addresses on the target device.
It is not common on simple embedded systems primarily because it is neither necessary nor supported by the operating system (if any). Dynamic linking implies a certain amount of runtime operating system support.
The embedded systems RTOS VxWorks supports dynamic linking in the sense that it can load and link partially linked object code from a network or file system at runtime. Similarly Larger embedded RTOSs such as QNX support dynamic linking, as does embedded Linux.
So yes large embedded systems may support dynamic linking. In many cases it is used primarily as a means to link LGPL licensed code to a closed source application. It can also be used as a means of simplifying and minimising the impact of deploying changes and updates to large systems.
I would like to build an OS some time in the future, and now thinking of some light sketches on how it would be. I have pretty much been coding in C compiled for the Windows environment (and some little Java). I would have to recompile any of my C programs should I want to run it under Linux. So the binaries, the product of compilation, must be different for each operating system. If I design a totally new OS from scratch, for both hobby and academic purpose, without using the Linux kernel or any known base code of an OS, what I understand as to happen is that I cannot compile my C programs with GCC since my OS will not be among its target systems. Here my question written on the title emerges. Thanks in advance for any hints.
It depends. You could easily choose to re-use an existing compiler, such as the immemorial example GCC, and thus you would reap the benefits of an existing compiler. But there are some big provisos that must be cleared up.
Regards of whether or not you choose to build a new compiler, the challenge will remain in porting a C library. You technically can use C without a standard library (which is what the Linux kernel, or any self-hosted example for that matter, has to do, for example) but this is a ridiculous proposition for programs intended to run under an operating system, as most systems impose memory restrictions, etc, meaning that you cannot just have carte blanche in terms of using memory. Thusly, a C library call such as malloc is required.
Since any programs under your kernel (99% of your OS in all likelihood) will need a set of functions to link against, porting a C library is your biggest task. The C library is a huge monolith, and writing your own would be rather silly, especially with many implementations already available, the most well known being GCC's. So, the question you really should be asking is, do you want to write my own version of libc? (The answer is almost always no, and most alternative implementations are for niche use cases.) Plus, if you want to make your OS POSIX-compliant, then you'll have to implement more functions, adding to the hassle.
Whether you write your own compiler for your OS is a minor detail compared to which C library will be included with it. You can always use your own compiler with an already-written implementation of the C library.
My advice to your rather opinion-based question: no. Port an existing compiler such as GCC or clang, and then use that. Plus, that has several advantages:
Compatibility with existing tools and toolchains
A familiar program (no need for your users to learn how to use a new compiler)
They're open source - and in spite of that, you'd be insane to go at it alone. Heck, even Apple integrated two already existing compilers - GCC and clang - into their toolchains rather than do it themselves, and they're a billion-dollar company.
Take a look at this page. It demonstrates how to port GCC to your OS using Newlib as your C library.
No, you can just port an existing compiler. You can even choose an existing executable format, such as ELF, and use your standard GCC + GNU Binutils toolchain. You will need to port the standard library and C runtime, and you will need to write an ELF loader into your operating system.
I suspect the majority of the work will be in porting the C library.
A search turned up this page: Porting GCC to your OS
(1) No, you usually don't have to write your own compiler. Writing a good optimizing compiler can be actually big task which I would better avoid.
But in order to enable writing applications for your OS in some higher level language you will either need to provide
some (2.1) API emulation layer (so that code written and compiled for other OS can be run on your OS)
or you'll have to (2.2) port some existing compiler to your OS
or at least make your OS a new available (2.3) target platform in an existing compiler
or some other option I don't know about
The choices are multiple each with its own pros/cons.
Some examples (other then the obvious GCC already mentioned by #dietrich-epp, #sevenbits) to help you decide which way you want to follow:
(3.1) Free Pascal (see http://www.freepascal.org) compiler can be extended with another target platform
Free Pascal is a 32,64 and 16 bit professional Pascal compiler. It can target multiple processor architectures: Intel x86, AMD64/x86-64, PowerPC, PowerPC64, SPARC, and ARM. Supported operating systems include Linux, FreeBSD, Haiku, Mac OS X/iOS/Darwin, DOS, Win32, Win64, WinCE, OS/2, MorphOS, Nintendo GBA, Nintendo DS, and Nintendo Wii. Additionally, JVM, MIPS (big and little endian variants), i8086 and Motorola 68k architecture targets are available in the development versions
...
Source: http://www.freepascal.org
(3.2) Inferno Operating System (see http://www.vitanuova.com/inferno) has its own application language (see Limbo) with OS specific words, own compiler etc. Applications run in virtual machine (see Dis)
Inferno® is a compact operating system designed for building distributed and networked systems on a wide variety of devices and platforms. With many advanced and unique features, Inferno puts an unrivalled set of tools into your hands...Inferno can run as a user application on top of an existing operating system or as a stand alone operating system...
Source: http://www.vitanuova.com/inferno
(3.3) Squeak (see http://en.wikipedia.org/wiki/Squeak) is a self contained OS with graphics and everything. It uses Smalltalk-80 as the language. Compiler included, applications run in virtual machine (see Cog VM). The VM could be emitted as portable C code and then ported to a bare-bone hardware.
Squeak is a modern, open source, full-featured implementation of the powerful Smalltalk programming language and environment. Squeak is highly-portable, running on almost any platform you could name and you can really truly write once run anywhere. Squeak is the vehicle for a wide range of projects from multimedia applications and educational platforms to commercial web application development...
Source: http://www.squeak.org
(3.4) MenuetOS (see http://www.menuetos.net/) is 64bit OS written in assembly language. Flat Assembler (see FASM) compiler which can emit native binaries was ported to the OS including OS API and is included in basic installation. Later on C library was also ported
MenuetOS is an Operating System in development for the PC written entirely in 32/64 bit assembly language...supports 32/64 bit x86 assembly programming for smaller, faster and less resource hungry applications...Menuet isn't based on other operating system nor has it roots within UNIX or the POSIX standards. The design goal, since the first release in year 2000, has been to remove the extra layers between different parts of an OS, which normally complicate programming and create bugs...
Source: http://www.menuetos.net
(3.5) Google's Android OS (see Wikipedia: Android (operating system)) ported Java Virtual Machine (see Dalvik later replaced by Android Runtime) and provided OS APIs for the Java programming language, reusing existing compilers and IDEs just consuming the produced binaries
Android Runtime (ART) is an application runtime environment used by the Android mobile operating system. ART replaces Dalvik, which is the process virtual machine originally used by Android, and performs transformation of the application's bytecode into native instructions that are later executed by the device's runtime environment...
Source: http://en.wikipedia.org/wiki/Android_Runtime
There are many more useful examples available. Whether you have to or don't have to basically depends on the programming paradigm your new OS will introduce. Why you want to build it and how will it differ from the existing ones.
Examples for no are: (3.1), (3.4), (3.5)
Examples for yes are: (3.2), (3.3)
If you compile a program in say, C, on a Linux based platform, then port it to use the MacOS libraries, will it work?
Is the core machine-code that comes from a compiler compatible on both Mac and Linux?
The reason I ask this is because both are "UNIX based" so I would think this is true, but I'm not really sure.
No, Linux and Mac OS X binaries are not cross-compatible.
For one thing, Linux executables use a format called ELF.
Mac OS X executables use Mach-O format.
Thus, even if a lot of the libraries ordinarily compile separately on each system, they would not be portable in binary format.
Furthermore, Linux is not actually UNIX-based. It does share a number of common features and tools with UNIX, but a lot of that has to do with computing standards like POSIX.
All this said, people can and do create pretty cool ways to deal with the problem of cross-compatibility.
EDIT:
Finally, to address your point on byte-code: when making a binary, compilers usually generate machine code that is specific to the platform you're developing on. (This isn't always the case, but it usually is.)
In general you can easily port a program across various Unix brands. However you need (at least) to recompile it on each platform.
Executables (binaries) are not usable on several platforms, because an executable is tightly coupled with the operating system's ABI (Application Binary Interface), i.e. the conventions of how an application communicates with the operating system.
For instance if your program prints a string onto the console using the POSIX write call, the ABI specifies:
How a system call is done (Linux used to call the 0x80 software interrupt on x86, now it uses the specific sysenter instruction)
The system call number
How are the function's arguments transmitted to the system
Any kind of alignment
...
And this varies a lot across operating systems.
Note however that in some cases there may be “ABI adapters” allowing to run binaries of one OS onto another OS. For instance Wine allows you to run Windows executables on various Unix flavors, NDISwrapper allows you to use Windows network drivers on Linux.
"bytecode" usually refers to code executed by a virtual machine (e.g. for java or python). C is compiled to machine code, which the CPU can execute directly. Machine language is hardware-specific so it it would be the same under any OS running on an intel chip (even under Windows), but the details of how the machine code is wrapped into an executable file, and how it is integrated with system calls and dynamically linked libraries are different from system to system.
So no, you can't take compiled code and use it in a different OS. (However, there are "cross-compilers" that run on one OS but generate code that will run on another OS).
There is no "core byte-code that comes from a compiler". There is only machine code.
While the same machine instructions may be applicable under several operating systems (as long as they're run on the same hardware), there is much more to a hosted executable than that, and since a compiled and linked native executable for Linux has very different runtime and library requirements from one on BSD or Darwin, you won't be able to run one binary on the other system.
By contrast, Windows binaries can sometimes be executed under Linux, because Linux provides both a binary format loader for Windows's PE format, as well as an extensive API implementation (Wine). In principle this idea can be used on other platforms as well, but I'm not aware of anyone having written this for Linux<->Darwin. If you already have the source code, and it compiles in Linux, then you have a good chance of it also compiling under MacOS (modulo UI components, of course).
Well, maybe... but most probably not.
But if it does, it's not "because both are UNIX" it's because:
Mac computers happen to use the same processor nowadays (this was very different in the past)
You happen to use a program that has no dependency on any library at all (very unlikely)
You happen to use the same runtime libraries
You happen to use a loader/binary format that is compatible with both.
I don't quite understand the compiling process of the Linux kernel when I install
a Linux system on my machine.
Here are some things that confused me:
The kernel is written in C, however how did the kernel get compiled without a compiler installed?
If the C compiler is installed on my machine before the kernel is compiled, how can the compiler itself get compiled without a compiler installed?
I was so confused for a couple of days, thanks for the response.
The first round of binaries for your Linux box were built on some other Linux box (probably).
The binaries for the first Linux system were built on some other platform.
The binaries for that computer can trace their root back to an original system that was built on yet another platform.
...
Push this far enough, and you find compilers built with more primitive tools, which were in turn built on machines other than their host.
...
Keep pushing and you find computers built so that their instructions could be entered by setting switches on the front panel of the machine.
Very cool stuff.
The rule is "build the tools to build the tools to build the tools...". Very much like the tools which run our physical environment. Also known as "pulling yourself up by the bootstraps".
I think you should distinguish between:
compile, v: To use a compiler to process source code and produce executable code [1].
and
install, v: To connect, set up or prepare something for use [2].
Compilation produces binary executables from source code. Installation merely puts those binary executables in the right place to run them later. So, installation and use do not require compilation if the binaries are available. Think about ”compile” and “install” like about “cook” and “serve”, correspondingly.
Now, your questions:
The kernel is written in C, however how did the kernel get compiled without a compiler installed?
The kernel cannot be compiled without a compiler, but it can be installed from a compiled binary.
Usually, when you install an operating system, you install an pre-compiled kernel (binary executable). It was compiled by someone else. And only if you want to compile the kernel yourself, you need the source and the compiler, and all the other tools.
Even in ”source-based” distributions like gentoo you start from running a compiled binary.
So, you can live your entire life without compiling kernels, because you have them compiled by someone else.
If the C compiler is installed on my machine before the kernel is compiled, how can the compiler itself get compiled without a compiler installed?
The compiler cannot be run if there is no kernel (OS). So one has to install a compiled kernel to run the compiler, but does not need to compile the kernel himself.
Again, the most common practice is to install compiled binaries of the compiler, and use them to compile anything else (including the compiler itself and the kernel).
Now, chicken and egg problem. The first binary is compiled by someone else... See an excellent answer by dmckee.
The term describing this phenomenon is bootstrapping, it's an interesting concept to read up on. If you think about embedded development, it becomes clear that a lot of devices, say alarm clocks, microwaves, remote controls, that require software aren't powerful enough to compile their own software. In fact, these sorts of devices typically don't have enough resources to run anything remotely as complicated as a compiler.
Their software is developed on a desktop machine and then copied once it's been compiled.
If this sort of thing interests you, an article that comes to mind off the top of my head is: Reflections on Trusting Trust (pdf), it's a classic and a fun read.
The kernel doesn't compile itself -- it's compiled by a C compiler in userspace. In most CPU architectures, the CPU has a number of bits in special registers that represent what privileges the code currently running has. In x86, these are the current privilege level bits (CPL) in the code segment (CS) register. If the CPL bits are 00, the code is said to be running in security ring 0, also known as kernel mode. If the CPL bits are 11, the code is said to be running in security ring 3, also known as user mode. The other two combinations, 01 and 10 (security rings 1 and 2 respectively) are seldom used.
The rules about what code can and can't do in user mode versus kernel mode are rather complicated, but suffice to say, user mode has severely reduced privileges.
Now, when people talk about the kernel of an operating system, they're referring to the portions of the OS's code that get to run in kernel mode with elevated privileges. Generally, the kernel authors try to keep the kernel as small as possible for security reasons, so that code which doesn't need extra privileges doesn't have them.
The C compiler is one example of such a program -- it doesn't need the extra privileges offered by kernel mode, so it runs in user mode, like most other programs.
In the case of Linux, the kernel consists of two parts: the source code of the kernel, and the compiled executable of the kernel. Any machine with a C compiler can compile the kernel from the source code into the binary image. The question, then, is what to do with that binary image.
When you install Linux on a new system, you're installing a precompiled binary image, usually from either physical media (such as a CD DVD) or from the network. The BIOS will load the (binary image of the) kernel's bootloader from the media or network, and then the bootloader will install the (binary image of the) kernel onto your hard disk. Then, when you reboot, the BIOS loads the kernel's bootloader from your hard disk, and the bootloader loads the kernel into memory, and you're off and running.
If you want to recompile your own kernel, that's a little trickier, but it can be done.
Which one was there first? the chicken or the egg?
Eggs have been around since the time of the dinosaurs..
..some confuse everything by saying chickens are actually descendants of the great beasts.. long story short: The technology (Egg) was existent prior to the Current product (Chicken)
You need a kernel to build a kernel, i.e. you build one with the other.
The first kernel can be anything you want (preferably something sensible that can create your desired end product ^__^)
This tutorial from Bran's Kernel Development teaches you to develop and build a smallish kernel which you can then test with a Virtual Machine of your choice.
Meaning: you write and compile a kernel someplace, and read it on an empty (no OS) virtual machine.
What happens with those Linux installs follows the same idea with added complexity.
It's not turtles all the way down. Just like you say, you can't compile an operating system that has never been compiled before on a system that's running that operating system. Similarly, at least the very first build of a compiler must be done on another compiler (and usually some subsequent builds too, if that first build turns out not to be able to compile its own source code just yet).
I think the very first Linux kernels were compiled on a Minix box, though I'm not certain about that. GCC was available at the time. One of the very early goals of many operating systems is to run a compiler well enough to compile their own source code. Going further, the first compiler was almost certainly written in assembly language. The first assemblers were written by those poor folks who had to write in raw machine code.
You may want to check out the Linux From Scratch project. You actually build two systems in the book: a "temporary system" that is built on a system you didn't build yourself, and then the "LFS system" that is built on your temporary system. The way the book is currently written, you actually build the temporary system on another Linux box, but in theory you could adapt it to build the temporary system on a completely different OS.
If I am understanding your question correctly. The kernel isn't "compiling itself" these days. Most Linux distributions today provide system installation through a linux live cd. The kernel is loaded from the CD into memory and operates as it would normally as if it were installed to disk. With a linux environment up and running on your system it is easy to just commit the necessary files to your disk.
If you were talking about the bootstrapping issue; dmckee summed it up pretty nice.
Just offering another possibility...