I was looking at the Linux From Scratch project awhile ago and was sort of disapointed that you needed an existing copy of Linux on your machine to build it. I know that Linux is very easy to obtain, install, etc. but I was hoping to build the LFS project outside of the modern operating systems (Unix/Linux/OS-X/Windows/Etc.) and in something like DOS.
My question is, how might I build a project whether it be C, C++ or some other language with a C compiler, without building that project within another operating system. By operating system I mean Unix, Linux, OS-X, Windows, and every other GUI capable 'modern-ish' OS.
So specifically I'm looking for something that works much like DOS. I'm not above using DOS if thats all that is available, however I'm thinking something that has the ability to use all available memory, processing power, etc. I want to start my computer and be welcomed by a "prompt" from which I can build or execute a program (like another Operating System).
In order to build a program you need to: execute other programs (compiler, linker), access a filesystem both for reading the code and writing out the compiled files, and so on. You need a "real" operating system, even more so if you want to "use all available memory" and processing power. If you don't like the "high level appearence" of GUI capable OSes, just try one of the many stripped-down linux distros: for instance, "damn small linux" comes to mind.
I think the closest you're going to come is a Gentoo Linux Stage 1 install. It basically gives you a prompt and then you compile EVERYTHING, including the kernel, from that minimal starting point. It's about as close as you're going to get without keying in the binary for the bootloader by hand ;)
My guess is, it will be lots of work, but this DOS compiler may help DJGPP. Minix may also be an option, but it does have X Windows. Beyond that, you are going to be hard pressed to find anything.
Related
I've run into an issue with writing some code in c. My basic problem is that I am pressed for time and the code I am dealing with has lots of bugs which I have to "erradicate" before tomorrow evening.
The bigger part of the problem is that there is no adequate IDE that can do real time debugging, especially when using threads or spawning processes with fork(). I tried Mono, Eclipse, and finally NetBeans, and have concluded that those are very good but not for coding in C. More over the learning curve to utilize the command line debugger properly is quite steep. (Like I mentioned earlier... I am pressed on time.)
So, since I am a C# developer by profession I was wondering whether I can pull this off in VS2003/VS2005/VS2008/VS2010. If I abstain from using system calls, can I do this?
Of particular interest are FILE* descriptor and fread(), fclose(), fseek() methods. I know they are part of the standard C library, however are they tied to the platform itself? Are the headers the same in Linux vs Windows? What about fork() or shared memory?
Maybe if I use VS2010 to build parts of the component at a time (by mocking inputs and stuff), debug those, and then migrate the working code in the overall Linux project would prove most useful?
Any input would be greatly appreciated.
The bigger part of the problem is that there is no adequate IDE that can do real time debugging, especially when using threads or spawning processes with fork().
The Eclipse CDT would probably have the best overall support for C/C++ development and integrated debugging.
Note that multithreaded and multiprocess debugging can be difficult at the best of times. Investing in a good logging framework would be advisable at this point, and probably more useful than relying on a debugger. There are many to choose from - have a look at Log4C++ and so on. Even printf in a pinch can be invaluable.
So, since I am a C# developer by profession I was wondering whether I can pull this off in VS2003/VS2005/VS2008/VS2010. If I abstain from using system calls, can I do this?
If you take care to only use portable calls and not Win32-specific APIs, you should be ok. Also, there are many libraries (for C++ libraries such as Boost++ that provide a rich set of functionality which work the same on Windows, Linux and others.
Of particular interest are FILE* descriptor and fread(), fclose(), fseek() methods. I know they are part of the standard C library, however are they tied to the platform itself? Are the headers the same in Linux vs Windows? What about fork() or shared memory?
Yes, the file I/O functions you mention are in <stdio.h> and part of the portable standard C library. They work essentially the same on both Windows and Linux, and are not tied to a particular platform.
However, fork() and the shared memory functions shmget() are POSIX functions, available on *nix platforms but not natively on Windows. The Cygwin project provides implementations of these functions in a library for ease of porting.
If you are using C++, Boost++ will give you portable versions of all these system-level calls.
Maybe if I use VS2010 to build parts of the component at a time (by mocking inputs and stuff), debug those, and then migrate the working code in the overall Linux project would prove most useful?
You could certainly do that. Just be mindful that Visual Studio has a tendency to lead you down the Win32 path, and you must be vigilant to not start using non-portable functions. Fortunately the library reference on MSDN gives you the compatibility information. In general, using standard C or POSIX calls will be portable. In my experience, it is actually easier to write on *nix and port to Windows, but YMMV.
Looks like I am the first to recommend Emacs here. Here is how Emacs works. When you install it, it is simply a text editor with a lot of extensions(debugger and C font-locking are included by default). As you start using it and install the extensions you miss, it becomes more than just an editor. It grows to become an IDE very soon and easily, then on to something that can eschew the OS under one frame.
Emacs might take long to learn, in the mean time, you could use Visual Slick Edit if you are not pressed on the cost part. I have used it on both platforms and seen it work good with version control, tags, etc.
Perhaps Code::Blocks? I love it and while it says it's for C++ it is, of course, very good for plain C as well.
The term has several definition according to Wikipedia, however what I'm really interested in is creating a program that has all its needed dependencies included within the source folder, so the end user doesn't need to install additional libraries for the app to install. For example, how Mac apps has all its dependencies all within the program itself already...
or is there a function that autotools does this? I'm programming in the Linux environment...
Are you talking about the source code of your application, or about your application binary?
The answer I'd give for both the cases depends on what libraries you're using.
If you're using libraries that you can find anywhere, that are somehow standard and/or that are quite big, you shouldn't attach them to your application, just require them both to build and to run your application.
Anyway don't be much concerned about your source code: little people will build your application, and they probably know something about programming and how a Linux system works; it won't be a big deal to require many (also not-so-common) dependences to build your application.
For what concerns the binary version it could be a little more problematic, since it will be used by end users who often don't know anything about libraries and programming stuff: you could choose to statically link the smallest and most uncommon libraries to your binary, in order to have less dependences.
You could do it, if you link statically, but it'd be somewhat unusual, and depending on what your program is supposed to do, you might be limiting yourself.
The alternative, if this is not just a one-off project, is to create a Linux Standard Base compatible RPM package and restrict yourself to linking against the libraries and symbols that LSB defines.
Run ldd on your program to discover all dependencies, then copy these to your directory, and add a program-wrapper script that issues
#!/bin/sh
LD_LIBRARY_PATH="${0##*/}:$LD_LIBRARY_PATH" exec "${0##*/}/real-program" "$#";
Duplicating the Mac OS X .app behavior on a plain POSIX system is difficult because it is very hard to guarantee that a process can find it's own executable (there are several way that will almost always work...). Mac OS X provides a OS service for this, but Linux (for instance) does not.
Once you've accomplished that feat, this becomes possible. Though, as others have mentioned, it loses the ability to share resource demands (disk space, RAM space, cache space) with other programs that use the same libraries because you'd be using static copies, or dynamically loading your own copy from the .app-like bundle.
A CS course I'm taking online suggests students compile their source code and run tools like valgrind on the OS UNIX. I'm completely new to UNIX, Linux, their tools, and coding in c. I've made some attempts at installing FreeBSD 8.1 on VMWare Player 3.1.3, and even managed to get VMWare Tools running. But the FreeBSD documentation has led me down many dead-ends in accomplishing common tasks i.e. mounting an NFS or USB device. It turns out that the packages I need to make this happen aren't installed or configured, and I don't see any straight answer on how to install them.
So, if I'm using UNIX only as a tool to run gcc, g++, valgrind for this CS course, and these can be run on Linux instead, it seems like I can get the job done faster using Ubuntu Linux.
Can Linux be used to compile and run c code identically on UNIX, if compiled on Linux? Or if not, what are the differences to look for?
Thanks
For the novice-level C programmer such as OP, the difference of environment is negligible. Go ahead with Linux.
I think for purposes of the course you could run your programs and tools on Linux,
but I guess the reason your teacher wants you to use FreeBSD is so that you learn other things besides just coding up your problems
The two should be effectively the same. The only major difference you might see would be due to different versions being used. I would check to see what versions of gcc, g++ and valgrind the teacher is having you use, and make sure that you have the same version running on your install of Linux.
You can also use MinGW or Cygwin. You mentioned VMWare, so I'm guessing you're trying to just get an environment up and running in a windows environment. They both allow you to use the compiler and some of the tools without a full install of a Linux based system. In a CS course they would be more than enough.
The main differences too look for:
Compiling C / C++ is not machine independent. You need to have a small environment to compile on UNIX anyway if you need to submit compiled programs to your professor.
C / C++ is rather portable if you don't use anything that's non-portable. It's very hard to verify that you didn't use something that's different between the two machines, so you may wish to compile on UNIX to verify you didn't let an unavailable library (or an specific to the OS procedure, argument, behavior, bugs, etc.) slip into your code.
The vendor of make between the two machines may differ. This means that while the core of make will operate similarly, certain features might not be available in both. In reality, you probably won't use most of makes extended features, but in a worst case scenario you might opt to maintain multiple Makefiles or limit yourself to a common subset of features.
At the end of the day, it all boils down to what your professor will want. Odds are 95+% that you can do 100% of the work in Linux, but the prof's requirements or grading environment might be such that you will have to copy your code into a UNIX account to build the final "submission" executable. Considering that university UNIX accounts aren't nearly as portable as Linux on a laptop, the cost of the "final verification / porting" to the University computer is likely to be small compared to the convenience of working on your homework more hours than you can manage in a fixed lab.
I don't quite understand the compiling process of the Linux kernel when I install
a Linux system on my machine.
Here are some things that confused me:
The kernel is written in C, however how did the kernel get compiled without a compiler installed?
If the C compiler is installed on my machine before the kernel is compiled, how can the compiler itself get compiled without a compiler installed?
I was so confused for a couple of days, thanks for the response.
The first round of binaries for your Linux box were built on some other Linux box (probably).
The binaries for the first Linux system were built on some other platform.
The binaries for that computer can trace their root back to an original system that was built on yet another platform.
...
Push this far enough, and you find compilers built with more primitive tools, which were in turn built on machines other than their host.
...
Keep pushing and you find computers built so that their instructions could be entered by setting switches on the front panel of the machine.
Very cool stuff.
The rule is "build the tools to build the tools to build the tools...". Very much like the tools which run our physical environment. Also known as "pulling yourself up by the bootstraps".
I think you should distinguish between:
compile, v: To use a compiler to process source code and produce executable code [1].
and
install, v: To connect, set up or prepare something for use [2].
Compilation produces binary executables from source code. Installation merely puts those binary executables in the right place to run them later. So, installation and use do not require compilation if the binaries are available. Think about ”compile” and “install” like about “cook” and “serve”, correspondingly.
Now, your questions:
The kernel is written in C, however how did the kernel get compiled without a compiler installed?
The kernel cannot be compiled without a compiler, but it can be installed from a compiled binary.
Usually, when you install an operating system, you install an pre-compiled kernel (binary executable). It was compiled by someone else. And only if you want to compile the kernel yourself, you need the source and the compiler, and all the other tools.
Even in ”source-based” distributions like gentoo you start from running a compiled binary.
So, you can live your entire life without compiling kernels, because you have them compiled by someone else.
If the C compiler is installed on my machine before the kernel is compiled, how can the compiler itself get compiled without a compiler installed?
The compiler cannot be run if there is no kernel (OS). So one has to install a compiled kernel to run the compiler, but does not need to compile the kernel himself.
Again, the most common practice is to install compiled binaries of the compiler, and use them to compile anything else (including the compiler itself and the kernel).
Now, chicken and egg problem. The first binary is compiled by someone else... See an excellent answer by dmckee.
The term describing this phenomenon is bootstrapping, it's an interesting concept to read up on. If you think about embedded development, it becomes clear that a lot of devices, say alarm clocks, microwaves, remote controls, that require software aren't powerful enough to compile their own software. In fact, these sorts of devices typically don't have enough resources to run anything remotely as complicated as a compiler.
Their software is developed on a desktop machine and then copied once it's been compiled.
If this sort of thing interests you, an article that comes to mind off the top of my head is: Reflections on Trusting Trust (pdf), it's a classic and a fun read.
The kernel doesn't compile itself -- it's compiled by a C compiler in userspace. In most CPU architectures, the CPU has a number of bits in special registers that represent what privileges the code currently running has. In x86, these are the current privilege level bits (CPL) in the code segment (CS) register. If the CPL bits are 00, the code is said to be running in security ring 0, also known as kernel mode. If the CPL bits are 11, the code is said to be running in security ring 3, also known as user mode. The other two combinations, 01 and 10 (security rings 1 and 2 respectively) are seldom used.
The rules about what code can and can't do in user mode versus kernel mode are rather complicated, but suffice to say, user mode has severely reduced privileges.
Now, when people talk about the kernel of an operating system, they're referring to the portions of the OS's code that get to run in kernel mode with elevated privileges. Generally, the kernel authors try to keep the kernel as small as possible for security reasons, so that code which doesn't need extra privileges doesn't have them.
The C compiler is one example of such a program -- it doesn't need the extra privileges offered by kernel mode, so it runs in user mode, like most other programs.
In the case of Linux, the kernel consists of two parts: the source code of the kernel, and the compiled executable of the kernel. Any machine with a C compiler can compile the kernel from the source code into the binary image. The question, then, is what to do with that binary image.
When you install Linux on a new system, you're installing a precompiled binary image, usually from either physical media (such as a CD DVD) or from the network. The BIOS will load the (binary image of the) kernel's bootloader from the media or network, and then the bootloader will install the (binary image of the) kernel onto your hard disk. Then, when you reboot, the BIOS loads the kernel's bootloader from your hard disk, and the bootloader loads the kernel into memory, and you're off and running.
If you want to recompile your own kernel, that's a little trickier, but it can be done.
Which one was there first? the chicken or the egg?
Eggs have been around since the time of the dinosaurs..
..some confuse everything by saying chickens are actually descendants of the great beasts.. long story short: The technology (Egg) was existent prior to the Current product (Chicken)
You need a kernel to build a kernel, i.e. you build one with the other.
The first kernel can be anything you want (preferably something sensible that can create your desired end product ^__^)
This tutorial from Bran's Kernel Development teaches you to develop and build a smallish kernel which you can then test with a Virtual Machine of your choice.
Meaning: you write and compile a kernel someplace, and read it on an empty (no OS) virtual machine.
What happens with those Linux installs follows the same idea with added complexity.
It's not turtles all the way down. Just like you say, you can't compile an operating system that has never been compiled before on a system that's running that operating system. Similarly, at least the very first build of a compiler must be done on another compiler (and usually some subsequent builds too, if that first build turns out not to be able to compile its own source code just yet).
I think the very first Linux kernels were compiled on a Minix box, though I'm not certain about that. GCC was available at the time. One of the very early goals of many operating systems is to run a compiler well enough to compile their own source code. Going further, the first compiler was almost certainly written in assembly language. The first assemblers were written by those poor folks who had to write in raw machine code.
You may want to check out the Linux From Scratch project. You actually build two systems in the book: a "temporary system" that is built on a system you didn't build yourself, and then the "LFS system" that is built on your temporary system. The way the book is currently written, you actually build the temporary system on another Linux box, but in theory you could adapt it to build the temporary system on a completely different OS.
If I am understanding your question correctly. The kernel isn't "compiling itself" these days. Most Linux distributions today provide system installation through a linux live cd. The kernel is loaded from the CD into memory and operates as it would normally as if it were installed to disk. With a linux environment up and running on your system it is easy to just commit the necessary files to your disk.
If you were talking about the bootstrapping issue; dmckee summed it up pretty nice.
Just offering another possibility...
I like programming challenges, and writing a kernel seems a programming challenge.
Unfortunately, kernels are particularly hard to test because they are basically the core of operating systems and so they can't be easily ran on top of an operating system.
However, I know about applications called Virtual Machines that can emulate computer hardware.
What is the easiest/best way to develop and test kernels(C+Assembly) using Virtual Machines?
While BOCHS seems to be better at letting you know when something goes horribly wrong with your pet OS... it is very slooooow! I use VirtualPC for general purpose testing and BOCHS when things get murky.
Also, you will more than likely be booting the OS every 2 minutes, so it helps to have some sort of automated way to build a boot image & fire off the Virtual PC.
I built a GRUB boot floppy image with all the necessary stuff to get it to boot the Kernel.Bin from the root. I use a batch file to copy this file to the virtual project directory, use FAT Image Generator to copy my kernel to the image. Then just launch the VirtualPC project. Vola!
Excerpt from my batch file:
COPY Images\Base.vfd Images\Boot.vfd /Y
fat_imgen.exe modify Images\Boot.vfd -f Source\Bin\KERNEL.BIN
COPY Images\Boot.vfd Emulators\VirtualPC\ /Y
START Emulators\VirtualPC\MyOS.vmc
One last suggestion: Set the VirtualPC process priority to low - trust me on this one!
I'd be happy to exchange some code!
Tools: DGJPP, NASM, GRUB.
Code: osdev.org, osdever.net
You might be interested in looking at HelenOS. Its a from scratch microkernel that has been ported to many architectures (boots just fine on bare metal) developed using simulators such as Simics and QEMU.
We use a static grub that is copied to the final ISO during the build process. Some things just have to be that way until the OS becomes self hosting. I highly recommend NOT implementing your own userspace C library unless you really do want to do everything from scratch .. you'll become self hosting much sooner :)
Though Simics is non-free, I highly recommend it (and its built in debugging/profiling tools) while making your kernel. Once you have some kind of kernel console and logger in place, QEMU does a very nice job.
It's straightforward. Set up a virtual machine, write your kernel, copy it to the virtual machine, boot the virtual machine.
You'll need to be more specific if you want more specific advice.
Probably just setting up a machine (x86, I guess), and then investigate exactly how it behaves during boot. There should be one or more files in the host machine's file system that act as the virtual machine's file system, and then you'd need to put some boot sector information there that causes your in-development kernel to boot.
That would of course mean that the build system on the host has a way to write the kernel to the virtual machine's file system, which might vary in difficulty.
Picking one at random, bochs seems to support editing the boot media from the outside using standard tools like dd etc.
The first question that you need to ask yourself is what hardware architecture are you targeting? I'll assume for the sake of this discussion that you are targeting the IA_32 architecture, which would probably be a wise choice as there is plenty of readily-available documentation on that processor.
If you're truly serious about this undertaking, then you will definitely want to run your debug/code/build/deploy cycle against an emulator or VM. Someone mentioned BOCHS, which is very popular. If emulation speed is your thing, there is also an emulator called Qemu that is faster than BOCHS.
I'd suggest that your development environment run under Linux or Windows, which again would probably be a wise choice due to the available documentation for those dev environments.
Make is your friend. Use it to automate the build/execute process. I'd advise you to pick your toolsets/compilers up front, and spend some time learning them well. It will save you in the long run.