I like programming challenges, and writing a kernel seems a programming challenge.
Unfortunately, kernels are particularly hard to test because they are basically the core of operating systems and so they can't be easily ran on top of an operating system.
However, I know about applications called Virtual Machines that can emulate computer hardware.
What is the easiest/best way to develop and test kernels(C+Assembly) using Virtual Machines?
While BOCHS seems to be better at letting you know when something goes horribly wrong with your pet OS... it is very slooooow! I use VirtualPC for general purpose testing and BOCHS when things get murky.
Also, you will more than likely be booting the OS every 2 minutes, so it helps to have some sort of automated way to build a boot image & fire off the Virtual PC.
I built a GRUB boot floppy image with all the necessary stuff to get it to boot the Kernel.Bin from the root. I use a batch file to copy this file to the virtual project directory, use FAT Image Generator to copy my kernel to the image. Then just launch the VirtualPC project. Vola!
Excerpt from my batch file:
COPY Images\Base.vfd Images\Boot.vfd /Y
fat_imgen.exe modify Images\Boot.vfd -f Source\Bin\KERNEL.BIN
COPY Images\Boot.vfd Emulators\VirtualPC\ /Y
START Emulators\VirtualPC\MyOS.vmc
One last suggestion: Set the VirtualPC process priority to low - trust me on this one!
I'd be happy to exchange some code!
Tools: DGJPP, NASM, GRUB.
Code: osdev.org, osdever.net
You might be interested in looking at HelenOS. Its a from scratch microkernel that has been ported to many architectures (boots just fine on bare metal) developed using simulators such as Simics and QEMU.
We use a static grub that is copied to the final ISO during the build process. Some things just have to be that way until the OS becomes self hosting. I highly recommend NOT implementing your own userspace C library unless you really do want to do everything from scratch .. you'll become self hosting much sooner :)
Though Simics is non-free, I highly recommend it (and its built in debugging/profiling tools) while making your kernel. Once you have some kind of kernel console and logger in place, QEMU does a very nice job.
It's straightforward. Set up a virtual machine, write your kernel, copy it to the virtual machine, boot the virtual machine.
You'll need to be more specific if you want more specific advice.
Probably just setting up a machine (x86, I guess), and then investigate exactly how it behaves during boot. There should be one or more files in the host machine's file system that act as the virtual machine's file system, and then you'd need to put some boot sector information there that causes your in-development kernel to boot.
That would of course mean that the build system on the host has a way to write the kernel to the virtual machine's file system, which might vary in difficulty.
Picking one at random, bochs seems to support editing the boot media from the outside using standard tools like dd etc.
The first question that you need to ask yourself is what hardware architecture are you targeting? I'll assume for the sake of this discussion that you are targeting the IA_32 architecture, which would probably be a wise choice as there is plenty of readily-available documentation on that processor.
If you're truly serious about this undertaking, then you will definitely want to run your debug/code/build/deploy cycle against an emulator or VM. Someone mentioned BOCHS, which is very popular. If emulation speed is your thing, there is also an emulator called Qemu that is faster than BOCHS.
I'd suggest that your development environment run under Linux or Windows, which again would probably be a wise choice due to the available documentation for those dev environments.
Make is your friend. Use it to automate the build/execute process. I'd advise you to pick your toolsets/compilers up front, and spend some time learning them well. It will save you in the long run.
Related
I trying to boot an elf microkernel in an UEFI environment. So i compiled a minimal boot loader and created an ESP image. This works fine if I boot via an HDD but I want to direct boot it via the qemu -kernel option (This is some special requirement as I am working with AMD SEV). This doesn't work.
I can boot my kernel like this with grub if I use grub mkimage with a fat image included i.e. like this:
mcopy -i "${basedir}/disk.fat" -- "${basedir}/kernel" ::kernel
mcopy -i "${basedir}/disk.fat" -- "${basedir}/module" ::module
grub-mkimage -O x86_64-efi
-c "${basedir}/grub-bootstrap.cfg"
-m "${basedir}/disk.fat"
-o "${basedir}/grub.efi"
But the goal for my system is minimalism and security hence the microkernel, so grub and it's vulnerabilities is out of question.
So my question is:
How to create a bootable application image similar to grub-mkimage?
I have read about efi stub boot but couldn't really figure out how to build an efi stub image.
Normally I am a bare metal embedded programmer, so the whole uefi boot thing is a bit weird to me. I am glad for any tips or recommendations. Also I figured stack overflow might not be the best place for such low level questions, can you maybe recommend other forums?
I want to direct boot it via the qemu -kernel option
Why? It's a qemu-specific hack that doesn't exist on anything else (including any real computer). By using this hack the only thing you're doing is failing to test anything you'd normally use to boot (and therefore failing to test anything that actually matters).
(This is some special requirement as I am working with AMD SEV)
That doesn't make any sense (it's a little bit like saying "I have a banana in my ear because I'm trying to learn how to play piano").
AMD's SEV is a set of extensions intended to enhance the security of virtual machines that has nothing at all to do with how you boot (or whether you boot from BIOS or UEFI or a qemu-specific hack).
I am glad for any tips or recommendations.
My recommendation is to stop using GRUB specific (multi-boot), Qemu specific (-kernel) and Linux/Unix specific (elf) tools and actually try to use UEFI. This will require you to write your own boot loader using (Microsoft's) PE32+ file format that uses UEFI's services itself. Note that GNU's tools (their "Gnu-EFI" stuff for GCC) is relatively awful (it puts a PE32+ wrapper around an ELF file and does run-time patching to make the resulting Franken-monster work); and there are much better alternatives now (e.g. the Clang/LLVM/lld toolchain).
If you care about security, then it'll also involve learning about UEFI SecureBoot (and key management, and digital signatures). If you care about secure virtual machines I'd also recommend learning about the SKINIT instruction from AMD"s manual (used to create a dynamic root of trust after boot); but don't forget that this is AMD specific and won't work on any Intel CPU, and is mostly obsolete (the "trusted measurement" stuff from BIOS and TPM was mostly superseded by SecureBoot anyway), and (even on Intel CPUs) if you're only the guest then the hyper-visor can emulate it in any way it wants (and it won't guarantee anything is secure).
Finally; note that booting a micro-kernel directly doesn't make much sense either. There's no device drivers in a micro-kernel; so after booting a micro-kernel you end up with a "can't start any device drivers because there are no device drivers" problem. Instead you need to load many files (e.g. maybe an initial RAM disk), then (e.g.) start some kind of "boot log handler" (to display error messages, etc); then find and start the kernel, then start other processes (e.g. "device manager" to detect devices and drivers; "VFS layer" to handle file systems and file IO; etc). For the whole thing; starting the kernel is just one relatively insignificant small step (not much more than starting a global shared library that provides multi-tasking) buried among a significantly larger amount of code that does all the work.
Sadly; booting a monolithic kernel directly can make sense because it can contain all the drivers (or at least, has enough built into the kernel's executable file to handle an initial RAM disk if it's "modular monolithic" with dynamically loaded drivers); and this "monolithic with stuff that doesn't belong in any micro-kernel" idea is what most beginner tutorials assume.
I have Fedora installed on my PC and I have a Friendly ARM Mini2440 board. I have successfully installed Linux kernel and everything is working. Now I have some image processing program, which I want to run on the board without OS. The only process running on board should be my program. And in that program how can I access the on board camera to take image from, and serial port to send output to the PC.
You're talking about what is often called a bare-metal environment. Google can help you, for example here. In a bare-metal environment you have to have a good understanding of your hardware because you have to take care of a lot of things that the OS normally handles.
I've been working (off and on) on bare-metal support for my ELLCC cross development tool-chain. I have the ARM implementation pretty far along but there is still quite a bit of work to do. I have written about some of my experiences on my blog.
First off, you have to get your program started. You'll need to write some start-up code, usually in assembly, to handle the initialization of the processor as it comes out of reset (or is powered on). The start-up code then typically passes control to code written in C that ultimately directly or indirectly calls your main() function. Getting to main() is a huge step in your bare-metal adventure!
Next, you need to decide how to support your hardware's I/O devices which in your case include the camera and serial port. How much of the standard C (or C++) library does your image processing require? You might need to add some support for functions like printf() or malloc() that normally need some kind of OS support. A simple "hello world" would be a good thing to try next.
ELLCC has examples of various levels of ARM bare-metal in the examples directory. They range from a simple main() up to and including MMU and TCP/IP support. The source for all of it can be browsed here.
I started writing this before I left for work this morning and didn't have time to finish. Both dwelch and Clifford had good suggestions. A bootloader might make your job a lot simpler and documentation on your hardware is crucial.
First you must realise that without an OS, you are responsible for bringing the board up from reset including configuring the PLL and SDRAM, and also for the driver code for every device on the board you wish to use. To do that required adequate documentation of the board and it devices.
It is possible that you can use the existing bootloader to configure the core and SDRAM, but that may not meet your requirement for the only process running on the board should be your image processing program.
Additionally you will need some means of loading and bootstrapping; again the existing Linux bootstrapper may suit.
It is by no means straightforward and cannot really be described in detail here.
I am just learning linux kernel programming with the LINUX KERNEL DEVELOPMENT book(I am beginner linux kernel programming but not on linux programming). It is possible to test programs in a kernel machine with VMware viritual on Ubuntu without damage my system ?
Yes you can safely test kernel modules on a virtual machine!
I'll give you some links that may help:
watch this site
http://free-electrons.com/
in particular this book:
http://free-electrons.com/doc/books/ldd3.pdf
Also this guide:
http://www.tldp.org/HOWTO/Module-HOWTO/
An embedded distro is even better
An Ubuntu guest is fine, but I prefer to keep things minimal and use an embedded distro, as this will make things:
simpler and easier to understand and control
faster
In particular, I recommend using:
Buildroot, which is highly configurable, documented and maintained, also builds host QEMU so easy to patch it up (e.g. to add your own devices since out-of-tree devices are not possible yet ?)
QEMU emulator: small comprehensible source, ARM support, official Android emulator, kernel GDB support
Embedded distros can generate rootfs images smaller than 10MiB, and it becomes possible to understand the entire userland setup, which will make it easier to focus on the kernel.
I have made a setup to make everything as automated as possible: https://github.com/cirosantilli/linux-kernel-module-cheat
I've been using a VM for a long time for Linux kernel programming and I've never had any problem. Actually, if you manage to violate the protections of a VM then you will probably be hired by Oracle or VMWare :D
However, I recommend you to read this post: https://security.stackexchange.com/questions/23452/is-it-safe-to-use-virtual-machines-when-examining-malware
Next term, I'll need to write a basic operating system for Motorola 68K processor as part of a course lab material.
Is there a Linux emulator of a basic hardware setup with that processor? So my partners and I can debug quicker on our computers instead of physically restarting the board and stuff.
Is it possible to apply test-driven development technique to OS development? Code will be mostly assembly and C. What will be the main difficulties with trying to test-drive this? Any advice on how to do it?
I would recommend developing an operating system for the classic Amiga computers, which had different versions of the 68000 processor. Since the Amiga computer is a complete computer and is extremely well documented, I thought this would be a good exercise.
There is an emulator for it called UAE (and Win-UAE) which is very exact and
can be configured with different kinds of processors (68000 - 68060) and other capabilities. Normally, you would also need to acquire ROMs for it, but since you are developing an operating system yourself, this is not necessary.
Tools you will need is either Cygwin (for developing under Windows) or a Linux computer. Then you will need cross compilers. This includes both a C compiler and an assembler. Here is a template for creating a simple ROM which changes screen color and flicks the power LED. It will create a file 'kick.rom' which UAE then searches for in the current directory.
Reference on the 68000 instruction set can be found at the links below. Be aware that different assembler programs may use slightly different syntax and instruction set.
If you need to demo the operating system on real hardware, there are modern Amiga clones sold on Ebay and other places. Search for "Minimig".
Update:
Nowadays AROS also runs on UAE as well as physical Amigas.
Refs:
[UAE]
[WinUAE]
[Cygwin]
[Cross Compilers]
[68000 reference]
I would suggest QEMU for m68k emulation.
(The system emulator you want in QEMU is "Coldfire" - that's what Freescale calls the successor to the m68k architecture).
You certainly can tdd this project. First off decouple all accesses to the hardware with simple routine calls, e.g. getch() and printf, then you can provide simple mocks that provide test input and check output. You can then write well over 90% of the project on a PC using gcc, msdev or xcode. Once you have got some confidence in the decoupling routines you will need very little access to the hardware, and only then to occasionally check that your mocks are acting as you expect.
Keep to C until you find a particular bottle neck, and only then resort to assembler.
There are a few new projects that use hardware simulated 68000 cpus, the C-One project, the Minimig (Mini Amiga) project and the Natami (Native Amiga) project - they are new 68k compatible Amiga systems.
C One, reconfigurable computer, Minimig, in development, prototypes done: FPGA Arcade and Natami.
The Easy68k http://www.easy68k.com simulator might help you.
The uClinux project started on a m68k board. They may have the tools you need...
I was looking at the Linux From Scratch project awhile ago and was sort of disapointed that you needed an existing copy of Linux on your machine to build it. I know that Linux is very easy to obtain, install, etc. but I was hoping to build the LFS project outside of the modern operating systems (Unix/Linux/OS-X/Windows/Etc.) and in something like DOS.
My question is, how might I build a project whether it be C, C++ or some other language with a C compiler, without building that project within another operating system. By operating system I mean Unix, Linux, OS-X, Windows, and every other GUI capable 'modern-ish' OS.
So specifically I'm looking for something that works much like DOS. I'm not above using DOS if thats all that is available, however I'm thinking something that has the ability to use all available memory, processing power, etc. I want to start my computer and be welcomed by a "prompt" from which I can build or execute a program (like another Operating System).
In order to build a program you need to: execute other programs (compiler, linker), access a filesystem both for reading the code and writing out the compiled files, and so on. You need a "real" operating system, even more so if you want to "use all available memory" and processing power. If you don't like the "high level appearence" of GUI capable OSes, just try one of the many stripped-down linux distros: for instance, "damn small linux" comes to mind.
I think the closest you're going to come is a Gentoo Linux Stage 1 install. It basically gives you a prompt and then you compile EVERYTHING, including the kernel, from that minimal starting point. It's about as close as you're going to get without keying in the binary for the bootloader by hand ;)
My guess is, it will be lots of work, but this DOS compiler may help DJGPP. Minix may also be an option, but it does have X Windows. Beyond that, you are going to be hard pressed to find anything.