tl;dr
I can't compile glibc on powerpc/mips/armeb/sparc. How can I test bigendian without emulating a whole system?
Problem Description
I am currently trying to test some code to ensure that it works on a big-endian system (it doesn't) and fix any big-endian errors. Currently I am using qemu-system-ppc with a PowerPC debian image and compiling within that image, as well as running gdb there, etc. However, this is extremely inefficient and I know there is a better way.
As such I followed this tutorial to create a cross compiler so I could compile my source to a big-endian system, then run the user-space qemu-??? to test it without the overhead of running the entire OS in emulated space (in particular with the -S switch I could run gdb on host which is much faster).
Attempt #1:
First try was powerpc. At first I just ran the configure with --target=powerpc but realized that I should use a full triplet and went with powerpc-unknown-linux-gnu. Unfortunately, when I get to the glibc portion I run into the problem that my gcc does not support IBM 128-bit long doubles (it only supports IEEE). I googled around and there doesn't seem to be a fix (aside from updating gcc past v4.1 - I'm at 6.2 right now).
Attempt #2:
Let's try mips then, I have more experience with it anyways (from university). This makes it to the glibc portion again, but this time it complains that it can't determine the ABI. I tried it with all of -linux-gnu -gneabi -eabi and -none but it couldn't determine the ABI for any of them.
Attempt #3:
Alright, time to try armeb. This time it gets past determining the ABI, then dumps a message saying that the target is not yet supported by glibc. Same story with sparc.
Question
Given that all of the above failed - how can I compile a cross-compiler from my little-endian system to a big-endian system which will generate executables I can run in user-space with qemu?
Related
Recently I upgraded up to Ubuntu 16.04.1 Xenial (from 14.04 Trusty) the build-host where I've compiled different linux kernels so far for my own project. Ubuntu 16.04.1 implies using a new updated environment for building binaries. These tools include a new gcc-5.4, libc6 (for userspace applications), etc. Also a new Ubuntu supplies (or suggests) a new kernel-package containing a new make-kpkg script and pulling different dependencies like build-essential, binutils, etc. with it
Ok, my task is to compile a linux kernel v3.10.12 (or v3.19) and run it within a VirtualBox machine (architecture x86_64, system Ubuntu 16.04.1). I used to be able to compile kernel-v3.10.12 and kernel-v3.19 in Ubuntu 14.04 Trusty deployed on the build server with the compiler gcc-4.8 and launch the kernels under the VirtualBox machine I mentioned above, but now something goes wrong while starting a kernel compiled
For example, let's consider v3.10.12 being compiled and run
For building the kernel I utilize 'make-kpkg' script provided by Ubuntu aptitude's package 'kernel-package'. I build the kernel for x86_64 using gcc-4.8 as I have always been doing
Once 'make-kpkg' has compiled the kernel and gathered linux-headers it starts packing them into deb-packages what makes me able to execute 'dpkg -i' on them in the system and install them in a 'debian' way
Okey, supposing I did it. Then I am going to reboot the system
When I choose my compiled kernel in the grub menu, it writes in the screen "Loading linux kernel... Loading initial ramdisk", then the inscription disappears, the screen goes black and I see only a cursor in the form of underscore "_" sign in the top-left side of the screen. That's all. Nothing is going to happen further. The booting process seems to have stuck
I tried swapping make-kpkg for an old one (from Trusty), swapping compiler gcc-4.8.5 for gcc-4.9, gcc-4.7, even gcc-5.2 having made a couple of supplementations inside the directory include/linux/ for the support of gcc-5.2, but nothing has come off, the result still persists being the same
I tried the same actions (on the same Ubuntu 16.04.1 and tool-chain) with new kernels 4. series* (for example, 4.6) meaning building the kernels, packing them into *.deb packages and installing into the VirtualBox machine and rebooting the system, and everything goes correctly, I see debug messages in the screen like I have always seen. I tried to use gcc-4.7, gcc-4.8, gcc-4.9, gcc-5.4 and everything works, I am able to load the linux-kernel-v4.6 appropriately and completely. But when I build 3.10.12 (or 3.19) kernels I cannot boot them properly and cannot have figured out why it has been happening
Actually, what I have found out is that the deal is in the kernel but not in initrd because I managed to substitute the 'broken' kernel by a working one having left 'initrd' built together with the 'broken' kernel and the debug logging started appearing and the kernel was loading until a rootfs came out to be mounted, at that moment the kernel didn't manage to load it and left in initramfs mode
Has someone faced the same issue I am observing? Actually I am almost exhausted having been struggling with this trouble for days
Maybe someone has any recipes or suggestion how to get rid of the problem?
I even put the 'panic' function code exactly in the first line of the function "asmlinkage void __init start_kernel(void)" but nothing happened, still the same black screen
Can the problem be related to a new glibc being used by gcc compiling my kernel? Personally, I am not prone to think so but in the world of linux everything can happen. On the other hand maybe toolchain (ld, as) somehow has affected? I am kindly asking to provide me a help.
I am nearly assured that someone before me has already encountered such an issue, I would have been searching for a topic alike but didn't find anything resembling
Thank you in advance
Short Answer
It's a glibc kernel version mismatch, if you need this you could create the glibc package such that it supports the kernel version that you need, by using the --enable-kernel flag at configuration time.
Long Answer
It's highly likely that your glibc was compiled in such a way that it only works down to a certain version of linux. This is done with the help of the --enable-kernel flag at the configuration stage. Any version older than the one specified in --enable-kernel will be rejected by glibc as a consequence no program will ever be loaded, like the init program presumably systemd's init.
This is from the configuration help of glibc
--enable-kernel=version
This option is currently only
useful on GNU/Linux systems. The version parameter should have the form X.Y.Z and describes the smallest version of the Linux kernel the generated library is expected to support. The higher the version number is, the less compatibility code is added, and the faster the code gets.
Finally I succeeded in this problem
Actually what I have done is to have compiled an old gcc-4.8.5 with an old glibc-2.19 on the host-system where I build the old-versioned kernels.
Glibc-2.19 was compiled with an option --enable-kernel=3.10.12 and with headers of an old-versioned linux-3.10.12. The compiler has turned out to be like a 'cross-compiler' with usage of glibc-2.19. So, I built an old kernel with the version 3.10.12 with this 'cross-compiler', which uses glibc-2.19, and everything has started working in a proper way
Thanks for the help and directing me to a right way to solve the problem, but I am obliged to notice that the deal was in host-system's glibc used but not in target-system glibc used as I had been assumedly said (but maybe I misunderstood #iharob)
I have cross-compiled a kernel, in an autodidactic manner, on a raspberry pi twice in the past.
This kind of things can sometimes a pain in the ... But fortunately there are some step-by-step tutorials.
So I am wondering whether there are general steps that have to be taken and that are the same on all the embedded systems (rpi, beaglebone, atmega controllers, etc...) in order to successfully cross-compile the kernel and make everything work?
My guess:
1) download the kernel source code
2) generate a .config file (which seems necessary)
3) get into the blue screen to do additional adjustements
with e.g.: make ARCH=arm CROSS_COMPILE=/usr/bin/arm-linux-gnueabi- menuconfig
4) compile the kernel:
make ARCH=arm CROSS_COMPILE=/usr/bin/arm-linux-gnueabi-
5) put it on the SD card or anything else
Would this be a correct general scheme for any cross-compilation on an embedded system?
Sorry for my ignorance, as I mentioned above I learned it by myself.
I would like to be able to setup a kernel on any embedded device.
Any more information or explanation would be more than welcome! As it seems this kind of things can always be done in multiple manners, it gets me confused.
I'd say your first two steps haven't much to do with cross-compiling. In fact it just comes down to having a cross toolchain targeting your platform correctly installed on your system.
The CROSS_COMPILE make variable of the kernel doesn't do anything other than prepending the string it is set to to any toolchain command (like e.g. gcc for compiling), so if your cross toolchain is installed in your search path, it would be enough to set it to just the desired target triplet with added hyphen, e.g. in your case CROSS_COMPILE=arm-linux-gnueabi-. This would lead to using the command arm-linux-gnueabi-gcc for compiling and so on.
For other embedded devices, you might need different cross toolchains (depending on their architecture), but the general process would indeed stay the same.
When writing software that is CPU arch dependent, such as C code running on x86 or C code running on ARM cpus. There generally is two ways to go about compiling this code, either Cross-Compile to the ARM CPU arch (if you're developing on an x86 system for example) or copy your code to a native arch cpu system and compile it naively.
I'm wondering if there is a benefit to the native approach vs the cross-compile approach? I noticed that the Fedora ARM team is using a build-server cluster of slow/low power ARM devices to "naively" compile their Fedora ARM spin... surely a project backed by Red Hat has access to some powerful build servers running x86 cpus that could get the job done in 1/2 the time... so why their choice? Am I missing something by cross-compiling my software?
The main benefit is that all ./configure scripts do not need to be tweaked when running natively. If you are using a shadow rootfs, then you still have configurations running uname to detect the CPU type, etc. For instance, see this question. pkgconfig and other tools try to ease cross-building, but packages normally get native-building on x86 correct first, and then maybe native-building on ARM. cross-building can be painful as each package may need individual tweaks.
Finally, if you are doing profile guided optimizations and running test suitesas per Joachim, it is pretty much impossible to do this in a cross build environment.
Compile speed on the ARM is significantly faster than the human package builders, read configure, edit configure, re-run configure, compile, link cycles.
This also fits well with a continuous integration strategy. Various packages, especially libraries, can be built/deployed/tested quickly. The testing of libraries may involve hundreds of dependent packages. Arm Linux distrubutions will typically need to prototype changes when upgrading and patching a base library which may have hundreds of dependent packages that at least need retesting. A slow cycle done by a computer is always better than a fast compile followed by manual human intervention.
No technically you're not missing anything by cross-compiling within the context of .c -> .o -> a.out (or whatever); A cross compiler will give you the same binary as a native compiler (versions etc. notwithstanding)
The "advantages" of building natively come from post-compile testing and managing complex systems.
1) If I can run unit-tests quickly after compiling I can get to any bugs/issues quickly the cycle is presumably shorter than the cross-compiling cycle;
2) if I am compiling some target software that has 3rd-party libraries that it uses, then building, deploying and then using them to build my target would probably be easier on native platform; I don't want to deal with the cross-compile builds of those because half of them have build processes written by crazy monkeys that make cross compiling them a pain.
Typically for most things one would try to get to a base build and the compile the rest natively. Unless I have a sick set up where my cross compiler is super wicked fast and I the time I save there is worth the set up required to make the rest of the things (such as unit testing and dependencies management) easier.
At least those are my thoughts
The only benefit of compiling natively is that you don't have to transfer the program to the target platform as it's already there.
However that is not such a big benefit when considering that most target platforms are massively underpowered compared to a modern x86 PC. The amounts of memory, faster CPU and especially much faster disks makes compilation times many times quicker on a PC. So much so that the advantage of native building isn't really an advantage anymore.
It depends a lot on the compiler. How does the toolchain handle the difference between native and cross compile. Is it simply a case of the toolchain always thinks it being built as a cross compiler, but one way to build it is to let the configure script auto-detect the host rather than you doing it manually (and auto-set the prefix, etc)?
Dont assume that just because it is built to be a native compiler it is really native. There are many instances where distros dumb down their native compiler (and kernel and other binaries) so that that distro runs on a wider range of systems. On an ARMv6 system you might be running a compiler that defaults to ARMv4 for example.
That begs a similar question to your own, if I build the toolchain with one default architecture then specify another is that different that building the toolchain for the target architecture?
Ideally you would hope that a mostly debugged compiler/toolchain would give you the same results whether you were native or cross compiled and independent of the default architecture. Now I have seen on an older llvm that the llvm-gcc when run on a 64 bit host, cross compiling to arm would build all ints as 64 bit adding a lot to the code, same compiler version, same source code on a 32 bit host would give different results (32 bit ints). Basically the -m32 switch did not work for llvm-gcc (at the time), I dont know if that is still the case as I switched to clang when doing llvm work and never looked back at llvm-gcc...llvm/clang for example is mostly a cross compiler all the time, the linker is the only thing that appears to be host specific, you can take an off the shelf llvm and compile for any of the targets on any host system (provided your build didnt disable any of the supported targets of course).
Although many people think "local compile" benefits more or at least it has no difference compared to "cross compile", the truth is quite the contrary.
For people who works on lower level, i.e. linux kernel, they usually suffer from copy around compile platform. Take x86 and ARM as example, direct idea is building ARM compile base, but it is a bad idea.
Binary is not same sometimes, for example,
# diff hello_x86.ko hello_arm.ko
Binary files hello_x86.ko and hello_arm.ko differ
# diff hello_x86_objdump.txt hello_arm_objdump.txt
2c8
< hello_x86.ko: file format elf64-littleaarch64
---
> hello_arm.ko: file format elf64-littleaarch64
26,27c26,27
< 8: 91000000 add x0, x0, #0x0
< c: 910003fd mov x29, sp
---
> 8: 910003fd mov x29, sp
> c: 91000000 add x0, x0, #0x0
Generally higher level app is OK to use both, lower level (hardware related) work is suggested to use x86 "cross compile" since it has much better toolchain.
Anyway, compile is a work about GCC Glibc and lib.so, and if one is familiar with these, either way should be easy to go.
PS: Below is the source code
# cat hello.c
#include <linux/module.h> /* Needed by all modules */
#include <linux/kernel.h> /* Needed for KERN_ALERT */
#include <linux/init.h> /* Needed for the macros */
static int hello3_data __initdata = 3;
static int __init hello_3_init(void)
{
printk(KERN_ALERT "Hello, world %d\n", hello3_data);
return 0;
}
static void __exit hello_3_exit(void)
{
printk(KERN_ALERT "Goodbye, world 3\n");
}
module_init(hello_3_init);
module_exit(hello_3_exit);
MODULE_LICENSE("GPL");
This is a hard question to ask because I'm positive I'm about to be bombarded with haters commenting on "if I can't write an operating system already, I won't ever to be able to write an operating system". Well I've read Modern OS by Tanembaum, Linux Kernel Development, Understanding the Linux kernel and others I still don't know if or not I can write an operating system and only by pushing forward to write one will I realise what I don't know. On top of that none of the books I read even bother to describe the boot sequence / compilation sequence.
Anyway I hate to be negative but I would just like to build the example code from the bkerndev tutorial below and have an absolutely minimum operating system:
http://www.osdever.net/bkerndev/index.php?the_id=90
You can download the associated source code in a zip format from here:
http://www.osdever.net/bkerndev/bkerndev.zip
When you try and compile this kernel you run into all sorts of errors caused by the fact that some of the code is broken. Another user was seeking help for this here on stack overflow here:
compiling my own kernel (not from linux-kernel source)
Although didn't get much help. I have addressed those errors by adding the gcc flag fleading-underscores and by changing some of the data types. You can see my code here:
http://github.com/PhillipTaylor/farmix
The code will compile sucessfully and leave me with a kernel.bin executable but when I boot into it from grub I get:
Error 13: Unrecognised or unsupported format (or something to that nature)
When I take the kernel.bin directly from the authors zip file and run it on my eeepc it boots absolutely fine so I think I have a problem with compiling the code correctly. The author is building it from a Windows machine, I believe, but I am trying to compile it using Fedora 10 i386 with GNU GCC 4.3 and I think this is what is causing the issue so I ask you, how do I build a valid executable kernel? Am I missing the correct target or the wrong binary format?
I would really appreciate someone helping me over this embarrassing "first step"
My comment above wasn't very clear. What I meant is "What does the 'file' command report on your kernel.bin vs. theirs?". The output of the linker is a raw binary file. It should start with a few magic words that grub recognizes. They are defined in start.asm near "mboot". I suspect yours is different than theirs.
I don't have nasm handy so I can't build, but you might want to start by comparing the first few words of the .bin file.
It turns out that the line used to compile the app was explicitly set to compile to "aout" format which was what the guide said and what I assumed to be true. Only reading stuff in the "barebones" guide convinced me that I may have been confused. As soon as I changed that one line to "nasm -f elf" it all worked.
This is a tag in my repository that points to a basic WORKING version of bkerndev tutorial code (How to write your own Operating system) for future reference and people who were in my position..
It comes with a makefile for building it from a 32 bit Linux system.
http://github.com/PhillipTaylor/farmix/tree/bkerndev_tutorial_working
I don't quite understand the compiling process of the Linux kernel when I install
a Linux system on my machine.
Here are some things that confused me:
The kernel is written in C, however how did the kernel get compiled without a compiler installed?
If the C compiler is installed on my machine before the kernel is compiled, how can the compiler itself get compiled without a compiler installed?
I was so confused for a couple of days, thanks for the response.
The first round of binaries for your Linux box were built on some other Linux box (probably).
The binaries for the first Linux system were built on some other platform.
The binaries for that computer can trace their root back to an original system that was built on yet another platform.
...
Push this far enough, and you find compilers built with more primitive tools, which were in turn built on machines other than their host.
...
Keep pushing and you find computers built so that their instructions could be entered by setting switches on the front panel of the machine.
Very cool stuff.
The rule is "build the tools to build the tools to build the tools...". Very much like the tools which run our physical environment. Also known as "pulling yourself up by the bootstraps".
I think you should distinguish between:
compile, v: To use a compiler to process source code and produce executable code [1].
and
install, v: To connect, set up or prepare something for use [2].
Compilation produces binary executables from source code. Installation merely puts those binary executables in the right place to run them later. So, installation and use do not require compilation if the binaries are available. Think about ”compile” and “install” like about “cook” and “serve”, correspondingly.
Now, your questions:
The kernel is written in C, however how did the kernel get compiled without a compiler installed?
The kernel cannot be compiled without a compiler, but it can be installed from a compiled binary.
Usually, when you install an operating system, you install an pre-compiled kernel (binary executable). It was compiled by someone else. And only if you want to compile the kernel yourself, you need the source and the compiler, and all the other tools.
Even in ”source-based” distributions like gentoo you start from running a compiled binary.
So, you can live your entire life without compiling kernels, because you have them compiled by someone else.
If the C compiler is installed on my machine before the kernel is compiled, how can the compiler itself get compiled without a compiler installed?
The compiler cannot be run if there is no kernel (OS). So one has to install a compiled kernel to run the compiler, but does not need to compile the kernel himself.
Again, the most common practice is to install compiled binaries of the compiler, and use them to compile anything else (including the compiler itself and the kernel).
Now, chicken and egg problem. The first binary is compiled by someone else... See an excellent answer by dmckee.
The term describing this phenomenon is bootstrapping, it's an interesting concept to read up on. If you think about embedded development, it becomes clear that a lot of devices, say alarm clocks, microwaves, remote controls, that require software aren't powerful enough to compile their own software. In fact, these sorts of devices typically don't have enough resources to run anything remotely as complicated as a compiler.
Their software is developed on a desktop machine and then copied once it's been compiled.
If this sort of thing interests you, an article that comes to mind off the top of my head is: Reflections on Trusting Trust (pdf), it's a classic and a fun read.
The kernel doesn't compile itself -- it's compiled by a C compiler in userspace. In most CPU architectures, the CPU has a number of bits in special registers that represent what privileges the code currently running has. In x86, these are the current privilege level bits (CPL) in the code segment (CS) register. If the CPL bits are 00, the code is said to be running in security ring 0, also known as kernel mode. If the CPL bits are 11, the code is said to be running in security ring 3, also known as user mode. The other two combinations, 01 and 10 (security rings 1 and 2 respectively) are seldom used.
The rules about what code can and can't do in user mode versus kernel mode are rather complicated, but suffice to say, user mode has severely reduced privileges.
Now, when people talk about the kernel of an operating system, they're referring to the portions of the OS's code that get to run in kernel mode with elevated privileges. Generally, the kernel authors try to keep the kernel as small as possible for security reasons, so that code which doesn't need extra privileges doesn't have them.
The C compiler is one example of such a program -- it doesn't need the extra privileges offered by kernel mode, so it runs in user mode, like most other programs.
In the case of Linux, the kernel consists of two parts: the source code of the kernel, and the compiled executable of the kernel. Any machine with a C compiler can compile the kernel from the source code into the binary image. The question, then, is what to do with that binary image.
When you install Linux on a new system, you're installing a precompiled binary image, usually from either physical media (such as a CD DVD) or from the network. The BIOS will load the (binary image of the) kernel's bootloader from the media or network, and then the bootloader will install the (binary image of the) kernel onto your hard disk. Then, when you reboot, the BIOS loads the kernel's bootloader from your hard disk, and the bootloader loads the kernel into memory, and you're off and running.
If you want to recompile your own kernel, that's a little trickier, but it can be done.
Which one was there first? the chicken or the egg?
Eggs have been around since the time of the dinosaurs..
..some confuse everything by saying chickens are actually descendants of the great beasts.. long story short: The technology (Egg) was existent prior to the Current product (Chicken)
You need a kernel to build a kernel, i.e. you build one with the other.
The first kernel can be anything you want (preferably something sensible that can create your desired end product ^__^)
This tutorial from Bran's Kernel Development teaches you to develop and build a smallish kernel which you can then test with a Virtual Machine of your choice.
Meaning: you write and compile a kernel someplace, and read it on an empty (no OS) virtual machine.
What happens with those Linux installs follows the same idea with added complexity.
It's not turtles all the way down. Just like you say, you can't compile an operating system that has never been compiled before on a system that's running that operating system. Similarly, at least the very first build of a compiler must be done on another compiler (and usually some subsequent builds too, if that first build turns out not to be able to compile its own source code just yet).
I think the very first Linux kernels were compiled on a Minix box, though I'm not certain about that. GCC was available at the time. One of the very early goals of many operating systems is to run a compiler well enough to compile their own source code. Going further, the first compiler was almost certainly written in assembly language. The first assemblers were written by those poor folks who had to write in raw machine code.
You may want to check out the Linux From Scratch project. You actually build two systems in the book: a "temporary system" that is built on a system you didn't build yourself, and then the "LFS system" that is built on your temporary system. The way the book is currently written, you actually build the temporary system on another Linux box, but in theory you could adapt it to build the temporary system on a completely different OS.
If I am understanding your question correctly. The kernel isn't "compiling itself" these days. Most Linux distributions today provide system installation through a linux live cd. The kernel is loaded from the CD into memory and operates as it would normally as if it were installed to disk. With a linux environment up and running on your system it is easy to just commit the necessary files to your disk.
If you were talking about the bootstrapping issue; dmckee summed it up pretty nice.
Just offering another possibility...