Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 7 years ago.
Improve this question
I have developed a basic kernel in assembly/c that runs a basic terminal. I have set it up to run off of an iso with grub.
I would like to continue this OS, but without a file system, I feel as if there's really nothing else I could do. After much time on the internet, I have come up with really nothing I can do to implement this.
People have said implement FAT or make a VFS, but nothing any further, nor tutorials, nor any references to anywhere.
Could someone explain how a file system works, where I can get started/where I can connect a pre-made system, and how to use it?
Also, I do not have access to standard libraries when compiling my os. I use gcc, nasm, ld, and grub-mkrescue(for the disk image). I use qemu for emulation.
EDIT to make less OT
Can someone describe, in detail how a file system works, so when I look at the sources of file systems that have been implemented, like FAT, I can understand how to apply it to my own operating system?
EDIT - Simpler
Even easier. How could I directly access the hard drive? My kernel runs completely in protected mode, so could I switch out and write directly to the hard drive. A file system could be implemented with a file looking like this:
name special char text special char
ie:
hello world.script 0x00 println "Hello, world!!" 0x00
Where you wouldn't need special segmentation, you would just look until you find the file name and the special character (something not in a string like '\0') and then read until you find the second non-string character.
Would there be a way to access the hard drive by switching in and out of protected mode or write a hard disk driver in order to implement this?
First, read wikipage on file systems to have some broad view.
The relevant resource about operating system development is OSdev (but perhaps your question is off-topic here). Kernelnewbies could also help (explaining how Linux is doing). OSdev have wikipages explaining FAT & Ext2 in details.
You could design an OS without any files (but some other persistence machinery). See this answer. You could have persistent processes (read also about application checkpointing, garbage collection, continuations, hibernation).
But you should read some good books about Operating Systems (e.g. by Tanenbaum, or the freely downloadable Operating Systems: Three Easy Pieces book). Be fluent with some existing free software OS, e.g. Linux (& POSIX), so read Advanced Linux Programming (at least to understand many concepts and get a good terminology).
IMHO, the FAT is such an ugly and inefficient file system that it is not worth looking into (except for legacy and compatibility reasons). Ext4 (see here) should be better & the wikipage on Ext2 has a nice picture.
You could adapt some library providing a file system (e.g. libext2) to your kernel.
You could perhaps adapt sqlite to work on a raw disk partition.
You might have a notion of file which is not like MSDOS (or Windows) or POSIX or <stdio.h> files. For example, it might be a sequence of fixed size records (e.g. of 1Kbyte), not a stream of bytes.
You could organize your OS as a microkernel and have file systems given by application code. Look into VSTa and HURD.
You need of course a disk driver, which fetches/writes blocks (of 4Kbytes) from your drive (disk I/O is always by blocks or disk sectors. Old small disks had 512 bytes blocks. New large disks have 4Kbytes ones, see advanced format). It should be interrupt driven and uses DMA. You need a task scheduler. AFAIU, you won't use the BIOS for this (perhaps the UEFI); you need to understand how common hardware (SATA & AHCI) works.
You should publish (today!) your toy OS as free software (e.g. under GPLv3+ on github) to get feedbacks and contributions.
You might copy (if licenses are compatible) existing code from other free software operating systems, and you certainly will study their source code to understand things.
So code some task scheduler, a page fault handler, a virtual memory, then add interrupt driven disk IO, and some file system code above that. Then you'll beginning to understand that an OS cannot be a small toy.... You might consider a microkernel or exokernel approach.
It would be simplest to use an existing open-source filesystem if the licence terms suit your needs. ELM FatFs is one such library, with no usage restrictions whatsoever. You only need to provide the device control interface layer using the provided stubs and examples.
Related
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 2 years ago.
Improve this question
I programmed a little code for motion detection device(DA14531 SmartBond TINY module based on a CortexM0+) and I am having some experiments with it. at the end and after debugging and testing, I generated an ISO file and now I want to flash the device. Is the process similar to burning the ISO file on a USB flash or is the process different? I only have one device and I dont want to do something irreversible so I came here for some guidance first.
I looked online for a while but nothing matches my specifique situation, so providing me with the correct links is also helpful
The ISO 9660 format was designed for optical disks, using it is likely irrelevant to your use case, since there is IMHO a near to zero chance you will find a tool that will allow you to flash directly your program in a Cortex-M0+ device from a file in ISO 9660 format.
And if you may flash the ISO file as is in your Cortex-M0+ flash memory, your device will likely be unable to boot since it does rely on very specific information (stack pointer, first instruction to be executed) to be flashed in a verify specific location, not mentioning the waste of flash memory space this would cause.
That is, if Dialog documentation does not specifically mention the possibility of flashing a file in ISO 9660 format, this is likely (and not surprising) that this is not possible using Dialog software and hardware support tools.
So when you read the documentation for this product you noticed there is an SWD interface which is certainly one way into the part. When you further examined the pro kit and other solutions from them you see they mention segger jlink interfaces for debugging etc. Further reinforcing SWD as at least one interface into the part. Through that interface (SWD is ARM) you access the flash controller (has nothing whatsoever to do with arm it is chip specific) and through that you write your application binary that the part will run (the machine code and data that the processor uses, your application).
ISO is closely related to PC's with a BIOS/EFI which also means x86, and has nothing whatsoever to do with a microcontroller much less a non x86, non BIOS/EFI PC/Laptop. It is extremely unlikely that you can get enough software on a cortex-m0(+) based platform that if you had an interface to media that can hold it that you could parse it and extract anything useful, and then have resources left to load and execute any programs in ram. No way whatsoever in any part I have heard of that you could do this in ram such that you could extract something you could load to the flash on the part. Plus you have to get that program into the part before you can then later support ISO, if you could, which you can't.
The only remote way an ISO makes any sense at all or has any context is if on your PC you boot off of an ISO image and that ISO image for the PC (not the mcu) contains a development system. For example a pre-prepared Linux operating system distro with the tools from the vendor for this part so that you don't have to install the development system on your computer you can run it off a ramdisk using a live image on an ISO. That development system would not use ISO files but the proper file formats to develop binaries and load them to the board via SWD or some other chip/board specific interface.
Beyond that there is no further reason to talk about ISO's and microcontrollers.
Some chip vendors (not arm, the chip vendor) may also provide a factory bootloader or logic that supports for example a uart, maybe spi, maybe i2c, maybe usb interface that you can use chip (not arm) specific software to talk to software running on that chip (the bootloader) that can then write to the flash. You can also write your own bootloader if there are enough resources in the system. The (arm based) mcu world is moving away from these bootloaders, two of the three main companies that used to always have them have started to remove them or disable them as a default feature.
Other companies provide no other interface than SWD to program the part, SWD or nothing. Certainly in the cortex-m0+ market where every penny counts and the extra flash for a bootloader and extra chip real estate, etc add to the overall cost for a legacy feature that is becoming less important because developers can now easily obtain SWD interface modules for a few dollars. It is not like the old days where a JTAG board cost $2000. At this time all cortex-m parts support SWD, making it the most useful interface and reminding developers that having tools that can access that interface being worth the ($5, plus time to learn to use it) investment.
The tools used to write the flash dictate what file formats are supported, these days a raw binary image or an elf file format are the main two. The old days included file formats like Intel hex and Motorola s-record but it is only old timers like me that favor those file formats, even though an elf is trivial to parse, and a raw binary image even simpler, about four lines of code.
Some chip vendors do not provide enough information to roll your own, but most often they do. Arm long ago released the SWD interface information, so it is technically possible to roll your own and then support whatever file format you want. But you would have to distribute this tool along with the ISO file, so you would what use a second ISO file to distribute the tools to read the first one. Based on your question and comments you are a long long way from writing tools like these. Especially when working tools like openocd exist that support the main file formats (elf and raw binary) and can speak SWD into the current line of cortex-m cores.
Again if you are suggesting using an ISO to distribute tools along with your binary to be loaded and run on a PC that might make sense, but it is easier for the end user to simply download the tools from the chip vendor or tools vendor and then download the binary file from you, rather than put in the extra work to deal with an ISO.
I searched for info about this but didn't find anything.
The idea is:
If I code a program in C, or any other languages, what else do I need to do for it to get recognized in BIOS and started by it as a DOS program or just a prompt program?
I got this idea after I booted an flash drive with windows using the ISO and Rufus, which put some code in the flash drive for the BIOS to recognize it and run, so I would like to do the same with a program of mine, for example.
Thanks in advance!
An interesting, but rather challenging exercise!
The BIOS will fetch a specific zone from the boot device, called a master boot record. In a "normal" situation with an OS and one or more partitions, the MBR will need to figure out where to find the OS, load that into memory, and pass control to it. At that time the regular boot sequence starts and somewhat later the OS will be running and be able to interact with you. More detail on the initial activities can be found here
Now, for educational purposes, this is not strictly necessary. You could write an MBR that just reads in a fixed part of the disk (the BIOS has functions that will allow you to read raw sectors off a disk, a disk can be considered as just a bunch of sectors each containing 512 bytes of information) and starts that code. You can find an open source MBR here and basically in any open source OS.
That was the "easy" part, because now you probably want to do something interesting. Unless you want to interact with each part of the hardware yourself, you will have to rely on the services provided by the BIOS to interact with keyboard, screen and disk. The traditionally best source about BIOS services is Ralf Brown's interrupt list.
One specific consideration: your C compiler comes with a standard library, and that library will need a specific OS for many of its operations (eg, to perform output to the screen, it will ask the operating system to perform that output, and the OS will typically use the BIOS or some direct access to the hardware to perform that task). So, in going the route explained above, you will also need to figure out a way to replace these services by some that use the BIOS and nothing more - ie, more or less rewrite the standard library.
In short, to arrive at something usable, you will be writing the essential parts of an operating system...
Actually BIOS is going to be dead in the next two years (INTEL will not support any BIOSes after this date) so you may want to learn UEFI standard. UEFI from v2.4 allows to write and add custom UEFI applications. (BTW the "traditional" BIOS settings on the UEFI computers is often implemented as a custom UEFI App).
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 7 years ago.
Improve this question
I know that a loader loads a program into memory. But how can I implement it using a program? Using assembly or C. This might be very useful. Or atleat a reference.
Maybe you already understand this, not sure. A program loader at a high level simply reads/downloads/accepts the program, parses the file format if required. Places the program in memory, and jumps/branches to it.
Now if you get more specific, say a bootloader for a processor you generally dont have a file system yet or such things so maybe you can only accept programs that are already on the flash, one of your main use cases, or allow developers to download test versions, destined to be the program on the flash, xmodem, ymodem, or other protocols. Maybe if there is ethernet then that way or usb if available and makes sense or removable media (sd cards, etc). At the end of the day you still support some type of format be it just the raw memory image of the program or some other formats (intel hex, motorola srecord, maybe elf, etc).
An operating system has a lot more work to do, because take windows or linux or mac right now, write a simple application that reads and parses a simple program, read that program into your applications memory space or malloc some, whatever, then try to branch to it. The operating system stops you, there are ways around this, but that is not the point, you are an application you are not the operating system. But if you were the operating system loader, then you simply have more permissions, being the operating system you have designed what your file format is, what the agreed entry point address is, what the system interface is for applications making calls, etc. Programs have to conform to your rules, you would then read the binary, parse it (perhaps you only support .elf file formats for example), allocate memory for the program per your rules and the programs desired allocation of resources (ideally, initially, part of the file format), per your operating systems rules you setup the virtual address space and point it where the program has been loaded, and then branch to the program changing from super user to user mode on the way.
Your question is extremely vague though, cant understand if you understand the basics and want detail (an application is not a loader on an os with protection, so simply go read the source for linux or bsd, etc), or dont understand the basics (make a little bootloader for a microcontroller or use an instruction set simulator if you dont want to buy a microcontroller).
I feel as if the best manner of doing what I think you are trying to do is fork a process off, and create a process running within it? This is, if it's what you're asking best done with the unistd.h library, in both C and C++, and if you want to get a bit more direct the PThreads library. However if you don't at the moment know how these things are called, I recommend heavy reading before you mistakenly create a fork bomb, and crash your system.Look into the openpub documentation if needed. However I heavily recommend cleaning up this question, and I also feel that it's been asked a bit often on this site as well.
Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 11 years ago.
Improve this question
Does anyone know where I might find sample solutions written in C for low level / systems level applications? A really good website or book recommendation would be cool too.
I've learned some of the basics, but would like to see some code within the context of a real solution written in C, and specifically for a lower-level problem. Id' be interested in how C is used within the context of OS programming, for example. What are some areas where C is used for lower-level programming?
Thanks.
I would suggest you to study MINIX3 from Tanenbaum: http://www.minix3.org/
Its a microkernel architecture, and with his book ( http://vig.prenhall.com/catalog/academic/product/0,1144,0131429388,00.html ) it is really enlightning.
As of my opinion, studying the linux kernel is a bit hardcore for a start ;), and out of a academical point of view the microkernel architecture is superior to the monolithic kernel.
Furthermore, with only a few thousands lines of code, unlike the Linux Kernel, its consumable in a realistic timetable.
And its a real serious project, the European Union sponsored some Millions towards it as far as i am aware of. I think i remind him saying that in one of his talks.
And you have a X-Server running there, a gcc-toolchain etcpp.
Have fun :)
EDIT: As i read the comments, someone mentions the Ruby interpreter. Its written in a mixture of C and Ruby, and as far as it was mentioned in one episode of se-radio.net, it is really nice sourcecode. Though i have to admit, i havent looked into it myself. Might be worth the dig into it if you have some interest in Ruby too.
I'd suggest looking at some (for you) interesting open source projects written in C. For example, there's busybox, a piece of software that runs on embedded devices and has lots of smaller programs to study. You could, for example, take the source for the telnet client on one side and the corresponding RFC on the other. Or, for a steeper learning curve, you could also try studying the open source OSes, like the Linux kernel (here's the tree for browsing) or the BSDs. It's a lot more involved than busybox, but you can still find some parts that are fairly easy to understand if you're familiar with the context.
Studying the Linux kernel, maybe in conjunction with one of the several books on the kernel or device drivers would provide a wealth of material. Much of this is available free.
any or all of the books by W. Richard Stevens that walk though the implementation (TCP/IP Illustrated) or use (UNIX Network Programming) of the networking stack or his Advanced Programming in the UNIX Environment book.
If you have a leaning toward Windows there are several good books, even if they're quite old, including:
Programming Server-Side Applications for Microsoft Windows 2000 by Richter and Clark
Programming Applications for Microsoft Windows by Richter
I would suggest the following sources might be interesting r.e. Operating Systems from a learning perspective. Be aware there have been many advancements actually present in modern kernels:
The original linux code.
xv6. This is a simple unix OS that goes along with MIT's excellent OpenCourseWare course on Operating Systems.
Other ideas:
The current grub stage 1 bootloader isn't that complicated - it's pretty hard to be complicated with 512 bytes to play with.
The Linux kernel module guide gives you an introduction to building kernel modules. You could experiment with building custom, yet pointless, drivers that add say character devices to /dev/ or proc devices to /proc and work towards implementing something interesting. People have implemented web servers in kernel space...
If you want to experiment with Windows kernels, have a go with Native NT applications. I'd start with printing a pointless boot message, then move up to drivers.
Beyond that, it's hard to suggest where you might want to go. Systems level is a wide space.
In the context of low level programming, C and C++ are portable assembler. In many of the above spaces the standard library is either partially or totally missing and extra functionality may be implemented by existing parts of the system-level code you're modifying, so you have to be aware of the API functions available to you in any given space and what you need to implement yourself, as well as what your memory and processing requirements must be. For example, a bootloader written to the MBR has to use bios interrupts and starts in real (16-bit) mode. Those are the constraints of the hardware design. Likewise, functions like fopen() aren't available in kernel space since they wrap system calls - you'd need to use kernel specific constructs to achieve this if it really made sense to write a file from kernel space.
I am wondering how the OS is reading/writing to the hard drive.
I would like as an exercise to implement a simple filesystem with no directories that can read and write files.
Where do I start?
Will C/C++ do the trick or do I have to go with a more low level approach?
Is it too much for one person to handle?
Take a look at FUSE: http://fuse.sourceforge.net/
This will allow you to write a filesystem without having to actually write a device driver. From there, I'd start with a single file. Basically create a file that's (for example) 100MB in length, then write your routines to read and write from that file.
Once you're happy with the results, then you can look into writing a device driver, and making your driver run against a physical disk.
The nice thing is you can use almost any language with FUSE, not just C/C++.
I found it quite easy to understand a simple filesystem while using the fat filesystem on the avr microcontroller.
http://elm-chan.org/fsw/ff/00index_e.html
Take look at the code you will figure out how fat works.
For learning the ideas of a file system it's not really necessary to use a disk i think. Just create an array of 512 byte byte-arrays. Just imagine this a your Harddisk an start to experiment a bit.
Also you may want to hava a look at some of the standard OS textbooks like http://codex.cs.yale.edu/avi/os-book/OS8/os8c/index.html
The answer to your first question, is that besides Fuse as someone else told you, you can also use Dokan that does the same for Windows, and from there is just a question of doing Reads and Writes to a physical partition (http://msdn.microsoft.com/en-us/library/aa363858%28v=vs.85%29.aspx (read particularly the section on Physical Disks and Volumes)).
Of course that in Linux or Unix besides using something like Fuse you only have to issue, a read or write call to the wanted device in /dev/xxx (if you are root), and in these terms the Unices are more friendly or more insecure depending on your point of view.
From there try to implement a simple filesystem like Fat, or something more exoteric like an tar filesystem, or even some simple filesystem based on Unix concepts like UFS or Minux, or just something that only logs the calls that are made and their arguments to a log file (and this will help you understand, the calls that are made to the filesystem driver during the regular use of your computer).
Now your second question (that is much more simple to answer), yes C/C++ will do the trick, since they are the lingua franca of system development, also a lot of your example code will be in C/C++ so you will at least read C/C++ in your development.
Now for your third question, yes, this is doable by one person, for example the ext filesystem (widely known in Linux world by it's successors as ext2 or ext3) was made by a single developer Theodore Ts'o, so don't think that these things aren't doable by a single person.
Now the final notes, remember that a real filesystem interacts with a lot of other subsystems in a regular kernel, for example, if you have a laptop and hibernate it the filesystem has to flush all changes made to the open files, if you have a pagefile on the partition or even if the pagefile has it's own filesystem, that will affect your filesystem, particularly the block sizes, since they will tend to be equal or powers of the page block size, because it's easy to just place a block from the filesystem on memory that by coincidence is equal to the page size (because that's just one transfer).
And also, security, since you will want to control the users and what files they read/write and that usually means that before opening a file, you will have to know what user is logged on, and what permissions he has for that file. And obviously without filesystem, users can't run any program or interact with the machine. Modern filesystem layers, also interact with the network subsystem due to the fact that there are network and distributed filesystems.
So if you want to go and learn about doing kernel filesystems, those are some of the things you will have to worry about (besides knowing a VFS interface)
P.S.: If you want to make Unix permissions work on Windows, you can use something like what MS uses for NFS on the server versions of windows (http://support.microsoft.com/kb/262965)