Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 8 years ago.
Improve this question
How can it be that a file on an ext3 partion in Linux, which, for example, has different metadata, can get transferred to a Windows NTFS and we still can open and read it correctly?
Don't you have to convert it somehow to make it compatible?
As far as I understand the metadata of the two FS are different, but what happens to these different metadata?
A file system is actually an abstract user interface to access the data behind it. It works in the same way that you can access data from a DB through a web-page.
You acess this interface with file utilities which create, list, copy, move and delete files, and alter metadata. You'll need then some NTFS utils, ext3 utils and so on (it's not a given that they will be present).
There are several aspects that the program doing the transfer (for example, nautilus) has to deal with:
-how to deal with long names and non standard characters like blank spaces, non ASCII (normally copying fails here, so better avoid this)
-endianess (the order of storing bytes). It's not the same reading 0A0B0C0D from left to right than from right to left. Both methods are at use, but the problem is old and therefore tools can deal with it, normally.
-things like Linux permissions get compromised when copying files through file systems (when transfering the file, not just accessing them through a file server like Samba). The recipient can change them to whatever he wants, being root and all. File systems like FAT don't support security at all, so as soon as you copy the file to it the security information is simply lost. Linux OSs can apply a standard set of permissions (for example, with umask, not letting any file being executable).
How a file gets copied:
Open old file to read.
Open new file to write.
Read/write bytes between files.
The file systems involved don't matter.
Related
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 7 years ago.
Improve this question
I know that a loader loads a program into memory. But how can I implement it using a program? Using assembly or C. This might be very useful. Or atleat a reference.
Maybe you already understand this, not sure. A program loader at a high level simply reads/downloads/accepts the program, parses the file format if required. Places the program in memory, and jumps/branches to it.
Now if you get more specific, say a bootloader for a processor you generally dont have a file system yet or such things so maybe you can only accept programs that are already on the flash, one of your main use cases, or allow developers to download test versions, destined to be the program on the flash, xmodem, ymodem, or other protocols. Maybe if there is ethernet then that way or usb if available and makes sense or removable media (sd cards, etc). At the end of the day you still support some type of format be it just the raw memory image of the program or some other formats (intel hex, motorola srecord, maybe elf, etc).
An operating system has a lot more work to do, because take windows or linux or mac right now, write a simple application that reads and parses a simple program, read that program into your applications memory space or malloc some, whatever, then try to branch to it. The operating system stops you, there are ways around this, but that is not the point, you are an application you are not the operating system. But if you were the operating system loader, then you simply have more permissions, being the operating system you have designed what your file format is, what the agreed entry point address is, what the system interface is for applications making calls, etc. Programs have to conform to your rules, you would then read the binary, parse it (perhaps you only support .elf file formats for example), allocate memory for the program per your rules and the programs desired allocation of resources (ideally, initially, part of the file format), per your operating systems rules you setup the virtual address space and point it where the program has been loaded, and then branch to the program changing from super user to user mode on the way.
Your question is extremely vague though, cant understand if you understand the basics and want detail (an application is not a loader on an os with protection, so simply go read the source for linux or bsd, etc), or dont understand the basics (make a little bootloader for a microcontroller or use an instruction set simulator if you dont want to buy a microcontroller).
I feel as if the best manner of doing what I think you are trying to do is fork a process off, and create a process running within it? This is, if it's what you're asking best done with the unistd.h library, in both C and C++, and if you want to get a bit more direct the PThreads library. However if you don't at the moment know how these things are called, I recommend heavy reading before you mistakenly create a fork bomb, and crash your system.Look into the openpub documentation if needed. However I heavily recommend cleaning up this question, and I also feel that it's been asked a bit often on this site as well.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 7 years ago.
Improve this question
I have developed a basic kernel in assembly/c that runs a basic terminal. I have set it up to run off of an iso with grub.
I would like to continue this OS, but without a file system, I feel as if there's really nothing else I could do. After much time on the internet, I have come up with really nothing I can do to implement this.
People have said implement FAT or make a VFS, but nothing any further, nor tutorials, nor any references to anywhere.
Could someone explain how a file system works, where I can get started/where I can connect a pre-made system, and how to use it?
Also, I do not have access to standard libraries when compiling my os. I use gcc, nasm, ld, and grub-mkrescue(for the disk image). I use qemu for emulation.
EDIT to make less OT
Can someone describe, in detail how a file system works, so when I look at the sources of file systems that have been implemented, like FAT, I can understand how to apply it to my own operating system?
EDIT - Simpler
Even easier. How could I directly access the hard drive? My kernel runs completely in protected mode, so could I switch out and write directly to the hard drive. A file system could be implemented with a file looking like this:
name special char text special char
ie:
hello world.script 0x00 println "Hello, world!!" 0x00
Where you wouldn't need special segmentation, you would just look until you find the file name and the special character (something not in a string like '\0') and then read until you find the second non-string character.
Would there be a way to access the hard drive by switching in and out of protected mode or write a hard disk driver in order to implement this?
First, read wikipage on file systems to have some broad view.
The relevant resource about operating system development is OSdev (but perhaps your question is off-topic here). Kernelnewbies could also help (explaining how Linux is doing). OSdev have wikipages explaining FAT & Ext2 in details.
You could design an OS without any files (but some other persistence machinery). See this answer. You could have persistent processes (read also about application checkpointing, garbage collection, continuations, hibernation).
But you should read some good books about Operating Systems (e.g. by Tanenbaum, or the freely downloadable Operating Systems: Three Easy Pieces book). Be fluent with some existing free software OS, e.g. Linux (& POSIX), so read Advanced Linux Programming (at least to understand many concepts and get a good terminology).
IMHO, the FAT is such an ugly and inefficient file system that it is not worth looking into (except for legacy and compatibility reasons). Ext4 (see here) should be better & the wikipage on Ext2 has a nice picture.
You could adapt some library providing a file system (e.g. libext2) to your kernel.
You could perhaps adapt sqlite to work on a raw disk partition.
You might have a notion of file which is not like MSDOS (or Windows) or POSIX or <stdio.h> files. For example, it might be a sequence of fixed size records (e.g. of 1Kbyte), not a stream of bytes.
You could organize your OS as a microkernel and have file systems given by application code. Look into VSTa and HURD.
You need of course a disk driver, which fetches/writes blocks (of 4Kbytes) from your drive (disk I/O is always by blocks or disk sectors. Old small disks had 512 bytes blocks. New large disks have 4Kbytes ones, see advanced format). It should be interrupt driven and uses DMA. You need a task scheduler. AFAIU, you won't use the BIOS for this (perhaps the UEFI); you need to understand how common hardware (SATA & AHCI) works.
You should publish (today!) your toy OS as free software (e.g. under GPLv3+ on github) to get feedbacks and contributions.
You might copy (if licenses are compatible) existing code from other free software operating systems, and you certainly will study their source code to understand things.
So code some task scheduler, a page fault handler, a virtual memory, then add interrupt driven disk IO, and some file system code above that. Then you'll beginning to understand that an OS cannot be a small toy.... You might consider a microkernel or exokernel approach.
It would be simplest to use an existing open-source filesystem if the licence terms suit your needs. ELM FatFs is one such library, with no usage restrictions whatsoever. You only need to provide the device control interface layer using the provided stubs and examples.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 years ago.
Improve this question
Regarding the execution of the c program, i am aware that compiler converts the code into machine language and then it is executed.But i would like to know which services are provided by the operating system to accomplish that. say iam using fopen function, please explain me how the operating system handles it, ie. reading the file from hard disk to loading into memory...etc . for all those operations,which system calls are internally called?how the explicit functions like fopen,printf are converted into system calls?
If it is possible to view the internal system calls in context to c programming,please let me know the path to be followed to see them?
Languages typically have their own APIs as part of their run-time support (e.g. fopen() in C's standard library). These are part of the language and not strictly part of the OS itself.
The language's run-time uses the OS's lower level APIs. For example, fopen() might use the kernel API's open() function (Linux); but then it might be a createfile() function in a DLL and not something in the kernel at all (Windows). In some cases, it's nothing like that and more like a message sent to a different process (common for micro-kernels).
Regardless of where it ends up (and how), it probably finds its way to some sort of "virtual file system" layer, and depending on whether it's in the VFS's caches it may or may not get forwarded from there to code responsible for handling a file system, which may or may not forward it to some sort of storage device driver (e.g. a USB flash device driver), which in turn might forward it to another device driver (e.g. a USB controller driver).
Mostly, it can be very different for different OSs, so there is no single answer that's correct for all of them.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 8 years ago.
Improve this question
I'm looking for some filesystem, or archiving file format, with transparent compressing/decompressing support, and can be opened in windows (as a file) by some tool(like winrar or 7-zip), and have some easy to use c interfaces
As far as I've found, zip fileformat is the closest, transparent decompressing, simple c interface (Minizip), but zip format is not suited for file lookup by filename. the UDF filesystem (in iso format) may be bettar at filename lookup (it's a guess, I haven't look very deep), but I haven't found a c interface
Yet I think there must be a more obvious and simpler approach than zip and iso, so, is there?
UPDATE
I have failed to find such a filesystem, especially transparent decompressing. yet there are good ones, like whefs, but without transparent decompressing. and none filesystem approaches, namely DBM(CDB, gdbm, etc)
Zip is the de facto standard solution for compression and archiving. Judging by the following diagram of the format (from Wikipedia), lookup by name occurs in the "central directory" and should not involve a scan through the entire file.
zziplib (and probably others) allows you to read the directory inside a zip file like it were a normal filesystem directory.
If you need archiving (i.e. sequential compressing and decompression, with no modifications), then ZIP is a standard and probably there's nothing close to ZIP in popularity. However ZIP is not effective when you manipulate files in it (i.e. when you need a virtual file system). In the latter case you can use CodeBase File System, Solid File System (our product) or one of similar products.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 7 years ago.
Improve this question
I'm not sure what the "general" name of something like this might be. I'm looking for a library that gives me a file format to store different types of binary data in an expanding single file.
open source, non-GPL (LGPL ok)
C interface
the file format is a single file
multiple files within using a POSIX-like file API (or multiple "blobs" within using some other API)
file/structure editing is done in-place
reliable first, performant second
Examples include:
the virtual drives of a virtual machine
whefs
HDF
CDF
NetCDF
Problems with the above:
whefs doesn't appear to be very mature, but best describes what I'm after
HDF, CDF, NetCDF are usable (also very reliable and fast), but they're rather complicated and I'm not entirely convinced of their support for opaque binary "blobs"
Edit:
Forgot to mention, one other relevant question:
Simple Virtual Filesystem in C/C++
Another similar question:
Is there an open-source alternative to Windows compound files?
Edit:
Added condition of in-place editing.
Edit:
whefs superseded by: whio_epfs
This appears to do what I was looking for: libgsf
Still need to test its reliability/performance and how cross-platform the binary format is.
It sounds like you're talking about the Linux loopback device, which lets you treat a file on a filesystem as a first-class block device (and then proceed to mkfs, mount, etc.)
(What sort of platform are you targetting? A fully-featured Unixlike? Something in the embedded space with a small footprint?)
The WxWindows library supports ZIP files (see http://docs.wxwidgets.org/stable/wx_wxarc.html#wxarc).
This has also the advantage that you can look at the contents using a ZIP manager (e.g. WINZIP).
A commercial alternative is ChillKat (http://www.chilkatsoft.com/)
If security is a concern, encrypt the file contents and mangle the file names in the ZIP archive.
Eet library from the Enlightenment project maybe?
http://en.wikipedia.org/wiki/Enlightenment_Foundation_Libraries#EET
http://docs.enlightenment.org/api/eet/html/
What about BerkeleyDB? It's not exactly a filesystem but it's quite transparent to store 'binary data' in a file. License seems to be quite permissive as well.