I have a few dd images and I wanted to know the best way of extracting files out of these using C. The images are of mixed file systems (fat32, ntfs, ext2/3) and the host machine doing the extraction would be an Ubuntu box, so you can assume kernel headers and GNU C library, etc.
Natively would be best, but external libraries that do the job would also be fine. A link to an example would be perfect.
This is a significant effort. You'd have to essentially reimplement filesystem drivers for NTFS, FAT, and EXT2. I've done this for FAT and NTFS, but it took more than two years, though much of that was reverse engineering NTFS.
Consider using the file mount option of the mount command so you can use the Ubuntu filesystem drivers and not reinvent the significantly large wheel. Then you can peruse the mounted filesystems.
Why programatically with C?
sudo mount -o loop,offset=[offset] -t auto [where] [what]
Where
offset is the offset in bytes from the beginning of the disk, in bytes
where is where on the current filesystem to mount the image
what is the disk image you're mounting
Look at The Sleuth Kit, which should work with all of the file system types you listed:
The original part of Sleuth Kit is a C library and collection of command line file and volume system forensic analysis tools. The file system tools allow you to examine file systems of a suspect computer in a non-intrusive fashion. Because the tools do not rely on the operating system to process the file systems, deleted and hidden content is shown. It runs on Windows and Unix platforms.
The Sleuth Kit's existing toolset is a great place to start if you're looking for sample code to work from.
Check out:
ext2fuse or fuse-ext2 projects, they contain some ext2/ext3 implementations on user space using FUSE.
ntfs-3g, an NTFS implementation also using FUSE.
FUSE itself, as there are lots of filesystem implementations on top of it.
Related
This is just for self-study purpose.
In Windows, Linux or Mac, we can mount many kinds of file system (including SAMBA, Google drive file system, etc) and access them as if there were ordinary files and folders in the hard disk. How to create such file system? I am referring to the file system itself, not to the server where the files are stored.
I prefer if the tutorial is for Mac or Linux (I would guess that the technique will be OS dependent)
This is not as trivial as you might think.
I had to do this some time ago. What I did was to integrate a TCP/IP-based file server into my program and used the network file system functionality of the OS.
Today the "FUSE" project is available for Linux (and as far as I understand correctly) for MacOS X (the last one seems to be named "MacFUSE").
The "FUSE" project provides you a special API which allows you to create a mountable file system...
You install "FUSE" and you can write programs that provide file systems to the OS.
I am trying to build a bash like script provides some functionalities such as ls,pwd,cat etc. working on NTFS in a linux system. Suppose that I have an NTFS image and I open that as a file with fopen. Then, I read some sectors such as BPB residing at 0x0B and fetched some general info about the NTFS image. I need to reach to the root directory pointer then traverse through the filesystem in order to implement those functions especially for ls and pwd. I google'd a lot about internal details and offsets of NTFS but I could not find out how to achieve the goal. I can not progress further without understandable documentation or samples.
Any help, documentation, hint, offset table etc. would be highly appreciated.
Thank you.
I'm guessing this is a learning exercise. So, first:
Writing a bashlike interpreter for a specific filesystem is the wrong thing to do. You should be concentrating on understanding the details of the NTFS filesystem instead.
Writing ls, cat to be able to work with files in a specific filesystem is the wrong thing to do. You should be concentrating on understanding the details of the NTFS filesystem instead.
If you write a filesystem driver (say using FUSE), then the original bash, ls, cat will automatically work with that filesystem. Because the driver will be able to translate system calls like open and read into the filesystem specific procedure.
Finally:
Learn about FUSE. It is awesome. See this Hello World FUSE module. Run it, play with it.
Download the sources for NTFS-3G, which is the NTFS driver used by most GNU/Linux distros these days. It uses FUSE. Learn how it works.
I am wondering if it's possible to write an application that will access a foreign filesystem, but without needing support for that filesystem from the operating system. For example, I'd like to write an app in C that runs on Mac OS X that can browse / copy files from an ext2/ext3 formatted disk. Of course, you'd have to do all the transfers through the application (not through the system using cp or the Finder), but that would be OK for my purpose. Is this possible?
There are user space libraries that allow you to access file systems.
The Linux-NTFS library (libntfs) allows you to access NTFS file systems and there are user space programs like ntfsfix to do things to the file system.
E2fsprogs does the same for ext2, ext3 and ext4 filesystems.
As Basile mentioned, Mtools is another one that provides access to FAT partitions.
There was even a program that does exactly what you're looking for on Windows. It's called ext2explore and allows you to access ext2 partitions from Windows.
It is possible. For example the GNU mtools utility are doing that (assuming a way to access the raw device or partition) for MS-DOS FAT file systems.
However, file systems inside the kernel are usually very well tested and optimized.
Yes and No. For a regular user Application is usually not possible because access to block devices is restricted to root only. Every block device should give read/write to the needed block device for that effect. This would need at best a server/client approach where a service is started on the machine and configured to give the permissions on a per block device manner.
The somewhat easier alternative would be you to use the MacFUSE implementation.
Look here:
http://code.google.com/p/macfuse/
http://groups.google.com/group/macfuse?pli=1
The MacFuse project seems no longer mantained, but can give you a starting point for your project.
The dirty and quick approach is the following as root chmod 666 /dev/diskN
You can hijack syscalls and library calls from your application and then redirect reads/writes to anything like a KV store or a distributed DB layer (using the regular calls for the "virtual devices" that you do not support).
Then, the possibilities are boundless because you don't have to reach the physical/virtual devices when someone asks for them (resolving privilege issues).
I'm just sort of curious: If I wanted to embed a filesystem in an executable, I assume I'd just embed an image of the filesystem using objcopy. However, what sort of filesystem should I use? Is there going to be an easy library for directly accessing the filesystem image?
I don't have any use for this, so responses like "don't do it, use actual files instead" are irrelevant.
There exist so-called virtual file system implementations, which were designed with custom storage in mind. To name a few: our Solid File System, CodeBase File System, Single File System by AidAim.
Most of them use files as the container, some (eg. our SolFS) let you have container in custom places and access it via callbacks.
Depending on practical requirements you can use ZIP or even TAR format as well.
If you want to expose the filesystem contained in your executable to the OS so that other applications could read it, then one of our virtual storage solutions (CallbackDisk or Callback File System) will do the job for you on Windows. On Linux and MacOS X there exist other ways to do this, including FUSE and MacFUSE libraries (analogs to our CBFS).
I am wondering how the OS is reading/writing to the hard drive.
I would like as an exercise to implement a simple filesystem with no directories that can read and write files.
Where do I start?
Will C/C++ do the trick or do I have to go with a more low level approach?
Is it too much for one person to handle?
Take a look at FUSE: http://fuse.sourceforge.net/
This will allow you to write a filesystem without having to actually write a device driver. From there, I'd start with a single file. Basically create a file that's (for example) 100MB in length, then write your routines to read and write from that file.
Once you're happy with the results, then you can look into writing a device driver, and making your driver run against a physical disk.
The nice thing is you can use almost any language with FUSE, not just C/C++.
I found it quite easy to understand a simple filesystem while using the fat filesystem on the avr microcontroller.
http://elm-chan.org/fsw/ff/00index_e.html
Take look at the code you will figure out how fat works.
For learning the ideas of a file system it's not really necessary to use a disk i think. Just create an array of 512 byte byte-arrays. Just imagine this a your Harddisk an start to experiment a bit.
Also you may want to hava a look at some of the standard OS textbooks like http://codex.cs.yale.edu/avi/os-book/OS8/os8c/index.html
The answer to your first question, is that besides Fuse as someone else told you, you can also use Dokan that does the same for Windows, and from there is just a question of doing Reads and Writes to a physical partition (http://msdn.microsoft.com/en-us/library/aa363858%28v=vs.85%29.aspx (read particularly the section on Physical Disks and Volumes)).
Of course that in Linux or Unix besides using something like Fuse you only have to issue, a read or write call to the wanted device in /dev/xxx (if you are root), and in these terms the Unices are more friendly or more insecure depending on your point of view.
From there try to implement a simple filesystem like Fat, or something more exoteric like an tar filesystem, or even some simple filesystem based on Unix concepts like UFS or Minux, or just something that only logs the calls that are made and their arguments to a log file (and this will help you understand, the calls that are made to the filesystem driver during the regular use of your computer).
Now your second question (that is much more simple to answer), yes C/C++ will do the trick, since they are the lingua franca of system development, also a lot of your example code will be in C/C++ so you will at least read C/C++ in your development.
Now for your third question, yes, this is doable by one person, for example the ext filesystem (widely known in Linux world by it's successors as ext2 or ext3) was made by a single developer Theodore Ts'o, so don't think that these things aren't doable by a single person.
Now the final notes, remember that a real filesystem interacts with a lot of other subsystems in a regular kernel, for example, if you have a laptop and hibernate it the filesystem has to flush all changes made to the open files, if you have a pagefile on the partition or even if the pagefile has it's own filesystem, that will affect your filesystem, particularly the block sizes, since they will tend to be equal or powers of the page block size, because it's easy to just place a block from the filesystem on memory that by coincidence is equal to the page size (because that's just one transfer).
And also, security, since you will want to control the users and what files they read/write and that usually means that before opening a file, you will have to know what user is logged on, and what permissions he has for that file. And obviously without filesystem, users can't run any program or interact with the machine. Modern filesystem layers, also interact with the network subsystem due to the fact that there are network and distributed filesystems.
So if you want to go and learn about doing kernel filesystems, those are some of the things you will have to worry about (besides knowing a VFS interface)
P.S.: If you want to make Unix permissions work on Windows, you can use something like what MS uses for NFS on the server versions of windows (http://support.microsoft.com/kb/262965)