I'm just sort of curious: If I wanted to embed a filesystem in an executable, I assume I'd just embed an image of the filesystem using objcopy. However, what sort of filesystem should I use? Is there going to be an easy library for directly accessing the filesystem image?
I don't have any use for this, so responses like "don't do it, use actual files instead" are irrelevant.
There exist so-called virtual file system implementations, which were designed with custom storage in mind. To name a few: our Solid File System, CodeBase File System, Single File System by AidAim.
Most of them use files as the container, some (eg. our SolFS) let you have container in custom places and access it via callbacks.
Depending on practical requirements you can use ZIP or even TAR format as well.
If you want to expose the filesystem contained in your executable to the OS so that other applications could read it, then one of our virtual storage solutions (CallbackDisk or Callback File System) will do the job for you on Windows. On Linux and MacOS X there exist other ways to do this, including FUSE and MacFUSE libraries (analogs to our CBFS).
Related
This question doesn't sound trivial, but I'll give it a try.
What I need to do, is to create a virtual filesystem and mount it as if it was an hard drive. When my application starts, a new hard drive should appear in the list of available devices. Now, I need that drive to be virtual. In particular, I need to be able to generate the content in a dynamic way.
Basically, I would want to be able to:
import some magic library
register the callbacks for, like, list folder, get the content of this file at these bytes, and so on
run my program
a new drive appears, and when I do an ls, it's my program that answers that ls via a callback
Is this even possible? In principle, I should be able to simulate a drive, but I wouldn't even know where to start.
On Linux FUSE library is built into the OS. On MacOS X there's OSXFUSE which is similar to FUSE on Linux.
On Windows there's CBFS Connect by our company, which offers its own API and a FUSE compatibility layer.
Mobile platforms (iOS, Android) neither offer such functions nor give a way to implement them.
I have a legacy Linux application written for C that relies upon static external files on the filesystem. I'd like to bundle all of them together into a single executable, so that the single-file executable doesn't rely upon anything in the filesystem. Is there a way to do this, without having to make lots of changes to the existing code?
I can link the program statically to avoid any dependencies on dynamic libraries, but the application also relies upon other static resources (other read-only files on the filesystem), which I'd like to embed into the application. I know how to embed an external file into the final executable using objcopy (e.g., as described here or here), but then I need to arrange for the program to use the embedded blob instead of trying to open a file on the filesystem. I've seen some ways to access these ELF sections at runtime (e.g., using linker symbol names or elfdataembed, as described here and here), but they require me to change every place in the program that accesses one of these external files to instead refer to the embedded resource. That sounds tedious and error-prone.
To reduce my workload and reduce bugs, I'd prefer to minimize the amount of changes needed to the application's code. How can I do this, minimizing changes to the application? I'm thinking maybe something that wraps open() to detect attempts to open one of the external files and redirecting them to read from the blob embedded in the executable, for instance, though I'm not sure about some of the details (e.g., how the open() wrapper can create a fake fd; whether I'll need to wrap all of the other filesystem functions as well) or what the pitfalls might be.
How can I achieve this? Or, is there an existing tool I should know about, to avoid re-inventing the wheel?
I am going to implement a file system in C and i'm wondering how can i test it without installing it in the kernel nor using FUSE API. Ideally what i'd like to do is to use dd command to create a virtual hard drive and interact with it using linux system calls like write and read (the idea is to not write drivers). Is that posible?
(I'm sorry if i misspelled words, but eanglish isn't my first language. Also i'm sorry if this is off-topic, it's my first question)
Thanks.
If you are really implementing a file system, you can test it in virtual machine.
Otherwise, you can implement a file system in a file which exist in real file system, and implement some functions like read/write/etc...
Virtual hard drive and virtual filesystem are a bit different things, - you write different functions and handle different requests when implementing them. Given that you implement a filesystem, your best bet on linux is to expose your filesystem via FUSE for testing. Then write different tests that will access your FUSE-based filesystem to perform various tasks.
Unfortunately testing a filesystem is hard and requires writing many tests. Manual testing with different software (file managers) is also required.
I am wondering if it's possible to write an application that will access a foreign filesystem, but without needing support for that filesystem from the operating system. For example, I'd like to write an app in C that runs on Mac OS X that can browse / copy files from an ext2/ext3 formatted disk. Of course, you'd have to do all the transfers through the application (not through the system using cp or the Finder), but that would be OK for my purpose. Is this possible?
There are user space libraries that allow you to access file systems.
The Linux-NTFS library (libntfs) allows you to access NTFS file systems and there are user space programs like ntfsfix to do things to the file system.
E2fsprogs does the same for ext2, ext3 and ext4 filesystems.
As Basile mentioned, Mtools is another one that provides access to FAT partitions.
There was even a program that does exactly what you're looking for on Windows. It's called ext2explore and allows you to access ext2 partitions from Windows.
It is possible. For example the GNU mtools utility are doing that (assuming a way to access the raw device or partition) for MS-DOS FAT file systems.
However, file systems inside the kernel are usually very well tested and optimized.
Yes and No. For a regular user Application is usually not possible because access to block devices is restricted to root only. Every block device should give read/write to the needed block device for that effect. This would need at best a server/client approach where a service is started on the machine and configured to give the permissions on a per block device manner.
The somewhat easier alternative would be you to use the MacFUSE implementation.
Look here:
http://code.google.com/p/macfuse/
http://groups.google.com/group/macfuse?pli=1
The MacFuse project seems no longer mantained, but can give you a starting point for your project.
The dirty and quick approach is the following as root chmod 666 /dev/diskN
You can hijack syscalls and library calls from your application and then redirect reads/writes to anything like a KV store or a distributed DB layer (using the regular calls for the "virtual devices" that you do not support).
Then, the possibilities are boundless because you don't have to reach the physical/virtual devices when someone asks for them (resolving privilege issues).
I have a few dd images and I wanted to know the best way of extracting files out of these using C. The images are of mixed file systems (fat32, ntfs, ext2/3) and the host machine doing the extraction would be an Ubuntu box, so you can assume kernel headers and GNU C library, etc.
Natively would be best, but external libraries that do the job would also be fine. A link to an example would be perfect.
This is a significant effort. You'd have to essentially reimplement filesystem drivers for NTFS, FAT, and EXT2. I've done this for FAT and NTFS, but it took more than two years, though much of that was reverse engineering NTFS.
Consider using the file mount option of the mount command so you can use the Ubuntu filesystem drivers and not reinvent the significantly large wheel. Then you can peruse the mounted filesystems.
Why programatically with C?
sudo mount -o loop,offset=[offset] -t auto [where] [what]
Where
offset is the offset in bytes from the beginning of the disk, in bytes
where is where on the current filesystem to mount the image
what is the disk image you're mounting
Look at The Sleuth Kit, which should work with all of the file system types you listed:
The original part of Sleuth Kit is a C library and collection of command line file and volume system forensic analysis tools. The file system tools allow you to examine file systems of a suspect computer in a non-intrusive fashion. Because the tools do not rely on the operating system to process the file systems, deleted and hidden content is shown. It runs on Windows and Unix platforms.
The Sleuth Kit's existing toolset is a great place to start if you're looking for sample code to work from.
Check out:
ext2fuse or fuse-ext2 projects, they contain some ext2/ext3 implementations on user space using FUSE.
ntfs-3g, an NTFS implementation also using FUSE.
FUSE itself, as there are lots of filesystem implementations on top of it.