I am trying to build a bash like script provides some functionalities such as ls,pwd,cat etc. working on NTFS in a linux system. Suppose that I have an NTFS image and I open that as a file with fopen. Then, I read some sectors such as BPB residing at 0x0B and fetched some general info about the NTFS image. I need to reach to the root directory pointer then traverse through the filesystem in order to implement those functions especially for ls and pwd. I google'd a lot about internal details and offsets of NTFS but I could not find out how to achieve the goal. I can not progress further without understandable documentation or samples.
Any help, documentation, hint, offset table etc. would be highly appreciated.
Thank you.
I'm guessing this is a learning exercise. So, first:
Writing a bashlike interpreter for a specific filesystem is the wrong thing to do. You should be concentrating on understanding the details of the NTFS filesystem instead.
Writing ls, cat to be able to work with files in a specific filesystem is the wrong thing to do. You should be concentrating on understanding the details of the NTFS filesystem instead.
If you write a filesystem driver (say using FUSE), then the original bash, ls, cat will automatically work with that filesystem. Because the driver will be able to translate system calls like open and read into the filesystem specific procedure.
Finally:
Learn about FUSE. It is awesome. See this Hello World FUSE module. Run it, play with it.
Download the sources for NTFS-3G, which is the NTFS driver used by most GNU/Linux distros these days. It uses FUSE. Learn how it works.
Related
I have 2 computers. One (computer 2) is getting files (.json) from 2 different processes, and then it passes this files (through ethernet cable) to the other computer (computer 1) (like in the attached image). This happens constantly not just one's.
To do this file transfers, my idea is the following:
Create a server/client sockets in c to comunicate both computers.
Tar the files (an amount say 4 files) in computer 2.
Recive the files in computer 1.
One way at first I wanted to do this, was with Netcat and tar, in bash. But the I read that it was not a good Ideas, because bash doesn't work well with file transfers. So I decided to do it in C (it has to be C, or C++, not python, but I am better at C so C is the option).
So now I am doing it with this sample code:
Send and Receive a file in socket programming in Linux with C/C++ (GCC/G++)
but I cant figure out the part of tar and sending Tar, and if this code would help for this.
Other way I wa thinking to do it was with zeromq, but I havent use this before, so I dont know if it is worth the extra studding.
Thanks a lot in advance for your answers.
image
The .tar file format is pretty straightforward (and documentation on how to read/write a .tar file yourself can be found via a Google search on ".tar file format" or similar), or you could just use the existing libtar reader/writer library if you'd prefer not to reinvent the wheel.
Another alternative (assuming you just need a quick program to handle a personal use-case and don't need a production-grade solution) would just be to call system() (or similar) to have it execute the appropriate shell commands on behalf of your C program -- but if that's good enough, one has to wonder why a simple bash script isn't also acceptable. Bash may not be good at file transfers natively, but the various command-line utilities you can call from bash (scp, rsync, tar, untar, etc) certainly work fine.
For a university assignment, we have to modify the ext2 file system to store files in the inode's block pointers if it's smaller than 60 bytes, and move to regular block storage once the file grows larger than that.
I have, what might admittedly be a silly question, but I was wondering if anyone with experience working in the ext2fs might be able to tell me whether the inode structure itself would have to be modified to accomplish this task?
And would modification of the inode, if it were required, impede the general running of the ext2 system?
for better understanding of any filesystem in Linux I recommend 'Linux kernel development' 3rd edition by Robert Love. (The Virtual Filesystem part)
after that you can read GNU documentations about ext2 filesystem.
and then start reading e2fsprogs. this is the tool for creating ext filesystems. if you want to modify filesystem structure, you need to build modified filesystem first. finally read the actual source code of kernel's ext2 driver.
keep in mind that there is no short way to do this. you should understand Linux VFS completely.
and one other thing.. when reading source files keep in mind that the most important parts of the code are data structures that act like objects.
use source code reading tools listed by GNU like cscope.
and yes! modification of inode structure CAN cause many sorts of problems.
good luck :)
Can anybody point me to a simple (can't stress this enough) implementation of an in memory file system? If I can create a file and do a simple cat file.txt it's more than enough.
I would like to use it as part of my toy OS.
The OSDev wiki site might help you. You can also ask your questions there and if you look at wiki and their forum you will pretty likely get your answer.
In my opinion, in-memory file systems should be as basic as possible. Here is a virtual file system implemented in user-mode from Windows, but its design principals can be used in your own OS. http://www.flipcode.com/archives/Programming_a_Virtual_File_System-Part_I.shtml . Even this might be too much for your basic OS. I say just wing it and create a linked list of file descriptors that include only file attributes, file name, and file path with each file.
This belongs on Super User IMHO, but anyway, you might want to look at ImDisk. It should be more than enough for just creating a RAM disk.
Wait, I misread... this is for your "toy OS"? Your toy OS supports file systems? You're going to have to implement it yourself, since there's no way something preexisting will work with your home-made OS.
I am wondering how the OS is reading/writing to the hard drive.
I would like as an exercise to implement a simple filesystem with no directories that can read and write files.
Where do I start?
Will C/C++ do the trick or do I have to go with a more low level approach?
Is it too much for one person to handle?
Take a look at FUSE: http://fuse.sourceforge.net/
This will allow you to write a filesystem without having to actually write a device driver. From there, I'd start with a single file. Basically create a file that's (for example) 100MB in length, then write your routines to read and write from that file.
Once you're happy with the results, then you can look into writing a device driver, and making your driver run against a physical disk.
The nice thing is you can use almost any language with FUSE, not just C/C++.
I found it quite easy to understand a simple filesystem while using the fat filesystem on the avr microcontroller.
http://elm-chan.org/fsw/ff/00index_e.html
Take look at the code you will figure out how fat works.
For learning the ideas of a file system it's not really necessary to use a disk i think. Just create an array of 512 byte byte-arrays. Just imagine this a your Harddisk an start to experiment a bit.
Also you may want to hava a look at some of the standard OS textbooks like http://codex.cs.yale.edu/avi/os-book/OS8/os8c/index.html
The answer to your first question, is that besides Fuse as someone else told you, you can also use Dokan that does the same for Windows, and from there is just a question of doing Reads and Writes to a physical partition (http://msdn.microsoft.com/en-us/library/aa363858%28v=vs.85%29.aspx (read particularly the section on Physical Disks and Volumes)).
Of course that in Linux or Unix besides using something like Fuse you only have to issue, a read or write call to the wanted device in /dev/xxx (if you are root), and in these terms the Unices are more friendly or more insecure depending on your point of view.
From there try to implement a simple filesystem like Fat, or something more exoteric like an tar filesystem, or even some simple filesystem based on Unix concepts like UFS or Minux, or just something that only logs the calls that are made and their arguments to a log file (and this will help you understand, the calls that are made to the filesystem driver during the regular use of your computer).
Now your second question (that is much more simple to answer), yes C/C++ will do the trick, since they are the lingua franca of system development, also a lot of your example code will be in C/C++ so you will at least read C/C++ in your development.
Now for your third question, yes, this is doable by one person, for example the ext filesystem (widely known in Linux world by it's successors as ext2 or ext3) was made by a single developer Theodore Ts'o, so don't think that these things aren't doable by a single person.
Now the final notes, remember that a real filesystem interacts with a lot of other subsystems in a regular kernel, for example, if you have a laptop and hibernate it the filesystem has to flush all changes made to the open files, if you have a pagefile on the partition or even if the pagefile has it's own filesystem, that will affect your filesystem, particularly the block sizes, since they will tend to be equal or powers of the page block size, because it's easy to just place a block from the filesystem on memory that by coincidence is equal to the page size (because that's just one transfer).
And also, security, since you will want to control the users and what files they read/write and that usually means that before opening a file, you will have to know what user is logged on, and what permissions he has for that file. And obviously without filesystem, users can't run any program or interact with the machine. Modern filesystem layers, also interact with the network subsystem due to the fact that there are network and distributed filesystems.
So if you want to go and learn about doing kernel filesystems, those are some of the things you will have to worry about (besides knowing a VFS interface)
P.S.: If you want to make Unix permissions work on Windows, you can use something like what MS uses for NFS on the server versions of windows (http://support.microsoft.com/kb/262965)
I have a few dd images and I wanted to know the best way of extracting files out of these using C. The images are of mixed file systems (fat32, ntfs, ext2/3) and the host machine doing the extraction would be an Ubuntu box, so you can assume kernel headers and GNU C library, etc.
Natively would be best, but external libraries that do the job would also be fine. A link to an example would be perfect.
This is a significant effort. You'd have to essentially reimplement filesystem drivers for NTFS, FAT, and EXT2. I've done this for FAT and NTFS, but it took more than two years, though much of that was reverse engineering NTFS.
Consider using the file mount option of the mount command so you can use the Ubuntu filesystem drivers and not reinvent the significantly large wheel. Then you can peruse the mounted filesystems.
Why programatically with C?
sudo mount -o loop,offset=[offset] -t auto [where] [what]
Where
offset is the offset in bytes from the beginning of the disk, in bytes
where is where on the current filesystem to mount the image
what is the disk image you're mounting
Look at The Sleuth Kit, which should work with all of the file system types you listed:
The original part of Sleuth Kit is a C library and collection of command line file and volume system forensic analysis tools. The file system tools allow you to examine file systems of a suspect computer in a non-intrusive fashion. Because the tools do not rely on the operating system to process the file systems, deleted and hidden content is shown. It runs on Windows and Unix platforms.
The Sleuth Kit's existing toolset is a great place to start if you're looking for sample code to work from.
Check out:
ext2fuse or fuse-ext2 projects, they contain some ext2/ext3 implementations on user space using FUSE.
ntfs-3g, an NTFS implementation also using FUSE.
FUSE itself, as there are lots of filesystem implementations on top of it.