Difference between "file pointer", "stream", "file descriptor" and ... "file"? - c

There are a few related concepts out there, namely file pointer, stream and file descriptor.
I know that a file pointer is a pointer to the data type FILE (declared in e.g. FILE.h and struct_FILE.h).
I know a file descriptor is an int, e.g. member _fileno of FILE (and _IO_FILE).
As for the subtle difference between stream and file, I am still learning.
But from here, I am not clear if there is yet another type of entity to which the "file status flags" apply.
Concretely, I wouldn't know if "file status flags" apply to a FILE, to a file descriptor, or what.
I am looking for official references that show the specifics.
Related:
What's the difference between a file descriptor and file pointer?
Whats is difference between file descriptor and file pointer?
What is the concept behind file pointer or the stream pointer?
Specification of file descriptors (I asked this)
difference between file descriptor and socket file descriptor

File Handle
When you visit a web site for the first time, the site might provide your browser with a cookie. The value of this cookie will automatically be provided to the web site on future requests by the browser.
The value of this cookie is likely gibberish to you, but it has meaning to that one specific web server. It's called a session id, and it's a key to look up a record in some kind of database. This record is called a session.
Sessions allow the web server to react to one request based on earlier requests and the consequences of earlier requests. For example, it allows the server to know that the browser provided credentials to the server in an earlier request, and that these credentials were successfully authenticated. This is why you don't need to resupply your credentials every time you want to post/vote/edit as a specific user on StackOverflow.
The cookie's value, the session id, is an opaque value. It doesn't have any meaning to you. The only way it's useful is by providing it back to the web server that gave it to you. Giving it to another web server isn't going to achieve anything useful. It's just a means of identifying a resource that exists in another system.
When that other system is an operating system, we call these resource-identifying opaque values "handles". This is by no means the only time the word handle is used this way, but it's the most common. In much the same way that a session id cookie provides the web server a way of linking web requests together, a handle provides the OS a way of linking system calls together. There are handles for all kinds of resources. There are window handles. There are handles for allocated memory buffers. And there are file handles.
By using the same file handle across multiple calls to read or write, the OS knows where the previous one left off and thus from where to continue. It also knows that you have access to the file from which you are reading or to which you are writing because those checks were done when the file was opened.
File handles aren't just for plain files. A file handle can also reference a pipe, a socket, or one of a number of other things. Once the handle is created, you just have to tell the OS you want to read from it or write to it, and it will use the handle to look up the information it needs to do that.
File Descriptor
This is the name given to file handles in the unix world. open(2) is said to return a file descriptor. read(2) is said to take a file descriptor.
FILE* aka FILE Pointer aka File Pointer
This is also a file handle. But unlike a file descriptor, it's not from the OS. A FILE* is a C library file handle. You can't pass a FILE* to read(2) (a system call) any more than you can pass a file descriptor to fread(3) (a C library function).
You should never access the members of FILE, assuming it even has any. Like all handles, it's meant to be opaque to those receiving it. It's meant to be a box into which you can't see. Code that breaks this convention isn't portable and can break at any time.
Most C library file handles reference an object that includes a file descriptor. (Ones returned by fmemopen and open_memstream don't.) It also includes support for buffering, and possibly more.
File Status Flags
This is not a term you will ever need to use. It's my first time hearing it. Or maybe I just forgot hearing it because it's not important. In the linked document, it's used to refer to a group of constants. Various system calls can be provided some combinations of some of the constants in this group for certain arguments. Refer to the documentation of each system to see what flags it can accept, and what meanings those flags has to it.
Stream
Earlier, I compared file handles to session ids. If a session id allows a web server to look up a session, what is a file handle used to look up? The documentation for the C library I/O functions calls it a stream.
A stream is a loose term that usually refers to a sequence of indeterminate length. It's a term commonly used in communication to refer to the data being communicated between a writer/sender/producer and a reader/receiver/consumer.
A stream is accessed sequentially, whether it's out of necessity or because it's convenient. The possibility of jumping to a different point in the stream doesn't automatically disqualify the use of the term. Like I mentioned above, it's a loose term.
The length of a stream is often unknown. It might be even be unknown to the sender. Take for example a task producing a stream on the fly, possibly from other streams. A stream could even be infinitely long. Sometimes, the length of the stream is knowable, but simply disregarded. And sometimes, the length is known but not in usable units. A program reading lines of variable length from a stream probably can't do anything useful with the length of the stream in bytes.
Take two programs communicating via a pipe like in cat <file1 | cat >file2. We can refer to the data going through the pipe as a stream. The sender may or may not know how many bytes/lines/messages it will eventually send. The sender will send some bytes and later some more, until it eventually signals that no more will follow. The reader often has no idea how many bytes/lines/messages will eventually be sent by the producer. It will get some bytes and later some more, until it's eventually notified that the end of the stream has been reached.
Sometimes, it's more about how the data is treated. For example, reading from a file is often treated as reading from a stream. While it's possible to obtain the length of a file, this information is often disregarded. Instead, the programs that disregard this information just keeps pulling bytes or lines from the file handle until it receives an indication that it reached the end of the stream.
Random access is an example of a file not being treated as a stream. Random access refers to the practice of retrieving data from arbitrary locations of the file. One might do this when one has an index of what's found in the file. An index is some mapping between a key and the location of the item identified by that key in the file. For example, if I know the data pertaining to a user is found at a certain location in a file, I can request that portion of the file from the OS rather than reading the file from the start.

Related

Is it possible to share file pointers between threads?

We need to address some performance issues on our application. Using Visual Studio's diagnostic hub, we discovered that file operations were responsible for our issue.
We are using a file to save critical data temporarily that has to be sent via TCP/IP to a server.
The file consists of a header where information is stored, where to find a specific data set (offset and length of up to 2500 data sets in this file). The second part is the data itself, addressed by the information in the header.
When data is written to the file, first the header is read, then data is written to the next empty location, and the information in the header is updated and written again.
When data needs to be sent (in the second thread), the header is read to obtain the next offset and length of data to be read. Then the data is read and sent via TCP/IP. Until we receive the TCP/IP ACK of the server, the header is not updated again. When the ACK arrives, the header information is updated to reflect that the used dataset is now empty.
All functions involved in this process made a fopen() and fclose() to the file, which caused our performance issues.
I know that there must be some synchronization when writing the header, because when the first thread puts in new data, the second thread must obey the new write pointer in the file (if not, the newly written data will be lost).
If I am able to sync access to the file pointer, so that any time one thread is writing at a specific location, the other thread would not write too, is it safe to use the FILE* variable in those two threads?
Or, would it be better to have two FILE* variables that read and write on their own portions of the file?
I know, that I could combine both operations into one thread, but then I must address delays from synchronous socket operations.
To address the performance issue, I rewrote my code, and now I am opening the file only once per thread, and do seek/read/write operations on the same file pointer. So there is no overhead on the file operations caused by repeatedly opening and closing the file.
Problem solved - at least for now.

What does opening a file actually do?

In all programming languages (that I use at least), you must open a file before you can read or write to it.
But what does this open operation actually do?
Manual pages for typical functions dont actually tell you anything other than it 'opens a file for reading/writing':
http://www.cplusplus.com/reference/cstdio/fopen/
https://docs.python.org/3/library/functions.html#open
Obviously, through usage of the function you can tell it involves creation of some kind of object which facilitates accessing a file.
Another way of putting this would be, if I were to implement an open function, what would it need to do on Linux?
In almost every high-level language, the function that opens a file is a wrapper around the corresponding kernel system call. It may do other fancy stuff as well, but in contemporary operating systems, opening a file must always go through the kernel.
This is why the arguments of the fopen library function, or Python's open closely resemble the arguments of the open(2) system call.
In addition to opening the file, these functions usually set up a buffer that will be consequently used with the read/write operations. The purpose of this buffer is to ensure that whenever you want to read N bytes, the corresponding library call will return N bytes, regardless of whether the calls to the underlying system calls return less.
I am not actually interested in implementing my own function; just in understanding what the hell is going on...'beyond the language' if you like.
In Unix-like operating systems, a successful call to open returns a "file descriptor" which is merely an integer in the context of the user process. This descriptor is consequently passed to any call that interacts with the opened file, and after calling close on it, the descriptor becomes invalid.
It is important to note that the call to open acts like a validation point at which various checks are made. If not all of the conditions are met, the call fails by returning -1 instead of the descriptor, and the kind of error is indicated in errno. The essential checks are:
Whether the file exists;
Whether the calling process is privileged to open this file in the specified mode. This is determined by matching the file permissions, owner ID and group ID to the respective ID's of the calling process.
In the context of the kernel, there has to be some kind of mapping between the process' file descriptors and the physically opened files. The internal data structure that is mapped to the descriptor may contain yet another buffer that deals with block-based devices, or an internal pointer that points to the current read/write position.
I'd suggest you take a look at this guide through a simplified version of the open() system call. It uses the following code snippet, which is representative of what happens behind the scenes when you open a file.
0 int sys_open(const char *filename, int flags, int mode) {
1 char *tmp = getname(filename);
2 int fd = get_unused_fd();
3 struct file *f = filp_open(tmp, flags, mode);
4 fd_install(fd, f);
5 putname(tmp);
6 return fd;
7 }
Briefly, here's what that code does, line by line:
Allocate a block of kernel-controlled memory and copy the filename into it from user-controlled memory.
Pick an unused file descriptor, which you can think of as an integer index into a growable list of currently open files. Each process has its own such list, though it's maintained by the kernel; your code can't access it directly. An entry in the list contains whatever information the underlying filesystem will use to pull bytes off the disk, such as inode number, process permissions, open flags, and so on.
The filp_open function has the implementation
struct file *filp_open(const char *filename, int flags, int mode) {
struct nameidata nd;
open_namei(filename, flags, mode, &nd);
return dentry_open(nd.dentry, nd.mnt, flags);
}
which does two things:
Use the filesystem to look up the inode (or more generally, whatever sort of internal identifier the filesystem uses) corresponding to the filename or path that was passed in.
Create a struct file with the essential information about the inode and return it. This struct becomes the entry in that list of open files that I mentioned earlier.
Store ("install") the returned struct into the process's list of open files.
Free the allocated block of kernel-controlled memory.
Return the file descriptor, which can then be passed to file operation functions like read(), write(), and close(). Each of these will hand off control to the kernel, which can use the file descriptor to look up the corresponding file pointer in the process's list, and use the information in that file pointer to actually perform the reading, writing, or closing.
If you're feeling ambitious, you can compare this simplified example to the implementation of the open() system call in the Linux kernel, a function called do_sys_open(). You shouldn't have any trouble finding the similarities.
Of course, this is only the "top layer" of what happens when you call open() - or more precisely, it's the highest-level piece of kernel code that gets invoked in the process of opening a file. A high-level programming language might add additional layers on top of this. There's a lot that goes on at lower levels. (Thanks to Ruslan and pjc50 for explaining.) Roughly, from top to bottom:
open_namei() and dentry_open() invoke filesystem code, which is also part of the kernel, to access metadata and content for files and directories. The filesystem reads raw bytes from the disk and interprets those byte patterns as a tree of files and directories.
The filesystem uses the block device layer, again part of the kernel, to obtain those raw bytes from the drive. (Fun fact: Linux lets you access raw data from the block device layer using /dev/sda and the like.)
The block device layer invokes a storage device driver, which is also kernel code, to translate from a medium-level instruction like "read sector X" to individual input/output instructions in machine code. There are several types of storage device drivers, including IDE, (S)ATA, SCSI, Firewire, and so on, corresponding to the different communication standards that a drive could use. (Note that the naming is a mess.)
The I/O instructions use the built-in capabilities of the processor chip and the motherboard controller to send and receive electrical signals on the wire going to the physical drive. This is hardware, not software.
On the other end of the wire, the disk's firmware (embedded control code) interprets the electrical signals to spin the platters and move the heads (HDD), or read a flash ROM cell (SSD), or whatever is necessary to access data on that type of storage device.
This may also be somewhat incorrect due to caching. :-P Seriously though, there are many details that I've left out - a person (not me) could write multiple books describing how this whole process works. But that should give you an idea.
Any file system or operating system you want to talk about is fine by me. Nice!
On a ZX Spectrum, initializing a LOAD command will put the system into a tight loop, reading the Audio In line.
Start-of-data is indicated by a constant tone, and after that a sequence of long/short pulses follow, where a short pulse is for a binary 0 and a longer one for a binary 1 (https://en.wikipedia.org/wiki/ZX_Spectrum_software). The tight load loop gathers bits until it fills a byte (8 bits), stores this into memory, increases the memory pointer, then loops back to scan for more bits.
Typically, the first thing a loader would read is a short, fixed format header, indicating at least the number of bytes to expect, and possibly additional information such as file name, file type and loading address. After reading this short header, the program could decide whether to continue loading the main bulk of the data, or exit the loading routine and display an appropriate message for the user.
An End-of-file state could be recognized by receiving as many bytes as expected (either a fixed number of bytes, hardwired in the software, or a variable number such as indicated in a header). An error was thrown if the loading loop did not receive a pulse in the expected frequency range for a certain amount of time.
A little background on this answer
The procedure described loads data from a regular audio tape - hence the need to scan Audio In (it connected with a standard plug to tape recorders). A LOAD command is technically the same as open a file - but it's physically tied to actually loading the file. This is because the tape recorder is not controlled by the computer, and you cannot (successfully) open a file but not load it.
The "tight loop" is mentioned because (1) the CPU, a Z80-A (if memory serves), was really slow: 3.5 MHz, and (2) the Spectrum had no internal clock! That means that it had to accurately keep count of the T-states (instruction times) for every. single. instruction. inside that loop, just to maintain the accurate beep timing.
Fortunately, that low CPU speed had the distinct advantage that you could calculate the number of cycles on a piece of paper, and thus the real world time that they would take.
It depends on the operating system what exactly happens when you open a file. Below I describe what happens in Linux as it gives you an idea what happens when you open a file and you could check the source code if you are interested in more detail. I am not covering permissions as it would make this answer too long.
In Linux every file is recognised by a structure called inode. Each structure has an unique number and every file only gets one inode number. This structure stores meta data for a file, for example file-size, file-permissions, time stamps and pointer to disk blocks, however, not the actual file name itself. Each file (and directory) contains a file name entry and the inode number for lookup. When you open a file, assuming you have the relevant permissions, a file descriptor is created using the unique inode number associated with file name. As many processes/applications can point to the same file, inode has a link field that maintains the total count of links to the file. If a file is present in a directory, its link count is one, if it has a hard link its link count will be two and if a file is opened by a process, the link count will be incremented by 1.
Bookkeeping, mostly. This includes various checks like "Does the file exist?" and "Do I have the permissions to open this file for writing?".
But that's all kernel stuff - unless you're implementing your own toy OS, there isn't much to delve into (if you are, have fun - it's a great learning experience). Of course, you should still learn all the possible error codes you can receive while opening a file, so that you can handle them properly - but those are usually nice little abstractions.
The most important part on the code level is that it gives you a handle to the open file, which you use for all of the other operations you do with a file. Couldn't you use the filename instead of this arbitrary handle? Well, sure - but using a handle gives you some advantages:
The system can keep track of all the files that are currently open, and prevent them from being deleted (for example).
Modern OSs are built around handles - there's tons of useful things you can do with handles, and all the different kinds of handles behave almost identically. For example, when an asynchronous I/O operation completes on a Windows file handle, the handle is signalled - this allows you to block on the handle until it's signalled, or to complete the operation entirely asynchronously. Waiting on a file handle is exactly the same as waiting on a thread handle (signalled e.g. when the thread ends), a process handle (again, signalled when the process ends), or a socket (when some asynchronous operation completes). Just as importantly, handles are owned by their respective processes, so when a process is terminated unexpectedly (or the application is poorly written), the OS knows what handles it can release.
Most operations are positional - you read from the last position in your file. By using a handle to identify a particular "opening" of a file, you can have multiple concurrent handles to the same file, each reading from their own places. In a way, the handle acts as a moveable window into the file (and a way to issue asynchronous I/O requests, which are very handy).
Handles are much smaller than file names. A handle is usually the size of a pointer, typically 4 or 8 bytes. On the other hand, filenames can have hundreds of bytes.
Handles allow the OS to move the file, even though applications have it open - the handle is still valid, and it still points to the same file, even though the file name has changed.
There's also some other tricks you can do (for example, share handles between processes to have a communication channel without using a physical file; on unix systems, files are also used for devices and various other virtual channels, so this isn't strictly necessary), but they aren't really tied to the open operation itself, so I'm not going to delve into that.
At the core of it when opening for reading nothing fancy actually needs to happen. All it needs to do is check the file exists and the application has enough privileges to read it and create a handle on which you can issue read commands to the file.
It's on those commands that actual reading will get dispatched.
The OS will often get a head start on reading by starting a read operation to fill the buffer associated with the handle. Then when you actually do the read it can return the contents of the buffer immediately rather then needing to wait on disk IO.
For opening a new file for write the OS will need to add a entry in the directory for the new (currently empty) file. And again a handle is created on which you can issue the write commands.
Basically, a call to open needs to find the file, and then record whatever it needs to so that later I/O operations can find it again. That's quite vague, but it will be true on all the operating systems I can immediately think of. The specifics vary from platform to platform. Many answers already on here talk about modern-day desktop operating systems. I've done a little programming on CP/M, so I will offer my knowledge about how it works on CP/M (MS-DOS probably works in the same way, but for security reasons, it is not normally done like this today).
On CP/M you have a thing called the FCB (as you mentioned C, you could call it a struct; it really is a 35-byte contiguous area in RAM containing various fields). The FCB has fields to write the file-name and a (4-bit) integer identifying the disk drive. Then, when you call the kernel's Open File, you pass a pointer to this struct by placing it in one of the CPU's registers. Some time later, the operating system returns with the struct slightly changed. Whatever I/O you do to this file, you pass a pointer to this struct to the system call.
What does CP/M do with this FCB? It reserves certain fields for its own use, and uses these to keep track of the file, so you had better not ever touch them from inside your program. The Open File operation searches through the table at the start of the disk for a file with the same name as what's in the FCB (the '?' wildcard character matches any character). If it finds a file, it copies some information into the FCB, including the file's physical location(s) on the disk, so that subsequent I/O calls ultimately call the BIOS which may pass these locations to the disk driver. At this level, specifics vary.
In simple terms, when you open a file you are actually requesting the operating system to load the desired file ( copy the contents of file ) from the secondary storage to ram for processing. And the reason behind this ( Loading a file ) is because you cannot process the file directly from the Hard-disk because of its extremely slow speed compared to Ram.
The open command will generate a system call which in turn copies the contents of the file from the secondary storage ( Hard-disk ) to Primary storage ( Ram ).
And we 'Close' a file because the modified contents of the file has to be reflected to the original file which is in the hard-disk. :)
Hope that helps.

Reading MFT in NTFS

While wandering the web looking for explanations of how to read/write MFT I found the folowing section:(http://www.installsetupconfig.com/win32programming/1996%20AppE_apnilife.pdf)
If NtfsProtectSystemFiles is set to FALSE, then the special files can
be opened. There are, however, some drawbacks associated with
attempting to do this: Because many of the special files are opened in
a special way when mounting the volume, they are not prepared to
handle the IRP_MJ_READ requests resulting from a call to ZwReadFile,
and the system crashes if such a request is received. These special
files can be read by mapping the special file with ZwCreateSection and
ZwMapViewOfSection and then reading the mapped data. A further problem
is that a few of the special files are not prepared to handle the
IRP_MJ_CLEANUP request that is generated when the last handle to a
file object is closed, and the system crashes if such a request is
received. The only option is to duplicate the open handle to the
special file into a process that never terminates (such as the system
process).
What does it mean “they are not prepared to handle the IRP_MJ_READ requests” what kind of preparation is needed? What is IRP_MJ_READ?
“Mapping the special file with ZwCreateSection and ZwMapViewOfSection and then reading the mapped data” How does that solve the problem?
What does it means “files are not prepared to handle the IRP_MJ_CLEANUP request that is generated when the last handle to a file object is closed” again what is that preparation? What is IRP_MJ_CLEANUP?
“Duplicate the open handle to the special file into a process that never terminates” How does that solve the problem?
That's old data (from 1996). And more than a little incorrect. The world has moved on since then.
You might try opening \$MFT to read the MFT but getting the access bits right might be problematic. You can also write them but that's really playing with fire. The file system does not expect that it's data structures will be modified without it's involvement.
You are much better off opening the partition raw and walking the disk structures directly.

what is a file handle and where it is useful for a programmer?

I am learning assembly language along with C. this new chapter I started talks about 'file handles', file handles for screen display and file handles for keyboard input etc. I don't know what is a file handle? I am referring to IBM PC ASSEMBLY LANGUAGE PROGRAMMING by Peter Abel
There is a generic concept generally called a "handle" in the context of computer software APIs. In the comments you have probably found a link to the Wikipedia article on that subject.
You are dealing with a specific implementation of a handle data type -- the IBM PC/DOS file handles returned from the int 0x21 interface. If you would like to learn more about these specific file handles, you might want to consult the book Undocumented DOS, which details the in-memory data structures which allow you to investigate these handles further.
Another specific type of handle is the file descriptor returned from the POSIX-standard interface named open(). This function is implemented in the C run-time library on platforms such as Linux, Windows NT, Mac OS, and many other systems. The integer returned from a call to open() may not be a negative number.
Unless you are running under DOS, your file handles are probably provided by the Windows NT Operating System. These file handles are returned from CreateFile() (which is used to open as well as create files), and the only illegal value for a handle returned from this function is INVALID_HANDLE_VALUE. I.e., the Windows NT API may return what would be considered (via casting) a "negative" integer, although it has opened the file.
In all of these cases, the file handle is used to refer to some data structure that keeps track of how the file is open. One important thing which is tracked is the current file position. The position or pointer is set in POSIX by the lseek() function and is read by the tell() function. Any read() or write() takes place from the position of the current file pointer.
Your program can open the same file under two different handles. In this case, the file pointer for each handle is distinct. Updating the file pointer of one handle using lseek() will not affect the file pointer of the other handle to the same file.
A file handle is an integer value which is used to address an open file. Such handles are highly operating system specific, but on systems that support the open() call, you create a handle like this:
int handle = open( "foo.txt", OTHER_STUFF_HERE );
You can then use the handle with read/write calls. The non-portability of handles mean that most people avoid them and instead use the stream library functions in C, such as fopen, fread, fwrite etc.
A handle is something the kernel uses internally to access some resource. Only the kernel really knows what it means, the user process is only told what value to use when it wants to access this resource. They have another advantage in that file handles can be shared among processes - whereas you can't do this with pointers.
Windows uses handles all over the place... files, bitmaps, device contexts, fonts, etc.

Open system call

I'm studying for my operating systems midterm and was wondering if I can get some help.
Can someone explain the checks and what the kernel does during the open() system call?
Thanks!
Very roughly, you can think of the following steps:
Translate the file name into an inode, which is the actual file system object describing the contents of the file, by traversing the filesystem data structures.
During this traversal, the kernel will check that you have sufficient access through the directory path to the file, and check access on the file itself. The precise checks depend on what modes were passed to open.
Create what's sometimes called an open file descriptor within the kernel. There is one of these objects for each file the kernel has opened on behalf of any process.
Allocate an unused index in the per-process file descriptor table, and point it at the open file descriptor.
Return this index from the system call as the file descriptor.
This description should be essentially correct for opening plain files and/or directories, but things are different for various sorts of special files, in particular for devices.
I would go back to what the prof told you - there a lot of things that happen during open(), depending on what you're opening (i.e. a device, a file, a directory), and unless you write what the professor's looking for, you'll lose points.
That being said, it mostly involves the checks to see if this open is valid (i.e. does this file exist, does the user have permissions to read/write it, etc), then an entry in the kernel handle table is allocated to keep track of the fd and its current file position (and of course, some other things)

Resources