POSIX statvfs() description says:
The following flags can be returned in the f_flag member:
ST_RDONLY - Read-only file system.
ST_NOSUID - Setuid/setgid bits ignored by exec.
It is unspecified whether all members of the statvfs structure have meaningful values on all file systems.
Also sys/statvfs.h description:
The <sys/statvfs.h> header shall define the following symbolic constants for the f_flag member:
ST_RDONLY - Read-only file system.
ST_NOSUID - Does not support the semantics of the ST_ISUID and ST_ISGID file mode bits.
How to interpret this correctly? I mean:
does it allow POSIX compliant system to return nonsense, where ST_RDONLY is meaningfull?
if statvfs structure member is meaningful for particular filesystem, is OS allowed to return nonsense (I understand some fields may have no meaning for synthetic filesystems like /proc)?
Is there any OS known to return incorrect ST_RDONLY or ST_NOSUID for filesystems used to store data/executables, while claiming POSIX compatibility of it's statvfs() implementation?
The POSIX spec requires very little of statvfs(), aside from its existence.
In particular, it requires that statvfs() fill the specified struct statvfs * buffer with "information about the file system," but does not guarantee the meaning of that information. In other words it could be complete garbage and in fact is on many systems (including HFS+ on OS X).
That includes the f_flag member of struct statvfs, which can be masked to ST_RDONLY and/or ST_NOSUID but many not be on all filesystems (even when it should be).
If you need to reliably obtain filesystem information across multiple platforms, you may (ironically) have to resort to an unstandardized function like statfs(). On Linux, however, statvfs() behaves pretty well on most non-synthetic filesystems.
Related
I would like to develop a device-driver on linux(written in C) and a user-space library wrapping all functions provided by my device-driver (also written in C). Just to make it more clear, my library wil provide the following methods:
int myOpen();
void myClose();
mySetConf(MyconfStruct conf)
etc.
The function will use the file associated to my device-driver, in particular:
myOpen will call the open() of my device-driver
myClose will call the close() of my device-driver
mySetConf will call the ioctl() of my device driver and pass the myConfStruct as a parameter to configure the device-driver using ioctl().
assume myConfStruct is a simple structure containing something like this:
typedef struct {
uint16_t var1;
uint8_t var2;
} myConfStruct;
I would like the myConfStruct to be a structure shared between both my user-application (library) and my kernel-driver using a single header.
Are there any best-practice while doing this?
I would like to have the structure defined into only one file, having it defined in multiple files seems to be quite error-prone if i plan on changing it in the future, but I understood that I should not include <linux/types.h> inside my user files and I shouldn't use <stdint.h> inside my device-driver.
So another question is also, how can I define the interface between a module and the user-application so that who is implementing the application is not forced to include any linux header?
What you are creating is a character device. The kernel documentation includes a specific section, the Linux driver implementer's guide, you should also read. Specifically, the ioctl based interfaces section, which also describes some of the considerations necessary (regarding alignment and 64-bit fields).
As to header files, see KernelHeaders article at kernelnewbies.org.
I would like to have the structure defined into only one file, having it defined in multiple files seems to be quite error-prone if i plan on changing it in the future.
No. You do specify the headers in two separate files: one for use in-kernel, and the other for use by userspace.
Kernel-userspace interface should be stable. You should take care to design your data structure so that you can extend it if necessary; preferably by adding some padding reserved for future use and required to be initialized to zero, and/or a version number at the beginning of the structure. Later versions must support all previous versions of the structure as well. Even if it is only a "toy" or "experimental" device driver, it is best to learn to do it right from the get go. This stuff is much, much harder to learn to "add afterwards"; I'm talking from deep experience here.
As a character device, you should also be prepared for the driver to be compiled on other architectures besides the one you are developing on. Even byte order ("endianness") can vary, although all Linux architectures are currently either ILP32 or LP64.
Also remember that there are several hardware architectures, including x86-64, that support both 64-bit and 32-bit userspace. So, even if you believe your driver will ever be used on x86-64, you cannot really assume the userspace is 64-bit (and not 32-bit). Look at existing code to see how it is done right; I recommend using e.g. bootlin's elixir to browse the Linux kernel sources.
Kernel-side header file should use __s8, __u8, __s16, __u16, __s32, __u32, __s64, or __u64. For pointers, use __u64, and u64_to_user_ptr().
Userspace-side header file should use <stdint.h> types (int8_t, uint8_t, int16_t, uint16_t, int32_t, uint32_t, int64_t, uint64_t) and uint64_t for pointers. Use a cast via (uintptr_t) for the conversion, i.e. ptr = (void *)(uintptr_t)u64; and u64 = (uintptr_t)ptr.
Ensure all members are naturally aligned. This means that an N-bit member is preceded by k×N bits of other members, where k is either zero or a positive integer. Thus, if your structure needs an unsigned and a signed 8-bit integer (one for version, one for foo), an 16-bit signed integer (bar), and a 64-bit unsigned integer (baz), considering the version should be first, you'll probably want
struct kernel_side {
__u8 version;
__s8 foo;
__u16 bar;
__u32 padding;
__u64 baz;
};
struct userspace_side {
uint8_t version;
int8_t foo;
uint16_t bar;
uint32_t padding;
uint64_t baz;
};
You can also have character arrays and such, but do note that a single ioctl data block is limited to 8191 bytes or less in length.
If you spend some time designing your interface structures, you'll find that careful design will avoid annoying issues like compat_ support (making them just simple wrappers). Personally, I end up creating a test version with a userspace test program to see what works best, and only then decide on the data structures.
Given a CPU architecture, is the exact binary form of a struct determined exactly?
For example, struct stat64 is used by glibc and the Linux kernel. I see glibc define it in sysdeps/unix/sysv/linux/x86/bits/stat.h as:
struct stat64 {
__dev_t st_dev; /* Device. */
# ifdef __x86_64__
__ino64_t st_ino; /* File serial number. */
__nlink_t st_nlink; /* Link count. */
/* ... et cetera ... */
}
My kernel was compiled already. Now when I compile new code using this definition, they have binary compatibility. Where is this guaranteed? The only guarantees I know of are:
The first element has offset 0
Elements declared later have higher offsets
So if the kernel code declares struct stat64 in the exact same way (in the C code), then I know that the binary form has:
st_dev # offset 0
st_ino # offset at least sizeof(__dev_t)
But I'm not currently aware of any way to determine the offset of st_ino. Kernighan & Ritchie give the simple example
struct X {
char c;
int i;
}
where on my x86-64 machine, offsetof(struct X, i) == 4. Perhaps there are some general alignment rules that determine the exact binary form of a struct for each CPU architecture?
Given a CPU architecture, is the exact binary form of a struct determined exactly?
No, the representation or layout (“binary form”) of a structure is ultimately determined by the C implementation, not by the CPU architecture. Most C implementations intended for normal purposes follow recommendations provided by the manufacturer and/or the operating system. However, there may be circumstances where, for example, a certain alignment for a particular type might give slightly better performance but is not required, and so one C implementation might choose to require that alignment while another does not, and this can result in different structure layout.
In addition, a C implementation might be designed for special purposes, such as providing compatibility with legacy code, in which case it might choose to replicate the alignment of some old compiler for another architecture rather than to use the alignment required by the target processor.
However, let’s consider structures in separate compilations using one C implementation. Then C 2018 6.2.7 1 says:
… Moreover, two structure, union, or enumerated types declared in separate translation units are compatible if their tags and members satisfy the following requirements: If one is declared with a tag, the other shall be declared with the same tag. If both are completed anywhere within their respective translation units, then the following additional requirements apply: there shall be a one-to-one correspondence between their members such that each pair of corresponding members are declared with compatible types; if one member of the pair is declared with an alignment specifier, the other is declared with an equivalent alignment specifier; and if one member of the pair is declared with a name, the other is declared with the same name. For two structures, corresponding members shall be declared in the same order. For two structures or unions, corresponding bit-fields shall have the same widths…
Therefore, if two structures are declared identically in separate translation units, or with the minor variations permitted in that passage, then they are compatible, which effectively means they have the same layout or representation.
Technically, that passage applies only to separate translation units of the same program. The C standard defines behaviors for one program; it does not explicitly define interactions between programs (or fragments of programs, such as kernel extensions) and the operating system, although to some extent you might consider the operating system and everything running in it as one program. However, for practical purposes, it applies to everything compiled with that C implementation.
This means that as long as you use the same C implementation as the kernel is compiled with, identically declared structures will have the same representation.
Another consideration is that we might use different compilers for compiling the kernel and compiling programs. The kernel might be compiled with Clang while a user prefers to use GCC. In this case, it is a matter for the compilers to document their behaviors. The C standard does not guarantee compatibility, but the compilers can, if they choose to, perhaps by both documenting that they adhere to a particular Application Binary Interface (ABI).
Also note that a “C implementation” as discussed above is not just a particular compiler but a particular compiler with particular switches. Various switches may change how a compiler behaves in ways that cause to be effectively a different C implementation, such as switches to conform to one version of the C standard or another, switches affecting whether structures are packed, switches affecting sizes of integer types, and so on.
I am reading "Computer Systems: A Programmer's Perspective", in chapter 8, it says sig_atomic can be use to be the type for flag which can be used in signal handler. But when I trace my header file, it is defined as int without volatile, isn't it means there is a possibility that when main modify the flag, it may use 'write back' instead of 'write through', which might be incorrect?
Because sig_atomic_t should in practice always be qualified with volatile (at least for variables set inside signal handlers). It is not (formally) related to C11 atomic types (even if the name could be confusing). But sig_atomic_t should in practice be some integral type which can be written to or read from memory in a single machine code instruction (and you need to qualify it as volatile to make that happen as needed, and instruct the compiler to not cache it in some register etc...).
For Linux, see also signal(7) and most importantly signal-safety(7).
Notice that the C11 standard does not tell a lot about signals (check by reading n1570, notably §7.14). In practice, you probably need some operating system support for them. See also POSIX signal concepts.
Read also Operating Systems: Three Easy Pieces
Please tell me, where can I find the wchar_t declaration.
I use linux, and I think it is 32 bits.
I need to declarate this type, because i can't use the standart library (it is used in my boot programm).
The files /usr/include/wchar.h and /usr/include/linux/stddef.h
don't contain the declaration of it.
Also what about mbstate_t?
If you are not using the standard library, then you do not need wchar_t. Or at least the standard library's idea of wchar_t doesn't matter to you. How could it? If you want wide character handling then you'll need to write whatever functions are needed for it, and you are free to define and use whatever types are most suitable / convenient for that purpose.
You probably will need the kernel headers, though, if you intend make system calls. I can't imagine how your boot program could avoid that, or why it would want to do. Those types you would need for that purpose will be defined among the kernel headers.
You will not need kernel headers, either: since the point of your program is to load and start the kernel, it cannot rely on system calls into the kernel to do its job.
The bottom line is that since you have to provide everything not available directly from the hardware / firmware -- which does not present a C interface -- no C definitions outside your own code are relevant to you. In particular, wchar_t is not a characteristic of any system; rather, it is a characteristic of a particular C library. Different C libraries for the very same system, including yours, can freely define it differently. If you choose to implement your own wide-character functions, there is no advantage whatever to choosing a wchar_t definition drawn from some other C library.
You don't need to go to the files containing the definition of wchar_t.
In the standard header stdint.h it can be found the information about integer types in your particular implementation.
For example, the constants WCHAR_MIN and WCHAR_MAX bring information about the range of values of wchar_t.
On the ohter hand, if WCHAR_MIN == 0, then wchar_t is an unsigned integer type. Else, it's a signed integer type (in this case it would prefirable to check the condition WCHAR_MIN < 0).
To know the amount of bytes used to represent a wchar_t object, you have, of course, the expression sizeof(wchar_t).
While looking for ways to find the size of a file given a FILE*, I came across this article advising against it. Instead, it seems to encourage using file descriptors and fstat.
However I was under the impression that fstat, open and file descriptors in general are not as portable (After a bit of searching, I've found something to this effect).
Is there a way to get the size of a file in ANSI C while keeping in line with the warnings in the article?
In standard C, the fseek/ftell dance is pretty much the only game in town. Anything else you'd do depends at least in some way on the specific environment your program runs in. Unfortunately said dance also has its problems as described in the articles you've linked.
I guess you could always read everything out of the file until EOF and keep track along the way - with fread() for example.
The article claims fseek(stream, 0, SEEK_END) is undefined behaviour by citing an out-of-context footnote.
The footnote appears in text dealing with wide-oriented streams, which are streams that the first operation that is performed on them is an operation on wide-characters.
This undefined behaviour stems from the combination of two paragraphs. First §7.19.2/5 says that:
— Binary wide-oriented streams have the file-positioning restrictions ascribed to both text and binary streams.
And the restrictions for file-positioning with text streams (§7.19.9.2/4) are:
For a text stream, either offset shall be zero, or offset shall be a value returned by an earlier successful call to the ftell function on a stream associated with the same file and whence shall be SEEK_SET.
This makes fseek(stream, 0, SEEK_END) undefined behaviour for wide-oriented streams. There is no such rule like §7.19.2/5 for byte-oriented streams.
Furthermore, when the standard says:
A binary stream need not meaningfully support fseek calls with a whence value of SEEK_END.
It doesn't mean it's undefined behaviour to do so. But if the stream supports it, it's ok.
Apparently this exists to allow binary files can have coarse size granularity, i.e. for the size to be a number of disk sectors rather than a number of bytes, and as such allows for an unspecified number of zeros to magically appear at the end of binary files. SEEK_END cannot be meaningfully supported in this case. Other examples include pipes or infinite files like /dev/zero. However, the C standard provides no way to distinguish between such cases, so you're stuck with system-dependent calls if you want to consider that.
Use fstat - requires the file descriptor - can get that from fileno from the FILE* - Hence the size is in your grasp along with other details.
i.e.
fstat(fileno(filePointer), &buf);
Where filePointer is the FILE *
and
buf is
struct stat {
dev_t st_dev; /* ID of device containing file */
ino_t st_ino; /* inode number */
mode_t st_mode; /* protection */
nlink_t st_nlink; /* number of hard links */
uid_t st_uid; /* user ID of owner */
gid_t st_gid; /* group ID of owner */
dev_t st_rdev; /* device ID (if special file) */
off_t st_size; /* total size, in bytes */
blksize_t st_blksize; /* blocksize for file system I/O */
blkcnt_t st_blocks; /* number of 512B blocks allocated */
time_t st_atime; /* time of last access */
time_t st_mtime; /* time of last modification */
time_t st_ctime; /* time of last status change */
};
The executive summary is that you must use fseek/ftell because there is no alternative (even the implementation specific ones) that is better.
The underlying issue is that the "size" of a file in bytes is not always the same as the length of the data in the file and that, in some circumstances, the length of the data is not available.
A POSIX example is what happens when you write data to a device; the operating system only knows the size of the device. Once the data has been written and the (FILE*) closed there is no record of the length of the data written. If the device is opened for read the fseek/ftell approach will either fail or give you the size of the whole device.
When the ANSI-C committee was sitting at the end of the 1980's a number of operating systems the members remembered simply did not store the length of the data in a file; rather they stored the disk blocks of the file and assumed that something in the data terminated it. The 'text' stream represents this. Opening a 'binary' stream on those files shows not only the magic terminator byte, but also any bytes beyond it that were never written but happen to be in the same disk block.
Consequently the C-90 standard was written so that it is valid to use the fseek trick; the result is a conformant program, but the result may not be what you expect. The behavior of that program is not 'undefined' in the C-90 definition and it is not 'implementation-defined' (because on UN*X it varies with the file). Neither is it 'invalid'. Rather you get a number you can't completely rely on or, maybe, depending on the parameters to fseek, -1 and an errno.
In practice if the trick succeeds you get a number that includes at least all the data, and this is probably what you want, and if the trick fails it is almost certainly someone else's fault.
John Bowler
different OS's provide different apis for this. For example in windows we have:
GetFileAttributes()
In MAC we have:
[[[NSFileManager defaultManager] attributesOfItemAtPath:someFilePath error:nil] fileSize];
But raw method is only by fread and fseek only:
How can I get a file's size in C?
You can't always avoid writing platform-specific code, especially when you have to deal with things that are a function of the platform. File sizes are a function of the file system, so as a rule I'd use the native filesystem API to get that information over the fseek/ftell dance. I'd create my own generic wrapper around it, so as to not pollute application logic with platform-specific details and make the code easier to port.
The article has a little problem of logic.
It (correctly) identifies that a certain usage of C functions has behavior which is not defined by ISO C. But then, to avoid this undefined behavior, the article proposes a solution: replace that usage with platform-specific functions. Unfortunately, the use of platform-specific functions is also undefined according to ISO C. Therefore, the advice does not solve the problem of undefined behavior.
The quote in my copy of the 1999 standard confirms that the alleged behavior is indeed undefined:
A binary stream need no meaningfully support fseek calls with a whence value of SEEK_END. [ISO 9899:1999 7.19.9.2 paragraph 3]
But undefined behavior does not mean "bad behavior"; it is simply behavior for which the ISO C standard gives no definition. Not all undefined behaviors are the same.
Some undefined behaviors are areas in the language where meaningful extensions can be provided. The platform fills the gap by defining a behavior.
Providing a working fseek which can seek from SEEK_END is an example of an extension in place of undefined behavior. It is possible to confirm whether or not a given platform supports fseek from SEEK_END, and if this is provisioned, then it is fine to use it.
Providing a separate function like lseek is also an extension in place of undefined behavior (the undefined behavior of calling a function which is not in ISO C and not defined in the C program). It is fine to use that, if available.
Note that those platforms which have functions like the POSIX lseek will also likely have an ISO C fseek which works from SEEK_END. Also note that on platforms where fseek on a binary file cannot seek from SEEK_END, the likely reason is that this is impossible to do (no API can be provided to do it and that is why the C library function fseek is not able to support it).
So, if fseek does provide the desired behavior on the given platform, then nothing has to be done to the program; it is a waste of effort to change it to use that platform's special function. On the other hand, if fseek does not provide the behavior, then likely nothing does, anyway.
Note that even including a nonstandard header which is not in the program is undefined behavior. (By omission of the definition of behavior.) For instance if the following appears in a C program:
#include <unistd.h>
the behavior is not defined after that. [See References below.] The behavior of the preprocessing directive #include is defined, of course. But this creates two possibilities: either the header <unistd.h> does not exist, in which case a diagnostic is required. Or the header does exist. But in that case, the contents are not known (as far as ISO C is concerned; no such header is documented for the Library). In this case, the include directive brings in an unknown chunk of code, incorporating it into the translation unit. It is impossible to define the behavior of an unknown chunk of code.
#include <platform-specific-header.h> is one of the escape hatches in the language for doing anything whatsoever on a given platform.
In point form:
Undefined behavior is not inherently "bad" and not inherently a security flaw (though of course it can be! E.g. buffer overruns linked to the undefined behaviors in the area of pointer arithmetic and dereferencing.)
Replacing one undefined behavior with another, only for the purpose of avoiding undefined behavior, is pointless.
Undefined behavior is just a special term used in ISO C to denote things that are outside of the scope of ISO C's definition. It does not mean "not defined by anyone in the world" and doesn't imply something is defective.
Relying on some undefined behaviors is necessary for making most real-world, useful programs, because many extensions are provided through undefined behavior, including platform-specific headers and functions.
Undefined behavior can be supplanted by definitions of behavior from outside of ISO C. For instance the POSIX.1 (IEEE 1003.1) series of standards defines the behavior of including <unistd.h>. An undefined ISO C program can be a well defined POSIX C program.
Some problems cannot be solved in C without relying on some kind of undefined behavior. An example of this is a program that wants to seek so many bytes backwards from the end of a file.
References:
Dan Pop in comp.std.c, Dec. 2002: http://groups.google.com/group/comp.std.c/msg/534ab15a7bc4e27e?dmode=source
Chris Torek, comp.std.c, on the subject of nonstandard functions being undefined behavior, Feb. 2002: http://groups.google.com/group/comp.lang.c/msg/2fddb081336543f1?dmode=source
Chris Engebretson, comp.lang.c, April 1997: http://groups.google.com/group/comp.lang.c/msg/3a3812dbcf31de24?dmode=source
Ben Pfaff, comp.lang.c, Dec 1998 [Jestful answer citing undefinedness of the inclusion of nonstandard headers]: http://groups.google.com/group/comp.lang.c/msg/73b26e6892a1ba4f?dmode=source
Lawrence Kirby, comp.lang.c, Sep 1998 [Explains effects of nonstandard headers]: http://groups.google.com/group/comp.lang.c/msg/c85a519fc63bd388?dmode=source
Christian Bau, comp.lang.c, Sep 1997 [Explains how the undefined behavior of #include <pascal.h> can bring in a pascal keyword for linkage.] http://groups.google.com/group/comp.lang.c/msg/e2762cfa9888d5c6?dmode=source