I want to be able to tell when my program's stdout is redirected to a file/device, and when it is left to print normally on the screen. How can this be done in C?
Update 1: From the comments, it seems to be system dependent. If so, then how can this be done with posix-compliant systems?
Perhaps isatty(stdout)?
Edit: As Roland and tripleee suggest, a better answer would be isatty(STDOUT_FILENO).
Look up isatty and more generally fileno.
I am afraid that you can't, at least with standard C in a platform independent manner. The idea behind standard input/output is that C will do it's IO from a standard place. That standard place could be a terminal or a file or anything else, that is not the consideration of C. So you can't detect what is standard IO currently used.
EDIT: If a platform specific solution is okay for you then please refer to other answers (and also edit the question accordingly).
If a Linux-specific solution is OK, you can examine the symlinks under the /proc directory for your process. E.g.,
$ exec 3>/dev/null
$ ls -l /proc/$$/fd
total 0
lrwx------ 1 root root 64 Sep 12 03:28 0 -> /dev/pts/1
lrwx------ 1 root root 64 Sep 12 03:29 1 -> /dev/pts/1
lrwx------ 1 root root 64 Sep 12 03:29 2 -> /dev/pts/1
lrwx------ 1 root root 64 Sep 12 03:29 255 -> /dev/pts/1
l-wx------ 1 root root 64 Sep 12 03:29 3 -> /dev/null
You might want to check this out:
http://www.cplusplus.com/reference/clibrary/cstdio/freopen/
I'm quoting from the link:
freopen
Reopen stream with different file or mode
freopen first tries to close any file already associated with the stream given as third parameter and disassociates it.
Then, whether that stream was successfuly closed or not, freopen opens the file whose name is passed in the first parameter, filename, and associates it with the specified stream just as fopen would do using the mode value specified as the second parameter.
This function is specially useful for redirecting predefined streams like stdin, stdout and stderr to specific files.
Though I'm not sure if this'll help you find out what it is pointing to in the first place.
Related
I try to get the name of the folder / file pointed by the symbolic link.
For example, when you do ls -l
You can get this result
-rw-r--r-- 1 macos staff 0 Feb 22 12:05 test
lrwxr-xr-x 1 macos staff 7 Feb 19 11:05 sl -> test
How could I do, if I know the path of sl to get the file/folder here it pointed to ?
I want to do it using stat if possible.
Any suggestion ?
So to summarize :
It's impossible to get the name of the file/folder pointed by a symbolic link, using lstat or stat.
The function readlink can do the job => Man : http://man7.org/linux/man-pages/man2/readlink.2.html. You will get example in the man
Thank you for helping me solve my problem
Let's say I have a module which has this function my_open:
int my_open( struct inode *inode, struct file *filp ) {
filp->private_data = //allocate private data
if( filp->f_mode & FMODE_READ )
//handle read opening
if( filp->f_mode & FMODE_WRITE )
//handle write opening
if (MINOR( inode->i_rdev )==2){
filp->f_op = &my_fops2;
}
return 0;
}
how do I use this function from the shell/terminal?
This is either the open function for a device driver,
or it’s a sheep in wolf’s clothing [sic].
In the unlikely event that this is ordinary, vanilla user-level code,
compile it into an executable and use that. But if it is a device driver’s open function,
First of all,
determine whether it is compiled/linked into the currently running kernel.
If it isn’t, make it so (compile/link it into the kernel, and reboot).
Alternatively,
you may have the ability to load it into the kernel dynamically.
Exactly how to do that depends on your particular operating system
and is out of scope for this question.
Find out whether it is a block device or a character device,
and what its major device number is.
I can’t tell you specifically how to do that, either;
consult your local resources.
OK, let’s suppose that it is a character device with major number 42.
Look through /dev (with ls -l) for entries that begin with c
(for “character”) and contain 42, something
where the size should be, like this:
drwxr-xr-x 1 root root 512 Feb 10 2015 .
drwxr-xr-x 1 root root 1024 Feb 10 2015 ..
crw-rw-rw- 1 root root 42, 0 Aug 15 18:31 foo
crw-rw-rw- 1 root root 42, 2 Aug 15 18:31 fu
crw-rw-rw- 1 root root 42, 17 Aug 15 18:31 fubar
If you can’t find any, create some. See man mknod for details.
You should probably create one with minor device number 2
and at least one with a different number
(because the code treats 2 as a special case).
Do to the /dev/whatever files whatever you want to,
based on the intended function of the driver.
(Determining the intended function of the driver is out of scope.)
For instance, you might try things like
od -cb /dev/foo
echo "Hello, world." > /dev/fu
Naturally, if it’s a block device,
replace c in the above instructions with b.
The problem incident:
Our production system started denying services with an error message "Too many open files in system". Most of the services were affected, including inability to start a new ssh session, or even log in into virtual console from the physical terminal. Luckily, one root ssh session was open, so we could interact with the system (morale: keep one root session always open!). As a side effect, some services (named, dbus-daemon, rsyslogd, avahi-daemon) saturated the CPU (100% load). The system also serves a large directory via NFS to a very busy client which was backing up 50000 small files at the moment. Restarting all kinds of services and programs normalized their CPU behavior, but did not solve the "Too many open files in system" problem.
The suspected cause
Most likely, some program is leaking file handles. Probably the culprit is my tcl program, which also saturated the CPU (not normal). However, killing it did not help, but, most disturbingly, lsof would not reveal large amounts of open files.
Some evidence
We had to reboot, so whatever information was collected is all we have.
root#xeon:~# cat /proc/sys/fs/file-max
205900
root#xeon:~# lsof
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
init 1 root cwd DIR 8,6 4096 2 /
init 1 root rtd DIR 8,6 4096 2 /
init 1 root txt REG 8,6 124704 7979050 /sbin/init
init 1 root mem REG 8,6 42580 5357606 /lib/i386-linux-gnu/libnss_files-2.13.so
init 1 root mem REG 8,6 243400 5357572 /lib/i386-linux-gnu/libdbus-1.so.3.5.4
...
A pretty normal list, definitely not 200K files, more like two hundred.
This is probably, where the problem started:
less /var/log/syslog
Mar 27 06:54:01 xeon CRON[16084]: (CRON) error (grandchild #16090 failed with exit status 1)
Mar 27 06:54:21 xeon kernel: [8848865.426732] VFS: file-max limit 205900 reached
Mar 27 06:54:29 xeon postfix/master[1435]: warning: master_wakeup_timer_event: service pickup(public/pickup): Too many open files in system
Mar 27 06:54:29 xeon kernel: [8848873.611491] VFS: file-max limit 205900 reached
Mar 27 06:54:32 xeon kernel: [8848876.293525] VFS: file-max limit 205900 reached
netstat did not show noticeable anomalies either.
The man pages for ps and top do not indicate an ability to show open file count. Probably the problem will repeat itself after a few months (that was our uptime).
Any ideas on what else can be done to identify the open files?
UPDATE
This question has changed the meaning, after qehgt identified the likely cause.
Apart from the bug in NFS v4 code, I suspect there is a design limitation in Linux and kernel-leaked file handles can NOT be identified. Consequently, the original question transforms into:
"Who is responsible for file handles in the Linux kernel?" and "Where do I post that question?". The 1st answer was helpful, but I am willing to accept a better answer.
Probably the root cause is a bug in NFSv4 implementation: https://stackoverflow.com/a/5205459/280758
They have similar symptoms.
I know what dup / dup2 does, but I have no idea when it would be used.
Any practical examples?
Thanks.
One example use would be I/O redirection. For this you fork a child process and close the stdin or stdout file descriptors (0 and 1) and then you do a dup() on another filedescriptor of your choice which will now be mapped to the lowest available file descriptor, which is in this case 0 or 1.
Using this you can now exec any child process which is possibly unaware of your application and whenever the child writes on the stdout (or reads from stdin, whatever you configured) the data gets written on the provided filedescriptor instead.
Shells use this to implement commands with pipes, e.g. /bin/ls | more by connecting the stdout of one process to the stdin of the other.
The best scenario to understand dup and dup2 is redirection.
First thing we need to know is that the system has 3 default file ids(or variables indicating output or input sources) that deals with the input and output. They are stdin, stdout, stderr, in integers they are 0,1,2. Most of the functions like fprintf or cout are directly output to stdout.
If we want to redirect the output, one way is give, for example, fprintf function more arguments indicating in and out.
However, there is a more elegant way: we can overwrite the default file ids to make them pointing to the file we want to receive the output. dup and dup2 exactly work in this situation.
Let's start with one simple example now: suppose we want to redirect the output of fprintf to a txt file named "chinaisbetter.txt". First of all we need to open this file
int fw=open("chinaisbetter.txt", O_APPEND|O_WRONLY);
Then we want stdout to point to "chinaisbetter.txt" by using dup function:
dup2(fw,1);
Now stdout(1) points to the descriptor of "chinaisbetter.txt" even though it's still 1, but the output is redirected now.
Then you can use printf as normal, but the results will be in the txt file instead of showing directly on the screen:
printf("Are you kidding me? \n");
PS:
This just gives a intuitive explanation, you may need to check the manpage or detailed information. Actually, we say "copy" here, they are not copying everything.
The file id here is referring to the handler of the file. The file descriptor mentioned above is a struct the records file's information.
When you are curious about POSIX functions, especially those that seem to duplicate themselves, it's generally good to check the standard itself. At the bottom you will usually see examples, as well as reasoning behind the implementation (and existence) of both.
In this case:
The following sections are informative.
Examples
Redirecting Standard Output to a File
The following example closes standard output for the current processes, re-assigns standard output to go to the file referenced by pfd, and closes the original file descriptor to clean up.
#include <unistd.h>
...
int pfd;
...
close(1);
dup(pfd);
close(pfd);
...
Redirecting Error Messages
The following example redirects messages from stderr to stdout.
#include <unistd.h>
...
dup2(2, 1); // 2-stderr; 1-stdout
...
Application Usage
None.
Rationale
The dup() and dup2() functions are redundant. Their services are also provided by the fcntl() function. They have been included in this volume of IEEE Std 1003.1-2001 primarily for historical reasons, since many existing applications use them.
While the brief code segment shown is very similar in behavior to dup2(), a conforming implementation based on other functions defined in this volume of IEEE Std 1003.1-2001 is significantly more complex. Least obvious is the possible effect of a signal-catching function that could be invoked between steps and allocate or deallocate file descriptors. This could be avoided by blocking signals.
The dup2() function is not marked obsolescent because it presents a type-safe version of functionality provided in a type-unsafe version by fcntl(). It is used in the POSIX Ada binding.
The dup2() function is not intended for use in critical regions as a synchronization mechanism.
In the description of [EBADF], the case of fildes being out of range is covered by the given case of fildes not being valid. The descriptions for fildes and fildes2 are different because the only kind of invalidity that is relevant for fildes2 is whether it is out of range; that is, it does not matter whether fildes2 refers to an open file when the dup2() call is made.
Future Directions
None.
See also
close(), fcntl(), open(), the Base Definitions volume of IEEE Std 1003.1-2001, <unistd.h>
Change History
First released in Issue 1. Derived from Issue 1 of the SVID.
One practical example is redirecting output messages to some other stream like some log file. Here is a sample code for I/O redirection.
Please refer to original post here
#include <stdio.h>
main()
{
int fd;
fpos_t pos;
printf("stdout, ");
fflush(stdout);
fgetpos(stdout, &pos);
fd = dup(fileno(stdout));
freopen("stdout.out", "w", stdout);
f();
fflush(stdout);
dup2(fd, fileno(stdout));
close(fd);
clearerr(stdout);
fsetpos(stdout, &pos); /* for C9X */
printf("stdout again\n");
}
f()
{
printf("stdout in f()");
}
I/O redirection in the shell would most likely be implemented using dup2/fcnlt system calls.
We can easily emulate the $program 2>&1 > logfile.log type of redirection using the dup2 function.
The program below redirects both stdout and stderr .i.e emulates behavior of $program 2>&1 > output using the dup2.
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <fcntl.h>
int
main(void){
int close_this_fd;
dup2(close_this_fd = open("output", O_WRONLY), 1);
dup2(1,2);
close(close_this_fd);
fprintf(stdout, "standard output\n");
fprintf(stderr, "standard error\n");
fflush(stdout);
sleep(100); //sleep to examine the filedes in /proc/pid/fd level.
return;
}
vagrant#precise64:/vagrant/advC$ ./a.out
^Z
[2]+ Stopped ./a.out
vagrant#precise64:/vagrant/advC$ cat output
standard error
standard output
vagrant#precise64:/vagrant/advC$ ll /proc/2761/fd
total 0
dr-x------ 2 vagrant vagrant 0 Jun 20 22:07 ./
dr-xr-xr-x 8 vagrant vagrant 0 Jun 20 22:07 ../
lrwx------ 1 vagrant vagrant 64 Jun 20 22:07 0 -> /dev/pts/0
l-wx------ 1 vagrant vagrant 64 Jun 20 22:07 1 -> /vagrant/advC/output
l-wx------ 1 vagrant vagrant 64 Jun 20 22:07 2 -> /vagrant/advC/output
I'd like to create a sparse file such that all-zero blocks don't take up actual disk space until I write data to them. Is it possible?
There seems to be some confusion as to whether the default Mac OS X filesystem (HFS+) supports holes in files. The following program demonstrates that this is not the case.
#include <stdio.h>
#include <string.h>
#include <fcntl.h>
#include <unistd.h>
void create_file_with_hole(void)
{
int fd = open("file.hole", O_WRONLY|O_TRUNC|O_CREAT, 0600);
write(fd, "Hello", 5);
lseek(fd, 99988, SEEK_CUR); // Make a hole
write(fd, "Goodbye", 7);
close(fd);
}
void create_file_without_hole(void)
{
int fd = open("file.nohole", O_WRONLY|O_TRUNC|O_CREAT, 0600);
write(fd, "Hello", 5);
char buf[99988];
memset(buf, 'a', 99988);
write(fd, buf, 99988); // Write lots of bytes
write(fd, "Goodbye", 7);
close(fd);
}
int main()
{
create_file_with_hole();
create_file_without_hole();
return 0;
}
The program creates two files, each 100,000 bytes in length, one of which has a hole of 99,988 bytes.
On Mac OS X 10.5 on an HFS+ partition, both files take up the same number of disk blocks (200):
$ ls -ls
total 400
200 -rw------- 1 user staff 100000 Oct 10 13:48 file.hole
200 -rw------- 1 user staff 100000 Oct 10 13:48 file.nohole
Whereas on CentOS 5, the file without holes consumes 88 more disk blocks than the other:
$ ls -ls
total 136
24 -rw------- 1 user nobody 100000 Oct 10 13:46 file.hole
112 -rw------- 1 user nobody 100000 Oct 10 13:46 file.nohole
As in other Unixes, it's a feature of the filesystem. Either the filesystem supports it for ALL files or it doesn't. Unlike Win32, you don't have to do anything special to make it happen. Also unlike Win32, there is no performance penalty for using a sparse file.
On MacOS, the default filesystem is HFS+ which does not support sparse files.
Update: MacOS used to support UFS volumes with sparse file support, but that has been removed. None of the currently supported filesystems feature sparse file support.
This thread becomes a comprehensive source of info about the sparse files. Here is the missing part for Win32:
Decent article with examples
Tool that estimates if it makes sense to make file as sparse
Regards
hdiutil can handle sparse images and files but unfortunately the framework it links against is private.
You could try defining external symbols as defined by the DiskImages framework below but this is most likely not acceptable for production code, plus since the framework is private you'd have to reverse engineer its use cases.
cristi:~ diciu$ otool -L /usr/bin/hdiutil
/usr/bin/hdiutil:
/System/Library/PrivateFrameworks/DiskImages.framework/Versions/A/DiskImages (compatibility version 1.0.8, current version 194.0.0)
[..]
cristi:~ diciu$ nm /System/Library/PrivateFrameworks/DiskImages.framework/Versions/A/DiskImages | awk -F' ' '{print $3}' | c++filt | grep -i sparse
[..]
CSparseFile::sector2Band(long long)
CSparseFile::addIndexNode()
CSparseFile::readIndexNode(long long, SparseFileIndexNode*)
CSparseFile::readHeaderNode(CBackingStore*, SparseFileHeaderNode*, unsigned long)
[... cut for brevity]
Later Edit
You could use hdiutil as an external process and have it create an sparse disk image for you. From the C process you would then create a file in the (mounted) sparse disk image.
If you seek (fseek, ftruncate, ...) to past the end, the file size will be increased without allocating blocks until you write to the holes. But there's no way to create a magic file that automatically converts blocks of zeroes to holes. You have to do it yourself.
This may be helpful to look at (the OpenBSD cp command inserts holes instead of writing zeroes).
patch
If you want portability, the last resort is to write your own access function so that you manage an index and a set of blocks.
In essence you manage a single file as the OS manages the disk keeping the chain of the blocks that are part of the file, the bitmap of allocated/free blocks etc.
Of course this will lead to a non optimized and slower access, I would reccomend this apprach only if the requirement to save space is absolutely critical and you have enough time to write a robust set of access functions.
And even in that case, I would first investigate if your problem is in need of a different solution. Probably you should store your data differently?
It looks like OS X supports sparse files on UDF volumes. I tried titaniumdecoy's test program on OS X 10.9 and it did generate a sparse file on a UDF disk image. Also, not that UFS is no longer supported in OS X, so if you need sparse files, UDF is the only natively supported file system that supports them.
I also tried the program on SMB shares. When the server is Ubuntu (ext4 filesystem) the program creates a sparse file, but 'ls -ls' through SMB doesn't show that. If you do 'ls -ls' on the Ubuntu host itself it does show the file is sparse. When the server is Windows XP (NTFS filesystem) the program does not generate a sparse file.