Recently, I am reading the UNIX Systems Programming, and doing some test.
I'm confused that why i truncate the file with truncate command in the terminal, then the read process read the same position return 0 which return the string "a" before. The process open the file, it will cache the file, beacase i change the file content, then read, the result is the same not changed, so why truncate file will affect the process immediately？In the UNIX Systems Programming, it says that v node include the current file size? so the size don't be read from the disk every time, it's cached in the memory.
the read process code source.
#include <unistd.h>
#include <stdio.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <string.h>
int main(int argc, char *argv[])
{
int fd = open("a.txt", O_RDONLY);
char buf[1024];
int n;
for(int i = 0;i < 10; i++){
if (lseek(fd, 0, 0) == -1){
return 1;
}
n = read(fd, buf, 1);
if (n < 0){
perror("error");
return 1;
}
printf("%d\n", n);
buf[n]=0;
printf("buf %s\n", buf);
sleep(2);
}
return 0;
}
a.txt only have one char:
a
Caching is usually intended to improve performance, without having any other detectable effects.
If the inode was cached by the process, as you describe, then modifications made by another process might not be visible by processes that had cached information. That would be bad.
In reality, disk blocks are cached, inodes are cached, various other structures might be cached, but in each case there is a single cache, in the kernel, which all processes share, so they all have a consistent view.
When one process truncates the file, the in-memory cache is updated, storing the new file size (which will eventually be written to disk, but probably not immediately). When the other process calls read again, the updated file size is read from the in-memory cache, not from disk.
Related
I am trying to create a simple program using pipes, even though that are easier options for the same task (fopen(), lseek(), ftell(), etc).
First I use execve() to perform a terminal cat, and send the information through the pipe so I may be able to print the size of the file descriptor and read it to a malloc'd char pointer. My solution is this one:
#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <sys/wait.h>
#include <sys/ioctl.h>
#include <fcntl.h>
int main (int argc, char *argv[]) {
int fd[2], active;
int pipe_sz = 0;
char *name;
pipe(fd);
//change /bin/cat according to your system
char *cmd[] = {"/bin/cat", "example.txt", NULL};
if (fork() == 0) {
//CHILD
dup2(fd[1], STDOUT_FILENO);
close(fd[1]);
active = execve(cmd[0], cmd, NULL);
} else {
//PARENT
wait(NULL);
int ioctl_sz = ioctl(fd[0], FIONREAD, &pipe_sz);
name = malloc(pipe_sz + 1);
int result = read(fd[0], name, pipe_sz);
name[pipe_sz] = '\0';
printf("LEN NAME: %ld\n", strlen(name));
close(fd[0]);
close(fd[1]);
}
return 0;
}
Everything works fine as long as I keep myself inside the pipe's limits (my LINUX system is able to hold 65536 bytes). I decided to push the limits, and see what could happen. As expected, the program stays stuck when my file was above 65536 bytes.
I do not want to manually change my pipe's limits through fcntl(pipefd, F_SETPIPE_SZ, size) (since I "don't know" my file's size). Therefore, I researched, and came across the pipe2(pipefd, O_NONBLOCK) in order to avoid my pipe from stopping the reading process. Unfortunately, I received only this message:
/bin/cat: write error: Resource temporarily unavailable
I even tried a while loop read() and realloc() to see if my code could at least give the char pointer result, but I was not successful, and part of file was lost.
Is it possible to produce a code that may provide the same results as mine above with files bigger than 65536 bytes?
Is there a way to interrupt the pipe if it takes too long to finish the process? Or even make a while loop to guess and resize the pipe through
fcntl()?
Thanks everyone for the help!
I'm trying to trigger some concurrent conflicts by having several processes writing to the same file, but couldn't:
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>
#include <fcntl.h>
#include <sys/wait.h>
void concurrent_write()
{
int create_fd = open("bar.txt", O_CREAT | O_TRUNC, 0644);
close(create_fd);
int repeat = 20;
int num = 4;
for (int process = 0; process < num; process++)
{
int rc = fork();
if (rc == 0)
{
// child
int write_fd = open("bar.txt", O_WRONLY | O_APPEND, 0644);
for (int idx = 0; idx < repeat; idx++)
{
sleep(1);
write(write_fd, "child writing\n", strlen("child writing\n"));
}
close(write_fd);
exit(0);
}
}
for (int process = 0; process < num; process++)
{
wait(NULL);
// wait for all children to exits
}
printf("write to `bar.txt`\n%d lines written by %d process\n", repeat * num, num);
printf("wc:");
if (fork() == 0)
{
// child
char *args[3];
args[0] = strdup("wc");
args[1] = strdup("bar.txt");
args[2] = NULL;
execvp(args[0], args);
}
}
int main(int argc, char *argv[])
{
concurrent_write();
return 0;
}
This program fork #num children and then have all of them write #repeat lines to a file. But every time (however I change #repeat and #num) I got the same result that the length of bar.txt (output file) matched the number of total written lines. Why is there no concurrent conflicts triggered?
Writing to a file can be divided into a two-step process:
Locate where you want to write.
Write data into the file.
You open a file with flag O_APPEND and it ensures that the two-step process is atomic. So, you can always find the lines of the file as the count you set.
See the open(2) man page:
O_APPEND
The file is opened in append mode. Before each write(2),
the file offset is positioned at the end of the file, as
if with lseek(2). The modification of the file offset and
the write operation are performed as a single atomic step.
In essence, one of the major design features of O_APPEND is precisely to prevent the sort of "concurrent conflicts" you mention. The typical example would be a log file that several processes must write to. Using O_APPEND ensures their messages do not overwrite each other.
Moreover, all data written by a single write call is written atomically, so provided that your write("child writing\n") successfully writes all its bytes (which for a regular file it usually would), they will not be interleaved with the bytes of any other such message.
First, write() calls with the O_APPEND flag should be atomic. Per POSIX write():
If the O_APPEND flag of the file status flags is set, the file offset shall be set to the end of the file prior to each write and no intervening file modification operation shall occur between changing the file offset and the write operation.
But that's not enough when there are multiple threads or processes making parallel write() calls on the same file - that does not guarantee that parallel write() calls are atomic.
POSIX does guarantee that parallel write() calls are also atomic:
All of the following functions shall be atomic with respect to each
other in the effects specified in POSIX.1-2017 when they operate on
regular files or symbolic links:
...
write()
...
See also Is file append atomic in UNIX?
Beware, though. Reading that question and its answers shows that Linux filesystems such as ext3 are not POSIX compliant once you get past a relatively small size operation, or possibly if you cross page and/or file system sector boundaries. I suspect XFS and ZFS will support write() atomicity much better given their origins.
And none of this applies to Windows.
I wrote this piece of code to show the basic working of how I would like to send some data (Strings) from the parent process to the child process. But I seem to have some problems. (I removed all error checking to make the code more readable)
When I run this piece of code I expect to see the two test strings to be displayed on the terminal, but I only see the first one.
When I uncomment the first “sleep(1)”, then I see both strings displayed.
But when I uncomment only the second “sleep(1)”, then I again only see the first string.
I suspect this problem has something to do with synchronization. That the strings get written to fast and the fifo write end closes before everything is read by the child process. That’s why we see the correct output when we introduce a sleep between the two write() commands.
But what I don’t understand is that we still get a faulty output when we only introduce a sleep after both write commands. Why can’t the child read both strings even if they are both written before it can read one?
How can I solve this problem? Do I need some synchronization code, and if so how should I implement this. Because I won’t write a “sleep(1)” after every write command.
And is the solution also viable for multiple processes that want to write to the same fifo? (but with still only one process that reads from the fifo)
#include <stdio.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <sys/wait.h>
#include <fcntl.h>
#include <unistd.h>
#include <string.h>
int main(int argc, char const *argv[]) {
mkfifo("test_fifo", 0666);
pid_t pid = fork();
if (pid == 0) {
int fd = open("test_fifo", O_RDONLY);
char data[400];
int rc;
do {
rc = read(fd, data, 400);
if (rc > 0) printf("Received data: %s\n", data);
} while(rc > 0);
}
else {
int fd = open("test_fifo", O_WRONLY);
char * string1 = "This is the first test string";
write(fd, string1, strlen(string1) + 1);
//sleep(1);
char * string2 = "This is the second test string";
write(fd, string2, strlen(string2) + 1);
//sleep(1);
close(fd);
wait(NULL);
}
return 0;
}
You are receiving both strings at the same time at the first call to read. Because %s prints up until a zero byte, the second string is just not displayed. The poor mans synchronization with sleep(1) allows child to "catch" the messages in two distinct read call.
read returns the count of bytes read. Use that number. Change the parent code to:
ssize_t rc;
do {
rc = read(fd, data, 400);
if (rc > 0) {
printf("Received data: ");
for (size_t i = 0; i < rc; ++i) {
if (data[i] == '\0') {
printf("\\x00");
continue;
}
printf("%c", data[i]);
}
printf("\n");
}
} while(rc >= 0);
and it shows on my pc:
Received data: This is the first test string\x00This is the second test string\x00
Why can’t the child read both strings even if they are both written before it can read one?
Well, the problem is not in reading, it's how you are displaying the data you read. (Still, reading could be improved, one should handle that pesky EAGAIN errno code).
How can I solve this problem?
If you want 1:1 relationship between read/write use a constant size packets or generally you have to know in advance how many bytes you want to read. Bytes written are "concatenated" together and lose structure. Or use pipe(3p) on which messages with size smaller then PIPE_BUF are guaranteed to be atomic. Or you could use POSIX message queue mq_receive/mq_send.
Or write a proper "deserializer", something that will buffer data and keep internal state and notify higher level only when a whole "message" was received, ie. detect when a zero byte was received in the stream of bytes and restore structure the the stream of bytes.
I am working on an application wherein i need to compare 10^8 entries(alphanumeric entries). To retrieve the entries from file( file size is 1.5 GB) and then to compare them, i need to take less than 5 minutes of time. So, what would b the effective way to do that, since, only retrieving time is exceeding 5 min. And i need to work on file only. please suggest a way out.
I m working on windows with 3GB RAM n 100Gb hard disk.
Read a part of the file, sort it, write it to a temporary file.
Merge-sort the resulting files.
Error handling and header includes are not included. You need to provide DataType and cmpfunc, samples are provided. You should be able to deduce the core workings from this snippet:
#include <sys/mman.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <stdlib.h>
#include <unistd.h>
typedef char DataType; // is this alphanumeric?
int cmpfunc(char const *left, char const *right)
{
return *right - *left;
}
int main(int argc, char **argv)
{
int fd = open(argv[1], O_RDWR|O_LARGEFILE);
if (fd == -1)
return 1;
struct stat st;
if (fstat(fd, &st) != 0)
return 1;
DataType *data = mmap(NULL, st.st_size, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0);
if (!data)
return 1;
qsort(data, st.st_size / sizeof(*data), cmpfunc);
if (0 != msync(data, st.st_size, MS_SYNC))
return 1;
if (-1 == munmap(data, st.st_size))
return 1;
if (0 != close(fd))
return 1;
return 0;
}
I can't imagine you can get much faster than this. Be sure you have enough virtual memory address space (1.5GB is pushing it but will probably just work on 32bit Linux, you'll be able to manage this on any 64bit OS). Note that this code is "limited" to working on a POSIX compliant system.
In terms of C and efficiency, this approach puts the entire operation in the hands of the OS, and the excellent qsort algorithm.
If retrieving time is exceeding 5 min it seems that you need to look at how you are reading this file. One thing that has caused bad performance for me is that a C implementation sometimes uses thread-safe I/O operations by default, and you can gain some speed by using thread-unsafe I/O.
What kind of computer will this be run on? Many computers nowadays have several gigabytes of memory, so perhaps it will work to just read it all into memory and then sort it there (with, for example, qsort)?
here's another question about splice(). I'm hoping to use it to copy files, and am trying to use two splice calls joined by a pipe like the example on splice's Wikipedia page. I wrote a simple test case which only tries to read the first 32K bytes from one file and write them to another:
#define _GNU_SOURCE
#include <fcntl.h>
#include <stdio.h>
#include <unistd.h>
#include <errno.h>
#include <string.h>
int main(int argc, char **argv) {
int pipefd[2];
int result;
FILE *in_file;
FILE *out_file;
result = pipe(pipefd);
in_file = fopen(argv[1], "rb");
out_file = fopen(argv[2], "wb");
result = splice(fileno(in_file), 0, pipefd[1], NULL, 32768, SPLICE_F_MORE | SPLICE_F_MOVE);
printf("%d\n", result);
result = splice(pipefd[0], NULL, fileno(out_file), 0, 32768, SPLICE_F_MORE | SPLICE_F_MOVE);
printf("%d\n", result);
if (result == -1)
printf("%d - %s\n", errno, strerror(errno));
close(pipefd[0]);
close(pipefd[1]);
fclose(in_file);
fclose(out_file);
return 0;
}
When I run this, the input file seems to be read properly, but the second splice call fails with EINVAL. Anybody know what I'm doing wrong here?
Thanks!
From the splice manpage:
EINVAL Target file system doesn't support splicing; target file is
opened in append mode; neither of the descriptors refers to a
pipe; or offset given for non-seekable device.
We know one of the descriptors is a pipe, and the file's not open in append mode. We also know no offset is given (0 is equivalent to NULL - did you mean to pass in a pointer to a zero offset?), so that's not the problem. Therefore, the filesystem you're using doesn't support splicing to files.
What kind of file system(s) are you copying to/from?
Your example runs on my system when both files are on ext3 but fails when I use an external drive (I forget offhand if it is DOS or NTFS). My guess is that one or both of your files are on a file system that splice does not support.
The splice(2) system call is for copying between files and pipes and not between files, so it can not be used to copy between files, as has been pointed out by the other answers.
As of Linux 4.5 however a new copy_file_range(2) system call is available that can copy between files. In the case of NFS it can even cause server side copying.
The linked man page contains a full example program.