This question already has answers here:
How to change characters in a text file using C's mmap()?
(2 answers)
Closed 2 years ago.
I am trying to read and write a struct using mmap, however the changes I do after the mmap are not being persisted to the disk or are not being loaded correctly.
Here's my code sample, when run the first time the file is created and the prints show the correct data, on the second time when the file already exists the struct is empty, filled with 0's. So it looks like the changes were not written to the file but I am having trouble figuring out why.
#include <stddef.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/mman.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <errno.h>
#include <unistd.h>
typedef struct {
int age;
char name[128];
} person;
int main(int argc, char *argv[argc]){
char filename [] = "data.person";
int fd = open(filename, O_RDWR | O_CREAT , S_IRWXU);
if (fd == -1) {
printf("Failed to create version vector file, error is '%s'", strerror(errno));
exit(1);
}
struct stat st;
fstat(fd, &st);
person *p;
if (st.st_size == 0) {
ftruncate(fd, sizeof(person));
p = (person *) mmap(0, sizeof(person), PROT_READ | PROT_WRITE, MAP_PRIVATE, fd, 0);
strcpy(p[0].name, "Hello");
p[0].age = 10;
msync(p, sizeof(person), MS_SYNC);
}else{
p = (person *) mmap(0, st.st_size, PROT_READ, MAP_PRIVATE, fd, 0);
if( p == MAP_FAILED){
printf("mmap failed with error '%s'\n", strerror(errno));
exit(0);
}
}
printf("%s %d\n", p->name, p->age);
munmap(p, sizeof(person));
close(fd);
}
My OS is manjaro 20 and the gcc version is 10.1
Do not use MAP_PRIVATE because:
Create a private copy-on-write mapping. Updates to the
mapping are not visible to other processes mapping the same
file, and are not carried through to the underlying file. It
is unspecified whether changes made to the file after the
mmap() call are visible in the mapped region.
I'm trying to share a text file between forked processes on my Ubuntu x86_64: the file will not be absurdly large, since strings will be written only if there is not already another identical string in the file; strings will be hostnames of visited websites, so I'll assume no more than 255 bytes for each hostname.
When it is a process' turn to write in shared object, it is OK; once all the processes wrote in shared object, msync should make the writing effective on the disk, but the mapped.txt file created only contain one string from arrayString, i.e. the string the last process wrote in shared object.
Here's the code:
#include <stdio.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <sys/mman.h>
#include <unistd.h>
#include <stdlib.h>
#include <semaphore.h>
#include <string.h>
// first forked process will write "first" in file, and so on
const char *arrayString[] = {
"first",
"second",
"third"
};
int main(void) {
int index;
int children = 3;
const char *filepath = "mapped.txt";
sem_t *sem;
sem = sem_open("semaphore", O_CREAT | O_EXCL, 0644, 1);
sem_unlink("semaphore");
int fd;
fd = open(filepath, O_RDWR | O_CREAT, 0644);
if (fd < 0) {
perror("open:");
return EXIT_FAILURE;
}
char *data;
data = (char *)mmap(NULL, getpagesize(), PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
if (data == MAP_FAILED) {
close(fd);
perror("mmap:");
return EXIT_FAILURE;
}
for (index=0; index<children; index++) {
if (fork() == 0) {
sem_wait(sem);
size_t textsize = strlen(arrayString[index])+1;
if (ftruncate(fd, sizeof(textsize)) == -1) {
perror("ftruncate:");
return EXIT_FAILURE;
}
for (size_t i = 0; i < textsize; i++) {
printf("%d Writing character %c at %zu\n", getpid(), arrayString[index][i], i);
data[i] = arrayString[index][i];
}
printf("%d wrote ", getpid());
for (size_t i = 0; i < textsize; i++) {
printf("%c", data[i]);
}
printf("\n");
if (msync(data, textsize, MS_SYNC) == -1) {
perror("Could not sync the file to disk");
}
sem_post(sem);
_exit(EXIT_SUCCESS);
}
}
close(fd);
return EXIT_SUCCESS;
}
This is one possible output of the code above for three child processes (this is fine):
20373 Writing character s at 0
20373 Writing character e at 1
20373 Writing character c at 2
20373 Writing character o at 3
20373 Writing character n at 4
20373 Writing character d at 5
20373 Writing character at 6
20373 wrote second
20374 Writing character t at 0
20374 Writing character h at 1
20374 Writing character i at 2
20374 Writing character r at 3
20374 Writing character d at 4
20374 Writing character at 5
20374 wrote third
20372 Writing character f at 0
20372 Writing character i at 1
20372 Writing character r at 2
20372 Writing character s at 3
20372 Writing character t at 4
20372 Writing character at 5
20372 wrote first
And here's the content of mapped.txt (this is bad):
first^#^#^#
I expected:
second
third
first
but all I get is only the string of the last process, with those strange symbols. I'd like to keep this file persistent in memory, but because of the I/O slowness, I'm trying to use memory mapping.
Any idea why my file only contains the string written by the last process accessing the shared file?
Edit: I think I get it, it seems to work now: I hope it will be of help to someone. Compiled with g++ -g -o mapthis mapthis.cpp -lrt -pthread. Beware that some error checking are missing, like for fsync, snprintf and lseek.
#include <stdio.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <sys/mman.h>
#include <unistd.h>
#include <stdlib.h>
#include <semaphore.h>
#include <time.h>
#include <string.h>
#include <sys/types.h>
#include <sys/wait.h>
const char *arrayString[] = {
"www.facebook.com",
"www.google.com",
"www.cnn.com",
"www.speechrepository.com",
"www.youtube.com",
"www.facebook.com",
"www.google.com",
"www.cnn.com",
"www.speechrepository.com",
"www.youtube.com",
"www.facebook.com",
"www.google.com",
"www.cnn.com",
"www.speechrepository.com",
"www.youtube.com"
};
int main(void) {
int index;
int children = sizeof(arrayString) / sizeof(char*);;
const char *filepath = "mapped.txt";
sem_t *sem;
char *data;
struct stat filestats;
sem = sem_open("semaphore", O_CREAT | O_EXCL, 0644, 1);
sem_unlink("semaphore");
int fd;
fd = open(filepath, O_RDWR | O_CREAT, 0644);
if (fd < 0) {
perror("open:");
return EXIT_FAILURE;
}
if (fstat(fd, &filestats) < 0) {
close(fd);
perror("fstat:");
return EXIT_FAILURE;
}
data = (char *)mmap(NULL, filestats.st_size ? filestats.st_size : 1, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
if (data == MAP_FAILED) {
close(fd);
perror("first map:");
return EXIT_FAILURE;
}
for (index=0; index<children; index++) {
sleep(1);
pid_t pid = fork();
if (pid == 0) {
int nw = 0;
int hostnameSize = 0;
const size_t origsize = filestats.st_size;
char *hostPos = NULL;
char *numPos = NULL;
char *backslashPos = NULL;
char tempBuff[64];
memset((char *)tempBuff, 0, sizeof(tempBuff));
sem_wait(sem);
// remap to current file size if it changed
fstat(fd, &filestats);
// file empty, just insert
if (filestats.st_size == 0) {
nw = snprintf(tempBuff, sizeof(tempBuff), "%s %010lu\n", arrayString[index], (unsigned long)time(NULL));
write(fd, tempBuff, nw);
fsync(fd);
}
else {
// file not empty, let's look for string
hostPos = strstr(data, arrayString[index]);
if (hostPos) {
// string is already inserted, search for offset of number of seconds
lseek(fd, hostPos-data, SEEK_SET);
numPos = strchr(hostPos, ' ')+1;
backslashPos = strchr(numPos, '\n');
long unsigned before = atoi(numPos);
long unsigned now = (unsigned long)time(NULL);
long unsigned difference = now - before;
printf("%s visited %ld seconds ago (%ld - %ld)\n",
arrayString[index], difference, now, before);
nw = snprintf(tempBuff, backslashPos-hostPos+1, "%s %010lu", arrayString[index], now);
write(fd, tempBuff, nw);
write(fd, "\n", 1);
fsync(fd);
}
else {
data = (char *)mremap(data, origsize, filestats.st_size, MREMAP_MAYMOVE);
if (data == MAP_FAILED) {
close(fd);
sem_post(sem);
perror("mmap:");
_exit(EXIT_FAILURE);
}
lseek(fd, 0, SEEK_END);
nw = snprintf(tempBuff, sizeof(tempBuff), "%s %010lu\n", arrayString[index], (unsigned long)time(NULL));
write(fd, tempBuff, nw);
fsync(fd);
}
}
munmap(data, filestats.st_size);
close(fd);
sem_post(sem);
_exit(EXIT_SUCCESS);
}
else if (pid > 0) {
wait(NULL);
}
}
munmap(data, filestats.st_size);
close(fd);
return EXIT_SUCCESS;
}
This line is problematic:
if (ftruncate(fd, sizeof(textsize)) == -1) {
textsize is a size_t, and taking its sizeof is just going to get 4 or 8 (on 32 and 64 bit systems). Looks like you're on a 64 bit system, so you're unconditionally truncating the file to 8 bytes in this case before every write. The "strange symbols" are just how your editor displays NUL/zero bytes. Even if you used ftruncate(fd, textsize), you'd still truncate down to just the string you're about to write, overwriting any data other children may have written; I doubt you want to ftruncate at all here.
For continual appends from separate processes (where they can't share information about the size or offset of the data they're adding), memory mapping just doesn't make sense; why aren't you just having each of them take the lock, lseek to end of file, then call write? You could still use memory mappings for the duplicate checking (some of it without locking), it would just be a bit different. Something like this:
int main(void) {
struct stat filestats;
int index;
int children = 3;
const char *filepath = "mapped.txt";
sem_t *sem;
char *data;
sem = sem_open("semaphore", O_CREAT | O_EXCL, 0644, 1);
sem_unlink("semaphore");
int fd;
fd = open(filepath, O_RDWR | O_CREAT, 0644);
if (fd < 0) {
perror("open:");
return EXIT_FAILURE;
}
// Mostly just to ensure it's mappable, we map the current size of the file
// If the file might already have values, and many child workers won't add
// to it, this might save some mapping work in the children; you could
// just map in the children when needed though
if (fstat(fd, &filestats) != 0) {
close(fd);
perror("fstat:");
return EXIT_FAILURE;
}
data = mmap(NULL, filestats.st_size, PROT_READ, MAP_SHARED, fd, 0);
if (data == MAP_FAILED) {
close(fd);
perror("mmap:");
return EXIT_FAILURE;
}
for (index=0; index<children; index++) {
if (fork() == 0) {
const size_t origsize = filestats.st_size;
sem_wait(sem);
// remap to current file size if it changed
// If you're not on Linux, you'd just have to mmap from scratch
// since mremap isn't standard
fstat(fd, &filestats);
if (origsize != filestats.st_size) {
data = mremap(data, origsize, filestats.st_size, MREMAP_MAYMOVE);
if (data == MAP_FAILED) {
close(fd);
sem_post(sem);
perror("mmap:");
_exit(EXIT_FAILURE);
}
}
// Not safe to use strstr since mapping might not end with NUL byte
// You'd need to workaround this, or implement a your own memstr-like function
if (!memstr(data, arrayString[index])) {
// Move fd to end of file, so we append new data
lseek(fd, 0, SEEK_END);
write(fd, arrayString[index], strlen(arrayString[index]));
write(fd, "\n", 1);
fsync(fd);
}
munmap(data, filestats.st_size);
close(fd);
sem_post(sem);
_exit(EXIT_SUCCESS);
}
}
munmap(data, filestats.st_size);
close(fd);
return EXIT_SUCCESS;
}
That memstr I referenced would need to be hand-implemented (or you'd need to do terrible things like ensure the file always had a NUL byte at the end so you could use strstr on it); you can get some tips on that here.
You're writing all the strings at offset 0 of the file, each over the top of the previous. The core of your loop should be something like
struct stat status;
fstat(fd, &status);
size_t cursize = status.st_size;
ftruncate(fd, cursize + textsize);
for (size_t i = 0; i < textsize; i++) {
data[cursize + i] = arrayString[index][i];
}
I am trying to create a shared memory area using examples and documentation I found online. My goal is IPC , so I can make different processes talk to each other.
This my C file
#include <stdio.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>
#include <sys/mman.h>
#include <errno.h>
int main (int argc, char *argv[])
{
struct stat sb;
off_t len;
char *p;
int fd;
fd = shm_open("test", O_RDWR | O_CREAT); //,S_IRUSR | S_IWUSR);
if (fd == -1) {
perror("open");
return 1;
}
if (fstat(fd, &sb)==-1){
perror("fstat");
return 1;
}
/*if (!S_ISREG(sb.st_mode)){
fprintf(stderr, "%s is not a file\n",fileName);
return 1;
}*/
p = mmap(0, sb.st_size, PROT_WRITE, MAP_SHARED, fd, 0);
if (p == MAP_FAILED){
perror("mmap");
return 1;
}
if (close(fd)==-1) {
perror("close");
return 1;
}
for (len = 0; len < sb.st_size; len++) {
putchar(p[len]);
}
if (munmap(p, sb.st_size) == -1) {
perror("munmao");
return 1;
}
fprintf(stderr,"\n");
return 0;
}
The problem is that I am getting a mmap: Invalid argument. I assume something is wrong with fd but have no clue how to fix it, any help would be appreciated. I am on Yosemite using latest XCODE .
You need to extend the size of the shared memory mapping, at least the first time when you create it. Right now its size is 0, and mmap is not going to allow you to make a zero length mapping.
So instead of your fstat() call, do e.g.:
size_t len = 4096;
if (ftruncate(fd, len) == -1) {
perror("ftruncate");
return 1;
}
And pass this len to mmap().
Your addr parameter is set to 0, which might be reserved. Did you mean to use NULL? This would be different than 0.
I have this struct in C:
struct first {
struct list *buf;
struct pointers *ptr;
};
A function to create a shared memory segment:
void * create_shared_memory(char *name, int size){
int *ptr;
int ret;
int fd = shm_open (name, O_RDWR | O_CREAT | O_TRUNC, S_IRUSR | S_IWUSR);
if (fd == -1) {
perror ("shm_open error!");
exit (1);
}
ret = ftruncate (fd, sizeof (size));
if (ret == -1) {
perror ("ftruncate error!");
exit (2);
}
ptr = mmap(0, sizeof (size), PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
if (ptr == MAP_FAILED) {
perror ("shm-mmap error!");
exit (3);
}
}
and a function to create a shared memory segment for that structure:
void shared_memory_structure(){
create_shared_memory("struct", sizeof(struct first));
}
However I get an error. I found the problem was that the pointers inside the structure are not being pointed to the shared memory segment I just created. How can I do this ?
try out this code first. it's a demo code taken from http://blog.csdn.net/liuzhanchen1987/article/details/7455208 , which is in Chinese:
#include <sys/mman.h>
#include <sys/types.h>
#include <fcntl.h>
#include <unistd.h>
typedef struct{
char name[4];
int age;
}people;
int
main(int argc, char** argv)
{
int i;
people *p_map;
char temp;
p_map=(people*)mmap(NULL,sizeof(people)*10,PROT_READ|PROT_WRITE, MAP_SHARED|MAP_ANONYMOUS,-1,0);
if(fork() == 0)
{
sleep(2);
for(i = 0;i<5;i++)
printf("child read: the %d people's age is %d\n",i+1,(*(p_map+i)).age);
(*p_map).age = 100;
munmap(p_map,sizeof(people)*10);
exit();
}
temp = 'a';
for(i = 0;i<5;i++)
{
temp += 1;
memcpy((*(p_map+i)).name, &temp,2);
(*(p_map+i)).age=20+i;
}
sleep(5);
printf( "parent read: the first people,s age is %d\n",(*p_map).age );
printf("umap\n");
munmap( p_map,sizeof(people)*10 );
printf( "umap ok\n" );
return 0;
}
expected result:
child read: the 1 people's age is 20
child read: the 2 people's age is 21
child read: the 3 people's age is 22
child read: the 4 people's age is 23
child read: the 5 people's age is 24
parent read: the first people,s age is 100
umap
umap ok
in which
mmap(NULL,size,PROT_READ|PROT_WRITE, MAP_SHARED|MAP_ANONYMOUS,-1,0);
is all what you need to do before forking(and its return value checking code), and its return value will be the allocated pages starting address(if it's valid).
if all processes are forked. it's very handy to just use mmap to allocate anonymous pages, and additionally, there is no side effect after the processes get collected.
if you use shm_open, then in somewhere there will be a share memory object gets created, and it will be the side effect remains in your system, even after all your processes get collected. however, it's necessary when you plan to have two irrelevant processes talk to each other.
The following C code illustrates a problem I'm seeing on Linux 2.6.30.5-43.fc11.x86_64:
#include <sys/types.h>
#include <sys/stat.h>
#include <sys/mman.h>
#include <fcntl.h>
#include <unistd.h>
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
int main() {
char buf[1024];
void *base;
int fd;
size_t pagesz = sysconf(_SC_PAGE_SIZE);
fd = open("<some file, at least 4*pagesz in length>", O_RDONLY);
if (fd < 0) {
perror("open");
return 1;
}
base = mmap(0, 4*pagesz, PROT_READ, MAP_SHARED, fd, 0);
if (base < 0) {
perror("mmap");
close(fd);
return 1;
}
memcpy(buf, (char*)base + 2*pagesz, 1024);
if (remap_file_pages(base, pagesz, 0, 2, 0) < 0) {
perror("remap_file_pages");
munmap(base, 4*pagesz);
close(fd);
return 1;
}
printf("%d\n", memcmp(buf, base, 1024));
munmap(base, 4*pagesz);
close(fd);
return 0;
}
This always fails with remap_file_pages() returning -1 and errno set to EINVAL. Looking at the kernel source I can see all the conditions in remap_file_pages() where it might fail but none of them seem to apply to my example. What's going on?
It's caused by the file being opened O_RDONLY. If you change the open mode to O_RDWR, it works (even if the mmap() still specifies just PROT_READ).
This code in do_mmap_pgoff is the root cause - it only marks the vma as VM_SHARED if the file was opened for writing:
vm_flags |= VM_SHARED | VM_MAYSHARE;
if (!(file->f_mode & FMODE_WRITE))
vm_flags &= ~(VM_MAYWRITE | VM_SHARED);
So in remap_file_pages(), you fail on the first check:
if (!vma || !(vma->vm_flags & VM_SHARED))
goto out;