I have a input file which has a header like this:
P6\n
width\n
height\n
depth\n
and then a struct is writen, pixel*, into this file, which is going to be mapped.
So, I want to skip the header and make my mmap function return the ptr to that structure. How can I do this? with lseek perhaps? Could you please exemplify?
I will leave part of my code here:
printf("Saving header to output file\n");
if (writeImageHeader(h, fpout) == -1) {
printf("Could not write to output file\n");
return -1;
}
last_index = (int)ftell(fpout);
//printf("offset after header= %d\n",last_index);
//alloc mem space for one row (width * size of one pixel struct)
row = malloc(h->width * sizeof (pixel));
/*Create a copy of the original image to the output file, which will be inverted*/
printf("Starting work\n");
for (i = 0; i < h->height; i++) {
printf("Reading row... ");
if (getImageRow(h->width, row, fpin) == -1) {
printf("Error while reading row\n");
}
printf("Got row %d || ", (i + 1));
printf("Saving row... ");
if (writeRow(h->width, row, fpout) == -1) {
printf("Error while reading row\n");
}
printf("Done\n");
}
/*Open file descriptor of the ouput file.
* O_RDWR - Read and Write operations both permitted
* O_CREAT - Create file if it doesn't already exist
* O_TRUNC - Delete existing contents of file*/
if ((fdout = open(argv[2], O_RDWR, FILE_MODE)) < 0) {
fprintf(stderr, "Can't create %s for writing\n", argv[2]);
exit(1);
}
/*Get size of the output file*/
if (fstat(fdout, &sbuf) == -1) {
perror("Stat error ---------->\n");
exit(1);
}
//printf("Size of output file: %d\n",(int)sbuf.st_size);
/*Maps output file to memory*/
if ((data = mmap((caddr_t) 0, sbuf.st_size, PROT_READ | PROT_WRITE, MAP_SHARED, fdout, 0)) == (caddr_t) (-1)) {
perror("Error mmaping");
exit(EXIT_FAILURE);
}
As you see, right now my ppm image is mapped to char* data, but I want to skip the header and map just to the pixel* part.
Here's my code with the suggestion of using 2 pointers, a char* from mmap and another one equals that + offset.
main
c functions
header
makefile
If you read the man page for mmap, you wil find that its final parameter is off_t offset. The description:
... continuing or at most 'len' bytes to be mapped from the object described by 'fd', starting at byte offset 'offset'.
I suspect if you pass your offset in as that parameter, it will do what you want.
You can't if the amount you need to skip is less than the system page size, since offset must be a multiple of the page size on some systems.
You just need to keep 2 pointers - the pointer to the start of the mmap'd block, and the pointer to the start of the data you want inside there. As in:
unsigned char *block = mmap(0, sbuf.st_size, PROT_READ | PROT_WRITE, MAP_SHARED, fdout, 0);
unsigned char *data = block + offset;
where offset is the offset in the file to the data you want.
So, from what I understand, can I do something like this?
off_t offset_after_header = lseek(fdout, last_index, SEEK_SET);
printf("Pointer is on %d\n",(int)offset_after_header);
/*Maps output file to memory*/
if ((data = mmap((caddr_t) 0, sbuf.st_size, PROT_READ | PROT_WRITE, MAP_SHARED, fdout, offset_after_header)) == (caddr_t) (-1)) {
perror("Error mmaping");
exit(EXIT_FAILURE);
}
and, from that, I could map my file to whatever type I want, in this case the pixel*
If this is ok, what cautions should I take? For example, like those Ignacio Vazquez-Abrams said
Um, you did notice the 'offset' parameter that you are supplying with a zero? Assuming you know the absolute offset of what you want, you pass it.
Related
Hello I am trying to back up a vector by mmap.
However, I have tried msync then munmap but it doesn't work. After I write to the (char *) then munmap the file, the file has no content. The mmap file is also created with flag MAP_SHARED. Would really appreciate it if anyone can help.
//update file descriptor
if ((fd = open(filename.c_str(), O_RDWR | S_IRWXU)) < 0) { //| O_CREAT
printf("ERROR opening file %s for writing", filename.c_str());
exit(1);
}
//lseek create a file large enough
off_t i = lseek(fd, frontier_size * URL_MAX_SIZE, SEEK_SET);
if (i != frontier_size * URL_MAX_SIZE) {
cout << "failed to seek";
}
//reposition and write 3 bytes to the file else will failed to read
char buff[3] = "ta";
ssize_t kk = lseek(fd, 0, SEEK_SET);
if (kk < 0) {
cout << "failed to reposition";
}
ssize_t temp_write = write(fd, (void *)& buff, 2);
if (temp_write < 0) {
cout << "failed to write";
cout << temp_write;
}
//reposition to begining
ssize_t k = lseek(fd, 0, SEEK_SET);
if (k < 0) {
cout << "failed to reposition";
}
char * map = (char *)mmap(0, frontier_size * URL_MAX_SIZE, PROT_WRITE, MAP_SHARED, fd, 0);
if (map == MAP_FAILED) {
printf("failed mmap");
exit(1);
}
mmap_frontier = map;
//write to frontier
for (int i = 0; i < frontier.size(); ++i) {
strcpy(mmap_frontier, frontier[i].c_str());
mmap_frontier += URL_MAX_SIZE;
}
mmap_frontier -= frontier.size() * URL_MAX_SIZE;
ssize_t k = lseek(fd, 0, SEEK_SET);
if (k < 0) {
cout << "failed to reposition";
}
int sync = msync((void *)0, frontier.size() * URL_MAX_SIZE, MS_ASYNC);
if (sync < 0 ) {
cout << "failed to sync";
}
int unmap = munmap((void *)0, frontier.size() * URL_MAX_SIZE);
if (unmap < 0) {
cout << "failed to unmap";
}
There are quite a few problems with your code, and with the question:
S_IRWXU is the 3rd argument to open(), not a flag for the 2nd parameter.
mmap() won't work correctly if the file is too small. You can use ftruncte() to set the file size correctly. You tried to seek past the total size of the mapping and write a couple of bytes ("ta"), but before doing that you issued the seek lseek(fd, 0, SEEK_SET) which means the file size was set to 3 rather than mapping_size+3.
You're not backing the vector with an mmapped file, the vector has nothing to do with it, the vector uses its own memory that isn't related in any way to this mapping (please edit your question...).
You called msync() with the address (void *)0, so the actual address which needs to be synced, map, is not being synced.
Likewise, you called munmap() with the address (void *)0, so the actual address which needs to be unmapped is not being unmapped.
You called msync() with MS_ASYNC, which means there's no guarantee that the sync happens before you read the file's contents.
Here's what's working for me (error handling omitted for brevity):
unsigned frontier_size = 2;
const unsigned URL_MAX_SIZE = 100;
int fd = open("data", O_RDWR);
loff_t size = frontier_size * URL_MAX_SIZE;
ftruncate(fd, size);
char *map = (char *)mmap(0, size, PROT_WRITE, MAP_SHARED, fd, 0);
strcpy(map, "hello there");
msync(map, size, MS_SYNC);
munmap(map, size);
close(fd);
I'm trying to read from an input file and ultimately reverse the buffer it reads from and write it to an output file. For now though, I'm testing to see if a buffer I read from would even make it to the output file, and so far it isn't and I'm getting an infinite loop. The buffer should read in PAGESIZE bytes (from a call to sysconf()) and if the file output is larger than the buffer, then the buffer should be written to the output file first then be flushed and reused again to get the rest of the input until the file descriptor returns 0 for no data left. This is what I have so far:
int fdRead = open(inputFile, O_RDONLY);
if (fdRead == -1)
err_sys("Error reading input file '%s', check spelling?\n", inputFile);
int fdWrite = open(outputFile, O_WRONLY | O_CREAT | O_TRUNC, 0644); //overwrites file if it exists
if (fdWrite == -1)
err_sys("Error creating output file '%s'\n", outputFile);
while (1) {
read(fdRead, buf, size);
if (fdRead == 0)
break;
if (fdRead == -1)
err_sys("Error reading from input file '%s'\n", inputFile);
lseek(fdRead, size, SEEK_CUR);
if (fdRead == -1)
err_sys("Error reading from input file '%s'\n", inputFile);
write(fdWrite, buf, size);
if (fdWrite == -1)
err_sys("Error writing to output file '%s'\n", outputFile);
lseek(fdWrite, size, SEEK_CUR);
if (fdWrite == -1)
err_sys("Error writing to output file '%s'\n", outputFile);
memset(buf, '\0', size);
}
close(fdRead);
close(fdWrite);
I suppose that fdRead is never returning 0, and thus not exiting the loop. My question is how do I fix that?
p.s: size is the call from sysconf() that gets the PAGESIZE, e.g
size = sysconf(_SC_PAGESIZE);
And inputFile and outputFile are both char * and I've tested that they return and store good strings.
Transcribing comments into an answer.
You need to capture the return value from read() — you're ignoring it and testing whether the file descriptor is 0 or negative after the read().
So if I did something like int bytesRead then tested for if (bytesRead == 0) instead of if (fdRead == 0), then that should solve my problem?
Yes, you need something like:
int nbytes = read(fdRead, buf, size);
if (nbytes <= 0) break;
You should use the positive nbytes in the write() call; you might not get all size bytes filled by the read().
Testing the file descriptor after the read is wrong (it won't have changed under normal circumstances), and ignoring the value returned by read() is wrong, and not using the value returned by read() in the call to write() is wrong.
OK so it should be write(fdWrite, buf, nbytes)?
Yes, it should be
int obytes;
if ((obytes = write(fdWrite, buf, nbytes)) != nbytes)
{
…oops — short write …
}
You get to decide what's the appropriate response to a short write (a positive value, but not the number of bytes you expected to write). If you're writing to a socket, it might be appropriate to try writing the unwritten section of the data again (that's why obytes is used to capture the number of bytes successfully written). If you're writing to a disk file, it probably means there's no space left, so there's no point (little point) in trying again. If obytes is negative, you've had a write error; there is usually little point in trying to continue.
This all has helped out a lot and it seems to be working okay. I've only run into one other problem. I tested this on a large file (Alice in Wonderland text file) and the output file is almost the whole thing, but cuts off the last two paragraphs or so. …
You need to review why you have the lseek() operations in the code. Neither of them should be necessary, and both are dubious. I think the lseek() on fdRead() means you miss chunks of text of size bytes each; I think the lseek() on fdWrite() means you insert size null bytes into the output file.
/*Here's the code which may help for that.i havemodified in it for your need.*/
int fdRead = open(inputFile, O_RDONLY);
int Ret_Val;
if (fdRead == -1)
err_sys("Error reading input file'%s',check spelling?\n",
inputFile );
int fdWrite = open(outputFile, O_WRONLY | O_CREAT
| O_TRUNC, 0644); //overwrites file if it exists
if (fdWrite == -1)
err_sys("Error creating output file '%s'\n", outputFile);
while (1) {
Ret_Val=read(fdRead, buf, size);
if (Ret_Val == 0)
break;
if (Ret_Val == -1)
err_sys("Error reading from input file'%s'\n",inputFile);
lseek(fdRead, size, SEEK_CUR);
if (Ret_Val == -1)
err_sys("Error reading from input file'%s'\n",
inputFile );
Ret_Val=write(fdWrite, buf, size);
if (Ret_Val == -1)
err_sys("Error writing to output file'%s'\n",outputFile);
lseek(fdWrite, size, SEEK_CUR);
if (Ret_Val == -1)
err_sys("Error writing to output file'%s'\n",outputFile);
memset(buf, '\0', size);
}
close(fdRead);
close(fdWrite);
I am trying to create an empty file if it does not exists. And than map it using mmap() so, that i can pass it to my other program for writing. I am not sure which arguments for mmap are suitable for an empty file. My code works for non empty files but gives error "Invalid argument" if file is empty
Code program1 (only creates an empty file if not exists)
int i;
int fd = open("/home/sungmin/dummy_programs/dummy.txt", O_RDONLY | O_CREAT, 0777);
char *pmap;
pid_t child;
if (fd == -1)
{
perror("Error opening file for writing");
exit(EXIT_FAILURE);
}
struct stat fileInfo = {0};
if (fstat(fd, &fileInfo) == -1)
{
perror("Error getting the file size");
exit(EXIT_FAILURE);
}
/*if (fileInfo.st_size == 0)
{
fprintf(stderr, "Error: File is empty, nothing to do\n");
exit(EXIT_FAILURE);
}*/
pmap = mmap(0, fileInfo.st_size, PROT_READ | PROT_EXEC , MAP_ANONYMOUS, fd, 0);
if (pmap == MAP_FAILED)
{
close(fd);
perror("Error mmapping the file");
exit(EXIT_FAILURE);
}
/* Calling fork function */
if((child=fork())==0){
printf("Iam Child process\n\n");
static char *argv[]={"This is some sample text. I need to write this text in my dummy file.","/home/sungmin/dummy_programs/dummy.txt",NULL};
execv("/home/sungmin/dummy_programs/pro2",argv);
exit(127);
}
else {
printf("Iam parent, waiting for child process to exit\n\n");
waitpid(child,0,0);
printf("Existing parent\n\n");
}
/* Don't forget to free the mmapped memory*/
if (munmap(pmap, fileInfo.st_size) == -1)
{
close(fd);
perror("Error un-mmapping the file");
exit(EXIT_FAILURE);
}
/* Un-mmaping doesn't close the file, so we still need to do that.*/
close(fd);
Code program2 (opens same file as program1 and writes text passed by program1)
size_t i;
int fd;
char *pmap;
pid_t child;
struct stat fileInfo = {0};
const char *text = argv[0];
fd = open(argv[1], O_RDWR | O_CREAT | O_TRUNC, (mode_t)0600);
if (fd == -1)
{
perror("Error opening file for writing");
exit(EXIT_FAILURE);
}
size_t textsize = strlen(text) + 1; // + \0 null character
if (lseek(fd, textsize-1, SEEK_SET) == -1)
{
close(fd);
perror("Error calling lseek() to 'stretch' the file");
exit(EXIT_FAILURE);
}
if (write(fd, "", 1) == -1)
{
close(fd);
perror("Error writing last byte of the file");
exit(EXIT_FAILURE);
}
pmap = mmap(0, textsize, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
if (pmap == MAP_FAILED)
{
close(fd);
perror("Error mmapping the file");
exit(EXIT_FAILURE);
}
/* Writting users text to file */
for (i = 0; i < textsize; i++)
{
pmap[i] = text[i];
}
// Write it now to disk
if (msync(pmap, textsize, MS_SYNC) == -1)
{
perror("Could not sync the file to disk");
}
/* Don't forget to free the mmapped memory*/
if (munmap(pmap, textsize) == -1)
{
close(fd);
perror("Error un-mmapping the file");
exit(EXIT_FAILURE);
}
/* Un-mmaping doesn't close the file, so we still need to do that.*/
close(fd);
You need to use truncate to extend the file length after creating it before mapping it.
Yes, the function name sounds wrong, but truncate can actually set the file length to any number. Be sure to use a multiple of 4K for best results.
Then, if you want to keep the mapping open to see data between Program 1 and 2, you need to get rid of ANONYMOUS and map with MAP_SHARED in Program 1. A mapping that isn't shared will not show changes made by other programs. Or it might, if it has to reload from disk. It's weird, don't mix SHARED and not-SHARED mappings.
Once you've changed Program 1 to use truncate, take that lseek and write code out of Program 2. The file will already have been created and extended by Program 1.
I created a process which calls mmap with MAP_SHARED flag set,when i attempt to copy a string to that address i receive Bus error core dumped,could some one please explain the reason behind it and how to fix it. Following is my code
int main()
{
int fd=0;
char* ret = NULL;
void *map_addr = NULL;
fd = open("./shared_file.txt", O_RDWR, S_IRUSR | S_IWUSR);
if(fd == -1) {
printf("errno = %d\n",errno);
printf("Aborting process1###########\n");
abort();
}
map_addr = mmap(NULL, 5*sizeof(int), PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0);
if(map_addr == MAP_FAILED) {
printf("mmap failed error no =%d\n",errno);
close(fd);
return -1;
}
printf("map_addr = %p#################\n",(int*)map_addr);
printf("processid = %d#################\n",(int)getpid());
ret = strcpy((char*)map_addr,"Stack Overflow");
if(ret == (char*)map_addr)
printf("strcpy success\n");
/*if(msync(map_addr, sizeof(int), MS_SYNC))
printf("msync failed errno = %d\n",errno);*/
close(fd);
sleep(120);
return (0);
}
The cause of a bus error is usually an attempt to dereference a pointer that has not been initialized properly and contains junk data that are not accessible in a multiple of 4 or 1 or as related to datatype sizes.
First you should check if the shared_file.txt file size is >= 20 bytes(assuming sizeof int is 4 bytes) as specified in the mmap() length argument(where you put 5*(sizeof(int))) in the line below:
map_addr = mmap(NULL, 5*sizeof(int), PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0);
If file size is less than 20 bytes, you could use fallocate call to pre allocate the memory.
If shared_file.txt file size is <= 20 bytes and then you ask mmap to map 20 bytes, it could cause a bus error when you write beyond the actual no. of bytes available in file, because that would be access to an undefined portion of a memory. MMAP_FAILED will not be returned in this case, during memory initialization.
A quick check is to see if you can write a single character in the mmap char* pointer. If you can't( you will get a SIGBUS), that means file size is zero.
I'd like to give a try at copying the contents of a file over to another one by using memory mapped I/O in Linux via mmap(). The intention is to check by myself if that's better than using fread() and fwrite() and how would it deal with big files (like couple of GiBs for example, since the file is read whole I want to know if I need to have such amount of memory for it).
This is the code I'm working with right now:
// Open original file descriptor:
int orig_fd = open(argv[1], O_RDONLY);
// Check if it was really opened:
if (orig_fd == -1) {
fprintf(stderr, "ERROR: File %s couldn't be opened:\n", argv[1]);
fprintf(stderr, "%d - %s\n", errno, strerror(errno));
exit(EX_NOINPUT);
}
// Idem for the destination file:
int dest_fd = open(argv[2], O_WRONLY | O_CREAT | O_TRUNC, 0644);
// Check if it was really opened:
if (dest_fd == -1) {
fprintf(stderr, "ERROR: File %s couldn't be opened:\n", argv[2]);
fprintf(stderr, "%d - %s\n", errno, strerror(errno));
// Close original file descriptor too:
close(orig_fd);
exit(EX_CANTCREAT);
}
// Acquire file size:
struct stat info = {0};
if (fstat(orig_fd, &info)) {
fprintf(stderr, "ERROR: Couldn't get info on %s:\n", argv[1]);
fprintf(stderr, "%d - %s\n", errno, strerror(errno));
// Close file descriptors:
close(orig_fd);
close(dest_fd);
exit(EX_IOERR);
}
// Set destination file size:
if (ftruncate(dest_fd, info.st_size)) {
fprintf(stderr, "ERROR: Unable to set %s file size:\n", argv[2]);
fprintf(stderr, "%d - %s\n", errno, strerror(errno));
// Close file descriptors:
close(orig_fd);
close(dest_fd);
exit(EX_IOERR);
}
// Map original file and close its descriptor:
char *orig = mmap(NULL, info.st_size, PROT_READ, MAP_PRIVATE, orig_fd, 0);
if (orig == MAP_FAILED) {
fprintf(stderr, "ERROR: Mapping of %s failed:\n", argv[1]);
fprintf(stderr, "%d - %s\n", errno, strerror(errno));
// Close file descriptors:
close(orig_fd);
close(dest_fd);
exit(EX_IOERR);
}
close(orig_fd);
// Map destination file and close its descriptor:
char *dest = mmap(NULL, info.st_size, PROT_WRITE, MAP_SHARED, dest_fd, 0);
if (dest == MAP_FAILED) {
fprintf(stderr, "ERROR: Mapping of %s failed:\n", argv[2]);
fprintf(stderr, "%d - %s\n", errno, strerror(errno));
// Close file descriptors and unmap first file:
munmap(orig, info.st_size);
close(dest_fd);
exit(EX_IOERR);
}
close(dest_fd);
// Copy file contents:
int i = info.st_size;
char *read_ptr = orig, *write_ptr = dest;
while (--i) {
*write_ptr++ = *read_ptr++;
}
// Unmap files:
munmap(orig, info.st_size);
munmap(dest, info.st_size);
I think it may be a way of doing it but I keep getting an error trying to map the destination file, concretely code 13 (permission denied).
I don't have a clue on why is it failing, I can write to that file since the file gets created and all and the file I'm trying to copy is just a couple of KiBs in size.
Can anybody spot the problem? How come I had permission to map the original file but not the destination one?
NOTE: If anyone is to use the loop to copy bytes posted in the question instead of memcpy for example, the loop condition should be i-- instead to copy all contents. Thanks to jxh for spotting that.
From the mmap() man page:
EACCES A file descriptor refers to a non-regular file. Or MAP_PRIVATE
was requested, but fd is not open for reading. Or MAP_SHARED
was requested and PROT_WRITE is set, but fd is not open in
read/write (O_RDWR) mode. Or PROT_WRITE is set, but the file is
append-only.
You are opening your destination file with O_WRONLY. Use O_RDWR instead.
Also, you should use memcpy to copy the memory rather than using your own loop:
memcpy(dest, orig, info.st_size);
Your loop has an off by 1 bug.
This works for me. Note that I had to open the destination O_RDWR. I suspect the kernel attempts to map whole pages from the file into memory (reading it) because you're updating it a byte or word at a time, and that might not change the whole page.
A couple of other points:
You don't need to close and unmap stuff on error if you're just going to exit.
Use memcpy and don't write your own byte-copying loop. Memcpy will be a lot better optimised in general. (Though it's not always the absolute best.)
You might want to read the source code to FreeBSD's "cp" utility. Take a look here and search for the use of mmap. http://svnweb.freebsd.org/base/stable/9/bin/cp/utils.c?revision=225736&view=markup
#include <stdio.h>
#include <stdlib.h>
#include <fcntl.h>
#include <unistd.h>
#include <sys/mman.h>
#include <string.h>
#include <sys/stat.h>
int main(int argc, char *argv[])
{
int s, d;
struct stat st;
void *sp, *dp;
s = open(argv[1], O_RDONLY);
if (s == -1) {
perror("open source");
exit(1);
}
d = open(argv[2], O_RDWR | O_CREAT | O_TRUNC, 0644);
if (d == -1) {
perror("open destintation");
exit(1);
}
if (fstat(s, &st)) {
perror("stat source");
exit(1);
}
if (ftruncate(d, st.st_size)) {
perror("truncate destination");
exit(1);
}
sp = mmap(NULL, st.st_size, PROT_READ, MAP_SHARED, s, 0);
if (sp == MAP_FAILED) {
perror("map source");
exit(1);
}
dp = mmap(NULL, st.st_size, PROT_WRITE | PROT_READ, MAP_SHARED, d, 0);
if (dp == MAP_FAILED) {
perror("map destintation");
exit(1);
}
memcpy(dp, sp, st.st_size);
return 0;
}
Original File: O_RDONLY open, MAP_PRIVATE mmap
destination file: O_WRONLY open, MAP_SHARED mmap
You need to open with O_RDWR flag for using MAP_SHARED.
Don't you actually need to do MAP_FILE | MAP_SHARED ?