how do i chunk a 1000MB file using mmapdemo - c

I'm following a tutorial from BeeJ's website and i've learnt quite a bit of it.
I'm in that stage of experimentation and want to somehow build upon this code.
In short, say the user types an address: 1050576 for a file that's roughly 1000MB, the program (shown below) should map the chunk in the file data up to a position where 1050576 falls between chunk beginning and its length.
When i run the code, it says the max address size is 924 and so i can't really chunk a file that's 1000mb.
Could someone help explain / provide an example on how i can tackle this?
/*
** mmapdemo.c -- demonstrates memory mapped files lamely.
*/
#include <stdio.h>
#include <stdlib.h>
#include <fcntl.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/mman.h>
#include <sys/stat.h>
#include <errno.h>
int main(int argc, char *argv[])
{
int fd, offset;
char *data;
struct stat sbuf;
if (argc != 2) {
fprintf(stderr, "usage: mmapdemo offset\n");
exit(1);
}
if ((fd = open("mmapdemo.c", O_RDONLY)) == -1) {
perror("open");
exit(1);
}
if (stat("mmapdemo.c", &sbuf) == -1) {
perror("stat");
exit(1);
}
offset = atoi(argv[1]);
if (offset < 0 || offset > sbuf.st_size-1) {
fprintf(stderr, "mmapdemo: offset must be in the range 0-%ld\n", sbuf.st_size-1);
exit(1);
}
if ((data = mmap((caddr_t)0, sbuf.st_size, PROT_READ, MAP_SHARED, fd, 0)) == (caddr_t)(-1)) {
perror("mmap");
exit(1);
}
printf("byte at offset %d is '%c'\n", offset, data[offset]);
return 0;
}

Related

Is the "mmap tutorial" incorrect, or does GCC miscompile it?

This mmap tutorial from 15 years ago ranks high in Google searches, but it actually runs subtly incorrectly on my Linux system.
mmap_write.c:
#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <unistd.h>
#include <fcntl.h>
#include <sys/mman.h>
#define FILEPATH "/tmp/mmapped.bin"
#define NUMINTS (1000)
#define FILESIZE (NUMINTS * sizeof(int))
int main(int argc, char *argv[])
{
int i;
int fd;
int result;
int *map; /* mmapped array of int's */
/* Open a file for writing.
* - Creating the file if it doesn't exist.
* - Truncating it to 0 size if it already exists. (not really needed)
*
* Note: "O_WRONLY" mode is not sufficient when mmaping.
*/
fd = open(FILEPATH, O_RDWR | O_CREAT | O_TRUNC, (mode_t)0600);
if (fd == -1) {
perror("Error opening file for writing");
exit(EXIT_FAILURE);
}
/* Stretch the file size to the size of the (mmapped) array of ints
*/
result = lseek(fd, FILESIZE-1, SEEK_SET);
if (result == -1) {
close(fd);
perror("Error calling lseek() to 'stretch' the file");
exit(EXIT_FAILURE);
}
/* Something needs to be written at the end of the file to
* have the file actually have the new size.
* Just writing an empty string at the current file position will do.
*
* Note:
* - The current position in the file is at the end of the stretched
* file due to the call to lseek().
* - An empty string is actually a single '\0' character, so a zero-byte
* will be written at the last byte of the file.
*/
result = write(fd, "", 1);
if (result != 1) {
close(fd);
perror("Error writing last byte of the file");
exit(EXIT_FAILURE);
}
/* Now the file is ready to be mmapped.
*/
map = mmap(0, FILESIZE, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
if (map == MAP_FAILED) {
close(fd);
perror("Error mmapping the file");
exit(EXIT_FAILURE);
}
/* Now write int's to the file as if it were memory (an array of ints).
*/
for (i = 1; i <=NUMINTS; ++i) {
map[i] = 2 * i;
}
/* Don't forget to free the mmapped memory
*/
if (munmap(map, FILESIZE) == -1) {
perror("Error un-mmapping the file");
/* Decide here whether to close(fd) and exit() or not. Depends... */
}
/* Un-mmaping doesn't close the file, so we still need to do that.
*/
close(fd);
return 0;
}
mmap_read.c:
#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <unistd.h>
#include <fcntl.h>
#include <sys/mman.h>
#define FILEPATH "/tmp/mmapped.bin"
#define NUMINTS (1000)
#define FILESIZE (NUMINTS * sizeof(int))
int main(int argc, char *argv[])
{
int i;
int fd;
int *map; /* mmapped array of int's */
fd = open(FILEPATH, O_RDONLY);
if (fd == -1) {
perror("Error opening file for reading");
exit(EXIT_FAILURE);
}
map = mmap(0, FILESIZE, PROT_READ, MAP_SHARED, fd, 0);
if (map == MAP_FAILED) {
close(fd);
perror("Error mmapping the file");
exit(EXIT_FAILURE);
}
/* Read the file int-by-int from the mmap
*/
for (i = 1; i <=NUMINTS; ++i) {
printf("%d: %d\n", i, map[i]);
}
if (munmap(map, FILESIZE) == -1) {
perror("Error un-mmapping the file");
}
close(fd);
return 0;
}
If the file does not already exist, the output of mmap_read is
...
998: 1996
999: 1998
1000: 2000
But if it does, the output is
...
998: 1996
999: 1998
1000: 0
Should the author have flushed the write? Or is GCC miscompiling the code?
Edit: I noticed that it's the prior existence or non-existence of the file that makes a difference, not the compilation flag.
You are starting at the second element, and writing 2000 after the end of the map.
for (i = 1; i <=NUMINTS; ++i) {
map[i] = 2 * i;
}
should be
for (i = 0; i < NUMINTS; ++i) {
map[i] = 2 * ( i + 1 );
}
Demo
It's not a buffering issue. write is a system call, so the data passed to the OS directly. It doesn't mean the data has been written to disk when write returns, but it is in the OS's hands, so it's as if it was on disk as far as OS functions are concerned, including its memory-mapping functionality.
In C indexes are from zero. Writing and reading index 1000 you invoke undefined behaviour
Change to in the write.:
for (i = 1; i <=NUMINTS; ++i) {
map[i - 1] = 2 * i;
}
and reading to:
for (i = 1; i <=NUMINTS; ++i) {
printf("%d: %d\n", i, map[i-1]);
}

got bus error when writing to mmap'ed file

Trying to use mmap to write to a file. Unfortunately the first write in the loop map[i] = i; will cause a bus error. Not sure why.
The PC runs Ubuntu 14.04 and the file /tmp/mmapped.bin has 12 bytes and the program is invoked with ./a.out 3.
Thanks
#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <unistd.h>
#include <fcntl.h>
#include <sys/mman.h>
#define FILEPATH "/tmp/mmapped.bin"
//#define NUMINTS (1000)
#define FILESIZE 0x400000000
int main(int argc, char *argv[])
{
int i;
int fd;
int *map; /* mmapped array of int's */
int size = atoi(argv[1]);
fd = open(FILEPATH, O_RDWR| O_CREAT | O_TRUNC);
if (fd == -1) {
perror("Error opening file for reading");
exit(EXIT_FAILURE);
}
map = mmap(0, 4 * size, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0);
if (map == MAP_FAILED) {
close(fd);
perror("Error mmapping the file");
exit(EXIT_FAILURE);
}
for (i = 1; i <= size; ++i) {
map[i] = i;
}
if (munmap(map, FILESIZE) == -1) {
perror("Error un-mmapping the file");
}
close(fd);
return 0;
}
In c you need to start at index 0. Because it will simply increment the pointer by the amount i and then dereference it. Your code dereferences the pointer beyond the allowed bound.
It should be,
for (i = 0; i < size; ++i) {
map[i] = i;
}
because it's equivalent to
for (i = 0; i < size; ++i) {
*(map + i) = i;
}
Also, use
map = mmap(0, size * sizeof *map, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
to ensure that enough space is allocated and that *(map + i) will be inside bounds. Don't use magic numbers.
According to the mmap man page a bus error (SIGBUS) happens when you read/write outside the bounds of the file.
The length of the mapping is separate from the length of the file. If your file is newly created its size will be 0, even if you specify a length with mmap. Resize the file with ftruncate after opening it.

How to use shm_open with mmap properly

I am trying to create a shared memory area using examples and documentation I found online. My goal is IPC , so I can make different processes talk to each other.
This my C file
#include <stdio.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>
#include <sys/mman.h>
#include <errno.h>
int main (int argc, char *argv[])
{
struct stat sb;
off_t len;
char *p;
int fd;
fd = shm_open("test", O_RDWR | O_CREAT); //,S_IRUSR | S_IWUSR);
if (fd == -1) {
perror("open");
return 1;
}
if (fstat(fd, &sb)==-1){
perror("fstat");
return 1;
}
/*if (!S_ISREG(sb.st_mode)){
fprintf(stderr, "%s is not a file\n",fileName);
return 1;
}*/
p = mmap(0, sb.st_size, PROT_WRITE, MAP_SHARED, fd, 0);
if (p == MAP_FAILED){
perror("mmap");
return 1;
}
if (close(fd)==-1) {
perror("close");
return 1;
}
for (len = 0; len < sb.st_size; len++) {
putchar(p[len]);
}
if (munmap(p, sb.st_size) == -1) {
perror("munmao");
return 1;
}
fprintf(stderr,"\n");
return 0;
}
The problem is that I am getting a mmap: Invalid argument. I assume something is wrong with fd but have no clue how to fix it, any help would be appreciated. I am on Yosemite using latest XCODE .
You need to extend the size of the shared memory mapping, at least the first time when you create it. Right now its size is 0, and mmap is not going to allow you to make a zero length mapping.
So instead of your fstat() call, do e.g.:
size_t len = 4096;
if (ftruncate(fd, len) == -1) {
perror("ftruncate");
return 1;
}
And pass this len to mmap().
Your addr parameter is set to 0, which might be reserved. Did you mean to use NULL? This would be different than 0.

‘ltrunc’ was not declared in this scope

I have problem with creating shared memory between two processes.
I getting this error, and I don't know what to do because I think a have all libraries included.
Log...
g++ exc8.c -o exc8
exc8.c: In function ‘int main(int, char**)’:
exc8.c:29:37: error: ‘ltrunc’ was not declared in this scope
size = ltrunc(fd, B_SIZE, SEEK_SET);
Code...
#include <stdlib.h>
#include <sys/mman.h>
#include <fcntl.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <unistd.h>
#include <sys/wait.h>
#include <errno.h>
#include <stdio.h>
#define B_SIZE 4
char* memory = new char[4];
int main(int argc, char *argv[]) {
int size, fd;
char *buf;
char memory[4];
fd = shm_open(memory, O_RDWR|O_CREAT, 0774);
if (fd < -0) {
perror("open");
exit(-1);
}
size = ltrunc(fd, B_SIZE, SEEK_SET);
if(size < 0) {
perror("trunc");
exit(-1);
}
buf = (char *)mmap(0, B_SIZE, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
if(buf == NULL) {
perror("map");
exit(-1);
}
}
I don't know where ltrunc() comes from, but
you can set the size of a shared memory object with ftruncate():
if (ftruncate(fd, B_SIZE) == -1) {
// Handle error
}
(from http://pubs.opengroup.org/onlinepubs/009695299/functions/shm_open.html).
ltrunc is not a standard function. It seems defined in QNX Platform using qcc as the compiler, which truncates a file at given position. Probably POSIX provides the truncate() and ftruncate() functions for the job.

Why is remap_file_pages() failing in this example?

The following C code illustrates a problem I'm seeing on Linux 2.6.30.5-43.fc11.x86_64:
#include <sys/types.h>
#include <sys/stat.h>
#include <sys/mman.h>
#include <fcntl.h>
#include <unistd.h>
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
int main() {
char buf[1024];
void *base;
int fd;
size_t pagesz = sysconf(_SC_PAGE_SIZE);
fd = open("<some file, at least 4*pagesz in length>", O_RDONLY);
if (fd < 0) {
perror("open");
return 1;
}
base = mmap(0, 4*pagesz, PROT_READ, MAP_SHARED, fd, 0);
if (base < 0) {
perror("mmap");
close(fd);
return 1;
}
memcpy(buf, (char*)base + 2*pagesz, 1024);
if (remap_file_pages(base, pagesz, 0, 2, 0) < 0) {
perror("remap_file_pages");
munmap(base, 4*pagesz);
close(fd);
return 1;
}
printf("%d\n", memcmp(buf, base, 1024));
munmap(base, 4*pagesz);
close(fd);
return 0;
}
This always fails with remap_file_pages() returning -1 and errno set to EINVAL. Looking at the kernel source I can see all the conditions in remap_file_pages() where it might fail but none of them seem to apply to my example. What's going on?
It's caused by the file being opened O_RDONLY. If you change the open mode to O_RDWR, it works (even if the mmap() still specifies just PROT_READ).
This code in do_mmap_pgoff is the root cause - it only marks the vma as VM_SHARED if the file was opened for writing:
vm_flags |= VM_SHARED | VM_MAYSHARE;
if (!(file->f_mode & FMODE_WRITE))
vm_flags &= ~(VM_MAYWRITE | VM_SHARED);
So in remap_file_pages(), you fail on the first check:
if (!vma || !(vma->vm_flags & VM_SHARED))
goto out;

Resources