I'm studying Memory Mapping on Windows and wrote the following piece code (I omitted error handling from the copy for the sake of readability):
HANDLE file_h = CreateFile(filename, GENERIC_READ | GENERIC_WRITE, 0, NULL, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, NULL);
HANDLE map_h = CreateFileMapping(file_h, NULL, PAGE_READWRITE, 0, lengthOfFile + padding, NULL);
char * map;
map = MapViewOfFile(map_h, FILE_MAP_COPY, 0, 0, lengthOfFile + padding);
Where lengthOfFile is the length of the file I previously calculated this way (for some reasons that are unrelevant to this case):
FILE * file;
file = fopen(filename,"rb");
fseek(file, 0, SEEK_END);
int lengthOfFile = ftell(file);
fseek(file, 0, SEEK_SET);
int last = (lengthOfFile % 4);
int n_pack = (int)(lengthOfFile / 4);
int padding = 4 - last;
And padding is some additional length I needed to add (you may figure out why by reading the code above).
After that, I perform some operations with the memory mapped file which involve its modification and the dispatch of its new value to another function.
How can I make it so that when I close both the file_h and map_h handles the source file (filename) stays unaltered (right now, as soon as I close its handle it gets modified because of that additional padding which apparently gets "flushed" to the source file right after its handle is closed) ?
I tried using the PAGE_WRITECOPY flag alongside the PAGE_READWRITE (which is needed to modify the content of the memory mapped file) one, but the CreateFileMapping function fails returning a ERROR_INVALID_PARAMETER (87) error.
In other words, I need to achieve the same behavior I managed to get in Unix using:
mmap(0,lengthOfFile + padding,PROT_READ | PROT_WRITE, **MAP_PRIVATE**,fileno(file),0);
I guess the key point is the MAP_PRIVATE attribute.
Related
I am trying to write a program that reads a file using 'mmap' for school. I am having some difficulty creating the map. Specifically, I am getting a segmentation fault. I am not really sure what I am doing wrong here so some concrete help would be appreciated. Thank you.
#include <stdio.h>
#include <stdlib.h>
#include <sys/mman.h>
int main(int argc, char* argv[])
{
printf("Hello world!\n");
FILE* fp;// File pointer
int fd;// File descriptor
size_t size;// Length of the file
char* map;// File mmap
/* Open the file */
fp = fopen("data.txt", "r+");
/* Get the file descriptor */
fd = fileno(fp);
printf("FD: %d\n", fd);
/* Get the size of the file */
fseek(fp, 0, SEEK_END);
size = ftell(fp);
fseek(fp, 0, SEEK_SET);
printf("SIZE: %d\n", size);
/* Map the file with mmap */
map = mmap(NULL, size, PROT_READ, 0, fd, 0);
if (map == MAP_FAILED)
{
printf("MMAP FAILED\n");
} else {
printf("MMAP SUCEEDED\n");
}
/* Do something with the map */
int i;
for (i = 0; i < size; i++)
{
char c;
c = map[i];
putchar(c);
}
fclose(fp);
return(0);
}
You are not specifying anything as the flag argument, you must either specify MAP_PRIVATE or MAP_SHARED as specified here:
The flags argument determines whether updates to the mapping are
visible to other processes mapping the same region, and whether
updates are carried through to the underlying file. This behavior is determined by including exactly one of the following values in flags:
MAP_SHARED Share this mapping. Updates to the mapping are visible to
other processes that map this file, and are carried through to
the underlying file. (To precisely control when updates are
carried through to the underlying file requires the use of
msync(2).)
MAP_PRIVATE
Create a private copy-on-write mapping. Updates to the
mapping are not visible to other processes mapping the same
file, and are not carried through to the underlying file. It
is unspecified whether changes made to the file after the
mmap() call are visible in the mapped region.
In your case, since you are just reading the file, MAP_PRIVATE should be enough.
Try with:
map = mmap(NULL, size, PROT_READ, MAP_PRIVATE, fd, 0);
I'm trying to get to work ReadFile function. Here's my code:
#define BUFFERSIZE 5
int main(int argc, char* argv[])
{
OVERLAPPED overlapIn = {};
HANDLE tHandle;
char buf[BUFFERSIZE] = {};
DWORD lpNumberOfBytesRead;
tHandle = CreateFile(
L"\\\\.\\D:",
GENERIC_READ,
FILE_SHARE_READ,
NULL,
OPEN_EXISTING,
FILE_ATTRIBUTE_NORMAL,
NULL);
if (tHandle == INVALID_HANDLE_VALUE)
{
DWORD error = GetLastError();
assert(0);
}
if (ReadFile(tHandle, &buf, BUFFERSIZE - 1, &lpNumberOfBytesRead, NULL) == 0)
{
int error = GetLastError();
printf("Terminal failure: Unable to read from disk.\n GetLastError=%d\n", error);
CloseHandle(tHandle);
return 1;
}
The GetLastError function returns code 87, which is ERROR_INVALID_PARAMETER.
It's clear that one of the parameters is wrong, but I have no idea which one, since I tried to do everything like it's written in the documentation.
This is described in the documentation for CreateFile:
Volume handles can be opened as noncached at the discretion of the particular file system, even when the noncached option is not specified in CreateFile. You should assume that all Microsoft file systems open volume handles as noncached.
The MSDN article on File Buffering describes the requirements for noncached handles:
File access sizes, including the optional file offset in the OVERLAPPED structure, if specified, must be for a number of bytes that is an integer multiple of the volume sector size. For example, if the sector size is 512 bytes, an application can request reads and writes of 512, 1,024, 1,536, or 2,048 bytes, but not of 335, 981, or 7,171 bytes.
File access buffer addresses for read and write operations should be physical sector-aligned, which means aligned on addresses in memory that are integer multiples of the volume's physical sector size. Depending on the disk, this requirement may not be enforced.
Rigorous code should check the sector size for the file system in question, then use this approach to allocate the memory. However, in my experience, the sector size has always been less than or equal to the allocation granularity, so you can get away with just using VirtualAlloc() to allocate a memory block.
buffer size needs to be aligned with hdd sectorsize
WIN32_FIND_DATA atr = {0};
DWORD BYTES_PER_SECTOR;
char path[MAX_PATH];
/* get path length current dir */
const size_t len = GetCurrentDirectory(0, 0);
/* set path to path char array */
GetCurrentDirectory(len, path);
/* windows function to get disk details */
GetDiskFreeSpace(NULL, NULL, &BYTES_PER_SECTOR, NULL, NULL);
/* find first file in dir */
find = FindFirstFile(path, &atr);
for(;find != INVALID_HANDLE_VALUE;){
/* get the file size */
DWORD filesize = atr.nFileSizeLow;
if(atr.nFileSizeHigh > 0){
filesize = atr.nFileSizeHigh;
filesize = (filesize << 31);
filesize = atr.nFileSizeLow;
}
/* sector size aligned file size */
size_t buffer_size = ((BYTES_PER_SECTOR + ((filesize + BYTES_PER_SECTOR)-1)) & ~(BYTES_PER_SECTOR -1));
/* create buffer */
DWORD buffer[buffer_size];
/* create a new file or open an existing file */
handle = CreateFile(&path[0], GENERIC_READ | GENERIC_WRITE, 0 , NULL, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL | FILE_FLAG_NO_BUFFERING, NULL))!=INVALID_HANDLE_VALUE)
/* read the file in to buffer */
ReadFile(handle, (void*)&buffer, buffer_size, &bytesread, NULL)
if(FindNextFile(find, &atr)==0){ printf("last file processed, leaving\n");break;};
}
CloseHandle(file);
FindClose(find);
What I need to do is to fill the entire file contents with zeros in the fastest way. I know some linux commands like cp actually gets what is the best block size information to write at a time, but I wasn't able to figure out if using this block size information is enough to have a nice performance and looks like the st_blksize from the stat() isn't giving me that block size.
Thank you !
Some answers to the comments:
This need to be done in C, not using utilities like shred.
There is no error in the usage of the stat()
st_blksize is returning a block greater than the file size,
don't know how can I handle that.
Using truncate()/ftruncate(), only the extra space is filled with
zeros, I need to overwrite the entire file data.
I'm thinking in something like:
fd = open("file.txt", O_WRONLY);
// check for errors (...)
while(TRUE)
{
ret = write(fd, buffer, sizeof(buffer));
if (ret == -1) break;
}
close(fd);
The problem is how to define the best buffer size "programmatically".
Fastest and simplest:
int fd = open("file", O_WRONLY);
off_t size = lseek(fd, 0, SEEK_END);
ftruncate(fd, 0);
ftruncate(fd, size);
Obviously it would be nice to add some error checking.
This solution is not what you want for secure obliteration of the file though. It will simply mark the old blocks used by the file as unused and leave a sparse file that doesn't occupy any physical space. If you want to clear the old contents of the file from the physical storage medium, you might try something like:
static const char zeros[4096];
int fd = open("file", O_WRONLY);
off_t size = lseek(fd, 0, SEEK_END);
lseek(fd, 0, SEEK_SET);
while (size>sizeof zeros)
size -= write(fd, zeros, sizeof zeros);
while (size)
size -= write(fd, zeros, size);
You could increase the size of zeros up to 32768 or so if testing shows that it improves performance, but beyond a certain point it should not help and will just be a waste.
With mmap (and without error checking):
stat(filename,&stat_buf);
len=stat_buf.st_size;
fd=open(filename,O_RDWR);
ptr=mmap(NULL,len,PROT_READ|PROT_WRITE,MAP_SHARED,fd,0);
memset(ptr,0,len);
munmap(ptr,len);
close(fd);
This should use the kernel's idea of block size, so you don't need to worry about it. Unless the file is larger than your address space.
This is my idea; notice I removed every error checking code for clarity.
int f = open("file", "w"); // open file
int len = lseek(f, 0, SEEK_END); // and get its length
lseek(f, 0, SEEK_BEG); // then go back at the beginning
char *buff = malloc(len); // create a buffer large enough
memset(buff, 0, len); // fill it with 0s
write(f, buff, len); // write back to file
close(f); // and close
I'm using the CreateFileMapping and MapViewOfFile functions to map a file into memory. After a certain point, I call VirtualProtect to change its protection from read-only to read and write. This call fails and GetLastError gives ERROR_INVALID_PARAMETER.
Here is a simplified version of my code that demonstrates the problem.
#include <stdio.h>
#include <stdlib.h>
#include <windows.h>
int main() {
HANDLE fd, md;
char *addr;
DWORD old;
BOOL ok;
fd = CreateFile("filename", GENERIC_READ|GENERIC_WRITE, 0, NULL, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, NULL);
md = CreateFileMapping(fd, NULL, PAGE_READWRITE, 0, 100, NULL);
addr = MapViewOfFile(md, FILE_MAP_READ, 0, 0, 100);
ok = VirtualProtect(addr, 100, PAGE_READWRITE, &old);
if (!ok) {
// we fall into this if block
DWORD err = GetLastError();
// this outputs "error protecting: 87"
printf("error protecting: %u\n", err);
return 1;
}
UnmapViewOfFile(addr);
CloseHandle(md);
CloseHandle(fd);
return 0;
}
What am I doing wrong here? Am I not allowed to call VirtualProtect on a region containing a mapped file?
Start out by creating the view with FILE_MAP_READ | FILE_MAP_WRITE and protect with PAGE_READONLY. Now you have no trouble making it PAGE_READWRITE later:
addr = MapViewOfFile(md, FILE_MAP_READ | FILE_MAP_WRITE, 0, 0, 100);
ok = VirtualProtect(addr, 100, PAGE_READONLY, &old);
//...
ok = VirtualProtect(addr, 100, PAGE_READWRITE, &old);
What happens in your code is that VirtualProtectEx (invoked by VirtualProtect of yours) fails with error STATUS_SECTION_PROTECTION (0xC000004E) - "A view to a section specifies a protection that is incompatible with the protection of the initial view" and this seems to be what you did indeed by creating a section view with more restrictive protection (FILE_MAP_READ).
This topic doesn't seem to be documented with enough details, so I think you'd better simply follow what Hans suggested.
According to http://msdn.microsoft.com/en-us/library/aa366556(v=vs.85).aspx this should be legal. According to VirtualProtect documentation, the new flags must be compatible with the "VirtualAlloc" flags - if this transfer to the "MapViewOfFile" flags, I'd suspect that you can tighten but not loosen the protection. Try mapping readwrite and changing protection to readonly.
The following is code I've used to create a memory mapped file:
fid = open(filename, O_CREAT | O_RDWR, 0660);
if ( 0 > fid )
{
throw error;
}
/* mapped offset pointer to data file */
offset_table_p = (ubyte_2 *) shmat(fid, 0, SHM_MAP);
/* Initialize table */
memset(offset_table_p, 0x00, (table_size + 1) * 2);
say table_size is around 2XXXXXXXX bytes.
During debug, I've noticed it fails while attempt to initializing the 'offset table pointer',
Can anyone provide me some inputs on why it's failing during intilalization?
is there any possibilities that my memory map file was not created with required table size?
As far as I can tell from reading documentation, you are doing it completely wrong.
Either use open() and mmap() or use shmget() and shmat().
If you use open() you will need to make the file long enough first. Use ftruncate() for that.
First things first:
Examine the file both before and after the open() call. If on Linux, you can use the code:
char paxbuff[1000]; // at start of function
sprintf (paxbuff,"ls -al %s",filename);
system (paxbuff);
fid = open(filename, O_CREAT | O_RDWR, 0660); // this line already exists.
system (paxbuff);
Then, after you call shmat(), check the return values and size thus:
offset_table_p = (ubyte_2 *) shmat(fid, 0, SHM_MAP); // already exists.
printf ("ret = %p, errno = %d\n",offset_table_p,errno);
printf ("sz = %d\n",table_size);
That should be enough to work out the problem.