I am trying to scan the address space to find my chunks of memory that have read/write permissions. It is acceptable to check a single address per page as each page have the same permissions. I know I should be getting Segmentation Fault: 11 when trying to write to a piece of memory I shouldn't be able to. This happens when I am trying to access higher addresses but when I am in the lower portion, say 0x00000100, I get the Bus error: 10.
NOTE: The code is compiled with the -m32 flag so it simulates a 32 bit machine.
ALSO NOTE: The memory for chunk_list has already been malloc'ed before this function is called.
I have copied the code below:
#include <signal.h>
#include <stdio.h>
#include <unistd.h>
#include "memchunk.h"
int get_mem_layout (struct memchunk *chunk_list, int size)
{
//grab the page size
long page = sysconf(_SC_PAGESIZE);
printf("The page size for this system is %ld bytes\n", page);
//page size is 4069 bytes
//test printing the number of words on a page
long words = page / 4;
printf("Which works out to %ld words per page\n", words);
//works out to 1024 words a page
//1024 = 0x400
//create the addy pointer
//start will be used after bus error: 10 is solved
void *start;
char * currAddy;
currAddy = (char*)0x01000000;
//someplace to store the addy to write to
//char * testWrite;
//looping through the first size pages
int i;
for(i = 0; i < size; i++){
//chunk start - wrong addy being written just testing
chunk_list[i].start = currAddy;
printf("addy is %p\n",currAddy);
sleep(1);
//try and write to the current addy
//testWrite = currAddy;
//*testWrite = 'a';
*currAddy = '1';
//+= 0x400 to get to next page
currAddy += 0x400;
}
//while loop here for all the addys - not there yet because still dealing with bus error: 10
return 0;
}
Any help would be greatly appreciated. I also left some other attempts at it commented out in the code, still all produce a bus error: 10 in the lower portion of the memory space.
EDIT: I will be dealing with seg faults using signals. I know how to deal with the seg fault, so is there a way to handle a bus error: 10 using signals as well?
Reading from or writing to unmapped memory is supposed to cause a bus fault. To discover whether a memory is there, install a handler for SEGFAULTs to react accordingly.
In a Linux SE (Security Enhanced) process, the program sections are loaded at randomized locations to frustrate viruses being able to rely on stable addresses.
In most virtual memory systems, a non-mapped space is usually left from address zero up a ways so attempts to dereference a NULL pointer or a structure based on a NULL pointer cause an exception. In the 1980s, the blank space was often 64K to 256K. On modern architectures, 16M is a reasonable choice to detect NULL-based accesses.
On many virtual memory systems, there is a system call to obtain per process mapped memory locations. On Linux, inspect the contents of /proc/self/maps.
Related
I'm running a program called VMdriver5.exe (left side of image I attached), that creates a process of another program called VMmapper.exe. Inside the VMdriver5.c code, it gets its PID (using GetCurrentProcessId()) and passes it's PID to VMmapper.exe upon creation, such that VMmapper shows the memory layout of VMdriver5.exe so I can see the virtual memory options.
I was wondering if I could get some help, because, when I call VirtualAlloc() on certain memory addresses, it works fine. However, as you can see in the image below, there is a region of memory with 503,808 bytes free, and I attempt to reserve 65,536 bytes of that space using VirtualAlloc(), but it fails for some reason. I pass the memory address 00185000 as one of the parameters.
The code I'm using is this:
lpvAddr = VirtualAlloc(vmAddress, units << 16, MEM_RESERVE, flProtect); // only works with PAGE_READONLY
if(lpvAddr == NULL) {
printf("Case 1: Reserve a region, VirtualAlloc() failed. Error: %ld\n", GetLastError());
}
else {
printf("Committed %lu bytes at address 0x%lp\n",dwPageSize, lpvAddr); // %lu = long unsigned decimal integer
}
break;
I would appreciate any help and can provide any more code or info if needed. Thanks guys and happy easter.
malloc/calloc apparently use swap space to satisfy a request that exceeds available free memory. And that pretty much hangs the system as the disk-use light remains constantly on. After it happened to me, and I wasn't immediately sure why, I wrote the following 5-line test program to check that this is indeed why the system was hanging,
/* --- test how many bytes can be malloc'ed successfully --- */
#include <stdio.h>
#include <stdlib.h>
int main ( int argc, char *argv[] ) {
unsigned int nmalloc = (argc>1? atoi(argv[1]) : 10000000 ),
size = (argc>2? atoi(argv[2]) : (0) );
unsigned char *pmalloc = (size>0? calloc(nmalloc,size):malloc(nmalloc));
fprintf( stdout," %s malloc'ed %d elements of %d bytes each.\n",
(pmalloc==NULL? "UNsuccessfully" : "Successfully"),
nmalloc, (size>0?size:1) );
if ( pmalloc != NULL ) free(pmalloc);
} /* --- end-of-function main() --- */
And that indeed hangs the system if the product of your two command-line args exceeds physical memory. Easiest solution is some way whereby malloc/calloc automatically just fail. Harder and non-portable was to write a little wrapper that popen()'s a free command, parses the output, and only calls malloc/calloc if the request can be satisfied by the available "free" memory, maybe with a little safety factor built in.
Is there any easier and more portable way to accomplish that? (Apparently similar to this question can calloc or malloc be used to allocate ONLY physical memory in OSX?, but I'm hoping for some kind of "yes" answer.)
E d i t--------------
Decided to follow Tom's /proc/meminfo suggestion. That is, rather than popen()'ing "free", just directly parse the existing and easily-parsible /proc/meminfo file. And then, a one-line macro of the form
#define noswapmalloc(n) ( (n) < 1000l*memfree(NULL)/2? malloc(n) : NULL )
finishes the job. memfree(), shown below, isn't as portable as I'd like, but can easily and transparently be replaced by a better solution if/when the need arises, which isn't now.
#include <stdio.h>
#include <stdlib.h>
#define _GNU_SOURCE /* for strcasestr() in string.h */
#include <string.h>
char *strcasestr(); /* non-standard extension */
/* ==========================================================================
* Function: memfree ( memtype )
* Purpose: return number of Kbytes of available memory
* (as reported in /proc/meminfo)
* --------------------------------------------------------------------------
* Arguments: memtype (I) (char *) to null-terminated, case-insensitive
* (sub)string matching first field in
* /proc/meminfo (NULL uses MemFree)
* --------------------------------------------------------------------------
* Returns: ( int ) #Kbytes of memory, or -1 for any error
* --------------------------------------------------------------------------
* Notes: o
* ======================================================================= */
/* --- entry point --- */
int memfree ( char *memtype ) {
/* ---
* allocations and declarations
* ------------------------------- */
static char memfile[99] = "/proc/meminfo"; /* linux standard */
static char deftype[99] = "MemFree"; /* default if caller passes null */
FILE *fp = fopen(memfile,"r"); /* open memfile for read */
char memline[999]; /* read memfile line-by-line */
int nkbytes = (-1); /* #Kbytes, init for error */
/* ---
* read memfile until line with desired memtype found
* ----------------------------------------------------- */
if ( memtype == NULL ) memtype = deftype; /* caller wants default */
if ( fp == NULL ) goto end_of_job; /* but we can't get it */
while ( fgets(memline,512,fp) /* read next line */
!= NULL ) { /* quit at eof (or error) */
if ( strcasestr(memline,memtype) /* look for memtype in line */
!= NULL ) { /* found line with memtype */
char *delim = strchr(memline,':'); /* colon following MemType */
if ( delim != NULL ) /* NULL if file format error? */
nkbytes = atoi(delim+1); /* num after colon is #Kbytes */
break; } /* no need to read further */
} /* --- end-of-while(fgets()!=NULL) --- */
end_of_job: /* back to caller with nkbytes */
if ( fp != NULL ) fclose(fp); /* close /proc/meminfo file */
return ( nkbytes ); /* and return nkbytes to caller */
} /* --- end-of-function memfree() --- */
#if defined(MEMFREETEST)
int main ( int argc, char *argv[] ) {
char *memtype = ( argc>1? argv[1] : NULL );
int memfree();
printf ( " memfree(\"%s\") = %d Kbytes\n Have a nice day.\n",
(memtype==NULL?" ":memtype), memfree(memtype) );
} /* --- end-of-function main() --- */
#endif
malloc/calloc apparently use swap space to satisfy a request that exceeds available free memory.
Well, no.
Malloc/calloc use virtual memory. The "virtual" means that it's not real - it's an artificially constructed illusion made out of fakery and lies. Your entire process is built on these artificially constructed illusions - a thread is a virtual CPU, a socket is a virtual network connection, the C language is really a specification for a "C abstract machine", a process is a virtual computer (that implements the languages' abstract machine).
You're not supposed to look behind the magic curtain. You're not supposed to know that physical memory exists. The system doesn't hang - the illusion is just slower, but that's fine because the C abstract machine says nothing about how long anything is supposed to take and does not provide any performance guarantees.
More importantly; because of the illusion, software works. It doesn't crash because there's not enough physical memory. Failure means that it takes an infinite amount of time for software to complete successfully, and "an infinite amount of time" is many orders of magnitude worse than "slower because of swap space".
How to get malloc/calloc to fail if request exceeds free physical memory (i.e., don't use swap)
If you are going to look behind the magic curtain, you need to define your goals carefully.
For one example, imagine if your process has 123 MiB of code and there's currently 1000 MiB of free physical RAM; but (because the code is in virtual memory) only a tiny piece of the code is using real RAM (and the rest of the code is on disk because the OS/executable loader used memory mapped files to avoid wasting real RAM until it's actually necessary). You decide to allocate 1000 MiB of memory (and because the OS creating the illusion isn't very good, unfortunately this causes 1000 MiB of real RAM to be allocated). Next, you execute some more code, but the code you execute isn't in real memory yet, so the OS has to fetch the code from the file on the disk into physical RAM, but you consumed all of the physical RAM so the OS has to send some of the data to swap space.
For another example, imagine if your process has 1 MiB of code and 1234 MiB of data that was carefully allocated to make sure that everything fits in physical memory. Then a completely different process is started and it allocates 6789 MiB of memory for its code and data; so the OS sends all of your process' data to swap space to satisfy the other process that you have no control over.
EDIT
The problem here is that the OS providing the illusion is not very good. When you allocate a large amount of virtual memory with malloc() or calloc(); the OS should be able to use a tiny piece of real memory to lie to you and avoid consuming a large amount of real memory. Specifically (for most modern operating systems running on normal hardware); the OS should be able to fill a huge area of virtual memory with a single page full of zeros that is mapped many times (at many virtual addresses) as "read only", so that allocating a huge amount of virtual memory costs almost no physical RAM at all (until you write to the virtual memory, causing the OS to allocate the least physical memory needed to satisfy the modifications). Of course if you eventually do write to all of the allocated virtual memory, then you'll end up exhausting physical memory and using some swap space; but this will probably happen gradually and not all at once - many tiny delays scattered over a large period of time are far less likely to be noticed than a single huge delay.
With this in mind; I'd be tempted to try using mmap(..., MAP_ANONYMOUS, ...) instead of the (poorly implemented) malloc() or calloc(). This might mean that you have to deal with the possibility that the allocated virtual memory isn't guaranteed to be initialized to zeros, but (depending on what you're using the memory for) that's likely to be easy to work around.
Expanding on a comment I made to the original question:
If you want to disable swapping, use the swapoff command (sudo swapoff -a). I usually run my machine that way, to avoid it freezing when firefox does something it shouldn't. You can use setrlimit() (or the ulimit command) to set a maximum VM size, but that won't properly compensate for some other process suddenly deciding to be a memory hog (see above).
Even if you choose one of the above options, you should read the rest of this answer to see how to avoid unnecessary initialisation on the first call to calloc().
As for your precise test harness, it turns out that you are triggering an unfortunate exception to GNU calloc()'s optimisation.
Here's a comment (now deleted) I made to another answer, which turns out to not be strictly speaking accurate:
I checked the glibc source for the default gnu/linux malloc library, and verified that calloc() does not normally manually clear memory which has just been mmap'd. And malloc() doesn't touch the memory at all.
It turns out that I missed one exception to the calloc optimisation. Because of the way the GNU malloc implementation initialises the malloc system, the first call to calloc always uses memset() to set the newly-allocated storage to 0. Every other call to calloc() passes through the entire calloc logic, which avoids calling memset() on storage which has been freshly mmap'd.
So the following modification to the test program shows radically different behaviour:
#include <stdio.h>
#include <stdlib.h>
int main ( int argc, char *argv[] ) {
/* These three lines were added */
void* tmp = calloc(1000, 1); /* force initialization */
printf("Allocated 1000 bytes at %p\n", tmp);
free(tmp);
/* The rest is unchanged */
unsigned int nmalloc = (argc>1? atoi(argv[1]) : 10000000 ),
size = (argc>2? atoi(argv[2]) : (0) );
unsigned char *pmalloc = (size>0? calloc(nmalloc,size):malloc(nmalloc));
fprintf( stdout," %s malloc'ed %d elements of %d bytes each.\n",
(pmalloc==NULL? "UNsuccessfully" : "Successfully"),
nmalloc, (size>0?size:1) );
if ( pmalloc != NULL ) free(pmalloc);
}
Note that if you set MALLOC_PERTURB_ to a non-zero value, then it is used to initialise malloc()'d blocks, and forces calloc()'d blocks to be initialised to 0. That's used in the test below.
In the following, I used /usr/bin/time to show the number of page faults during execution. Pay attention to the number of minor faults, which are the result of the operating system zero-initialising a previously unreferenced page in an anonymous mmap'd region (and some other occurrences, like mapping a page already present in Linux's page cache). Also look at the resident set size and, of course, the execution time.
$ gcc -Og -ggdb -Wall -o mall mall.c
$ # A simple malloc completes instantly without page faults
$ /usr/bin/time ./mall 4000000000
Allocated 1000 bytes at 0x55b94ff56260
Successfully malloc'ed -294967296 elements of 1 bytes each.
0.00user 0.00system 0:00.00elapsed 100%CPU (0avgtext+0avgdata 1600maxresident)k
0inputs+0outputs (0major+61minor)pagefaults 0swaps
$ # Unless we tell malloc to initialise memory
$ MALLOC_PERTURB_=35 /usr/bin/time ./mall 4000000000
Allocated 1000 bytes at 0x5648c2436260
Successfully malloc'ed -294967296 elements of 1 bytes each.
0.19user 1.23system 0:01.43elapsed 99%CPU (0avgtext+0avgdata 3907584maxresident)k
0inputs+0outputs (0major+976623minor)pagefaults 0swaps
# Same, with calloc. No page faults, instant completion.
$ /usr/bin/time ./mall 1000000000 4
Allocated 1000 bytes at 0x55e8257bb260
Successfully malloc'ed 1000000000 elements of 4 bytes each.
0.00user 0.00system 0:00.00elapsed 100%CPU (0avgtext+0avgdata 1656maxresident)k
0inputs+0outputs (0major+62minor)pagefaults 0swaps
$ # Again, setting the magic malloc config variable changes everything
$ MALLOC_PERMUTE_=35 /usr/bin/time ./mall 1000000000 4
Allocated 1000 bytes at 0x5646f391e260
Successfully malloc'ed 1000000000 elements of 4 bytes each.
0.00user 0.00system 0:00.00elapsed 100%CPU (0avgtext+0avgdata 1656maxresident)k
0inputs+0outputs (0major+62minor)pagefaults 0swaps
I am working on an embedded device with only 512MB of RAM and the device is running Linux kernel. I want to do the memory management of all the processes running in the userspace by my own library. is it possible to do so. from my understanding, the memory management is done by kernel, Is it possible to have that functionality in User space.
If your embedded device runs Linux, it has an MMU. Controling the MMU is normally a privileged operation, so only an operating system kernel has access to it. Therefore the answer is: No, you can't.
Of course you can write software running directly on the device, without operating system, but I guess that's not what you wanted. You should probably take one step back, ask yourself what gave you the idea about the memory management and what could be a better way to solve this original problem.
You can consider using setrlimit. Refer another Q&A.
I wrote the test code and run it on my PC. I can see that memory usage is limited. The exact relationship of units requires further analysis.
#include <stdlib.h>
#include <stdio.h>
#include <sys/time.h>
#include <sys/resource.h>
int main(int argc, char* argv)
{
long limitSize = 1;
long testSize = 140000;
// 1. BEFORE: getrlimit
{
struct rlimit asLimit;
getrlimit(RLIMIT_AS, &asLimit);
printf("BEFORE: rlimit(RLIMIT_AS) = %ld,%ld\n", asLimit.rlim_cur, asLimit.rlim_max);
}
// 2. BEFORE: test malloc
{
char *xx = malloc(testSize);
if (xx == NULL)
perror("malloc FAIL");
else
printf("malloc(%ld) OK\n", testSize);
free(xx);
}
// 3. setrlimit
{
struct rlimit new;
new.rlim_cur = limitSize;
new.rlim_max = limitSize;
setrlimit(RLIMIT_AS, &new);
}
// 4. AFTER: getrlimit
{
struct rlimit asLimit;
getrlimit(RLIMIT_AS, &asLimit);
printf("AFTER: rlimit(RLIMIT_AS) = %ld,%ld\n", asLimit.rlim_cur, asLimit.rlim_max);
}
// 5. AFTER: test malloc
{
char *xx = malloc(testSize);
if (xx == NULL)
perror("malloc FAIL");
else
printf("malloc(%ld) OK\n", testSize);
free(xx);
}
return 0;
}
Result:
BEFORE: rlimit(RLIMIT_AS) = -1,-1
malloc(140000) OK
AFTER: rlimit(RLIMIT_AS) = 1,1
malloc FAIL: Cannot allocate memory
From what I understand of your question, you want to somehow use your own library for handling memory of kernel processes. I presume you are doing this to make sure that rogue processes don't use too much memory, which allows your process to use as much memory as is available. I believe this idea is flawed.
For example, imagine this scenario:
Total memory 512MB
Process 1 limit of 128MB - Uses 64MB
Process 2 imit of 128MB - Uses 64MB
Process 3 limit of 256MB - Uses 256MB then runs out of memory, when in fact 128MB is still available.
I know you THINK this is the answer to your problem, and on 'normal' embedded systems, this would probably work, but you are using a complex kernel, running processes you don't have total control over. You should write YOUR software to be robust when memory gets tight because that is all you can control.
On trying to deal with heaps in WinApi I've got some strange results for HeapAlloc's actions. Lets consider the following code. The problem is that according to Microsoft Documentation on Windows API(next - Doc), I have to get two Success strings printed to the console. By I get an Error when trying to run this code with the Debud option in MSVC 2013. But the strangest thing is that when I try to run this code without Debug option, or run the compiled .exe file, I get the correct result.
#include <Windows.h>
#include <stdio.h>
int main()
{
LPSYSTEM_INFO sys;
HANDLE hNewHeap;
LPVOID ptr;
sys = (LPSYSTEM_INFO) HeapAlloc(GetProcessHeap(),
0,
sizeof(SYSTEM_INFO));
GetSystemInfo(sys);
printf("Page size: %u\n", sys->dwPageSize);//Here we get the
//'Page size: 4096' string
//printed to the console
hNewHeap = HeapCreate(0, 1, 1);
//That's easy. We create new heap object, getting its HADNLE descriptor.
//According to Doc, the initial heap size is set to page size, which is
//4096 on my computer, like maximum heap size is also done. So the heap
//size now is 4096.
ptr = HeapAlloc(hNewHeap, 0, 2624); //Here we allocate the memory
//block in our new heap, that might have 2624 bytes size.
if ( ptr ) printf("Success!\n");//Here we check if the HeapAlloc functio
//worked correctly and print the appropriate string.
else printf("Error!\n");
//On this time we get 'Success' string printed to the console and free
//allocated memory block
if ( ptr ) HeapFree(hNewHeap, 0, ptr);
ptr = HeapAlloc(hNewHeap, 0, 2525);//Here we try to allocate the memory
//block, which size is 2526. And, like previous time, we expect to get
//'Success'.
if ( ptr ) printf("Success!\n");
else printf("Error!\n");
//But we get 'Error' here!!!
if ( ptr ) HeapFree(hNewHeap, 0, ptr);
HeapDestroy(hNewHeap);
system("pause");
};
If you try the same with any less than 2624 number, you will not get an 'Error'. If you try to do this with any more that 2625 number, you WILL get an 'Error'. But we get an 'Error' only when the Debug option is on.
Can somebody explain me why is happening so?
P.S.: Sorry for bad English.
P.S.: The strange is also the number 2625 that do not correspond to any function or application size and that sometimes I get correct result, that is after restarting studio or making some changes in the code.(But only sometimes)
You are creating a fixed size heap. The documentation says:
The HeapCreate function rounds dwMaximumSize up to a multiple of the system page size and then reserves a block of that size in the process's virtual address space for the heap.
So your heap has a fixed size of 4096 bytes.
Subsequently your allocation from this heap fails because the heap is not large enough for the block being allocated. Again from the documentation:
The system uses memory from the private heap to store heap support structures, so not all of the specified heap size is available to the process. For example, if the HeapAlloc function requests 64 kilobytes (K) from a heap with a maximum size of 64K, the request may fail because of system overhead.
It is these heap support structures that are causing your confusion. You think that there is sufficient space in the heap for your second allocation, but you are not accounting for the heap support structures.
The documentation for HeapAlloc tells you to call GetExceptionCode on failure. Expect that to return STATUS_NO_MEMORY.
I have some data stored in a FLASH memory that I need to access with C pointers to be able to make a non-Linux graphics driver work (I think this requirement is DMA related, not sure). Calling read works, but I don't want to have intermediate RAM buffers between the FLASH and the non-Linux driver.
However, just creating a pointer and storing the address that I want on it is making Linux emit an exception about invalid access on me.
void *ptr = 0xdeadbeef;
int a = *ptr; // invalid access!
What am I missing here? And could someone point me to a material to make this concepts clear for me?
I'm reading about mmap but I'm not sure that this is what I need.
The problem you have is that linux runs your program in a virtual address space. So every address you use directly in the code (like 0xdeadbeef) is a virtual address that gets translated by the memory management unit into a physical address which is not necessarily the same as your virtual address. This allows easy separation of multiple independent processes and other stuff like paging, etc.
The problem is now, that in your case no physical address is mapped to the virtual address 0xdeadbeef causing the kernel to abort execution.
The call mmap you already found asks the kernel to assign a specific file (from a specific offset) to a virtual address of your process. Note that the returning address of mmap could be a completely different address. So don't make any assumptions about the virtual address you get.
Therefore there are examples with mmap and /dev/mem out there where the offset for the memory device is the physical address. After the kernel was able to assign the file from the offset you gave to a virtual address of your process you can access the memory area asif it were a direct access.
After you don't need the area anymore don't forget to munmap the area. Otherwise you'll cause something similar to a memory leak.
One problem with the /dev/mem method is that the user running the process needs access to this device. This could introduce a security issue (e.g. Samsung recently introduced such a security hole in their hand held devices)
A more secure way is the way described in a article i found (The Userspace I/O HOWTO) as you still have control about the memory areas accessable by the user's process.
You need to access the memory differently. Basically you need to open /dev/mem and use mmap(). (as you suggested). Simple example:
int openMem(unsigned int address, unsigned int size)
{
int mmapFD;
int page_size;
unsigned int page_start_address;
/* Minimum page size for the mmapped region. */
mask = size - 1;
/* Get the page size. */
page_size = (int) sysconf(_SC_PAGE_SIZE);
/* We have to map shared memory to beginning of memory page so adjust
* memory address accordingly. */
page_start_address = address - (address % page_size);
/* Open the file that will be mapped. */
if((mmapFD = open("/dev/mem", (O_RDWR | O_SYNC))) == -1)
{
printf("Opening shared memory device failed\n");
return -1;
}
mmap_base_address = mmap(0, size, (PROT_READ|PROT_WRITE), MAP_SHARED, mmapFD, (off_t)page_start_address & ~mask);
if(mmap_base_address == MAP_FAILED)
{
printf("Mapping memory failed\n");
return -1;
}
return 0;
}
unsigned int *getAddress(unsigned int address)
{
unsigned int log_address;
log_address = (int)((off_t)mmap_base_address + ((off_t)address & mask));
return (unsigned int*)log_address;
}
...
result = openMem(address, 0x10000);
if (result < 0)
return result;
target_address = getValue(address);
*(unsigned int*)target_address = value;
This would set "value" to "address".
You need to call ioremap - something like:
void *myaddr = ioremap(0xdeadbeef, size);
where size is the size of your memory region. You probably want to use a page-aligned address for the first argument, e.g. 0xdeadb000 - but I expect your actual device isn't at "0xdeadbeef" anyways.
Edit: The call to ioremap must be done from a driver!