I'm developing C module for php under linux and I'm trying to find a way that could help me profile my code by maximum memory spike (usage).
Using valgrind I can get total memory allocation within the code. But as it is with allocated memory it comes and goes ;). What I need to get is the highest memory usage that appeared during C application run so I could get total overview on memory requirements and have some measurement point for optimization of the code.
Does anyone know any tool/trick/good practice that could help ?
Take a look at Massif: http://valgrind.org/docs/manual/ms-manual.html
Have you checked massif (one of Valgrind's tool)?
this is actually what you are looking for
another possibility would be memusage (one of glibc's utilities, glibc-utils)
Related
I have a Keil uVision project that I would like to benchmark extensively. The code is currently running in simulator mode. To visualize results we simply store characters in a memory region and display said region as ASCII.
This approach works pretty well to obtain timings using the Cortex-M system ticks. However, I don't have an idea for the ram usage of the code:
Ideally I would the simulator to stop the execution when the maximum amount of ram was used.
I would also like to see maximum heap usage (or even per function).
Is there a way to do obtain these values? I'm aware of the maximum stack size reported by the build system.
Is there a way to limit the amount of ram available in the uVision simulator?
Thanks
There is a fairly obvious solution: Just count the ram in the memory window. First find the memory regions allocated for the heap and the stack (this can usually be found in the startup assembly file). Then go through the memory window in the debugger and just look where the memory didn't get changed.
Keil usually initializes the memory with 0 so the stack boundaries can easily be seen.
The total stack usage can be computed in the following way
$TOTAL = $TOP - $BOTTOM
If the boundary can't be seen it might make sense to first initialize the memory with a pattern (see here).
I have a question that keeps bothering me for the last week.
In Windows debugger there is the !heap -s command that outputs the virtual memory's heap status and calculates the external fragmentation using the formula :
External fragmentation = 1 - (larget free block / total free size)
Is there a similar method in linux, that outputs those statistics needed to calculate the effect?
Long story now:
I have a C application that keeps allocating and deallocating space of different sizes, using malloc and free, each allocation has different life span.
The platform I am using is Lubuntu, so ptmalloc2 algorithm is the default.
I am aware that those allocations are served in the virtual user space heap(except those that are larger than 128Kb, where the allocator uses mmap), and are mapped to physical pages when actually accessed .
The majority of the allocations is of size < 80 bytes, so they are served from FastBins.
Using Valgrind and Massif I can get the internal fragmentation, since it reports the extra bytes used for each allocation.
However, my main concern is how to figure out the external fragmentation.
I am aware of the /proc/[pid]/smaps heap size and the pmap-d[pid] anon statistics, but I find it difficult to interpret them in terms of external fragmentation.
I am also aware of LD_PRELOAD , and I can dynamically connect the /lib/i386-linux-gnu/libmemusage.so. This library outputs the heap total, peak and the distribution of the requested allocation sizes.
I know that __malloc__hook is deprecated now, and I do not really want to rely on implementation specific statistics like malloc_stats() and mallinfo() . However, If you have any suggestions using those two please let me know.
I can tell that the external fragmentation problem, is when a request can not be satisfied, because there is no contiguous space in the heap, but the requested total size is scattered all around that area.
I still have not figured out, how to get the statistics needed so I can calculate this effect. For example different formulas stating that I have to capture the live_memory or get the total_free_pages, or get the size of the largest_free_block.
How can I have a function to "traverse" through the heap and gather those statistics?
Thank you everyone in advance.
I believe that this will depend on the allocator you are using. That is, you would probably need a different strategy for whichever malloc (et al) and free implementation you are using. If the implementation doesn't offer the information you seek as an extension, you will probably have to read its source-code and type your own logic to examine the state of allocations.
I believe that the mapping of pages to swap-space and physical RAM is at a lower level and so won't particularly help you in your goal. The malloc (et al) and free implementation might or might not care about those lower-level details.
If you are certain that you are using ptmalloc2, are you able to find its source-code?
I'm looking for a tool which can help me to generate a memory accesses graph. I'm trying to optimize a search algorithm (written in c) and it would be very useful to know how the memory accesses are performed in order to optimize the memory accesses pattern.
I heard that cachegrind from valgrind could help me, but I think is not exactly what I'm looking for as I think it doesn't generate a trace of memory accesses and I already know the other information that can show me using the PAPI library.
The graph I want to generate has in the x axis the memory access order and in the y axis the memory address space (absolute or relative memory directions)
If you're using gcc or the gnu std C library, the malloc and free hooks might help. https://www.gnu.org/software/libc/manual/html_node/Hooks-for-Malloc.html#Hooks-for-Malloc
Some tools in this project do what you want:
http://www.inf.usi.ch/faculty/hauswirth/research/TraceVisualization.html
I don't know if you can download them anywhere or ask them to share a copy, I used them because I was a student of this professor!
what is the most efficient and accurate way/ API to measure heap memory consumption from the same running process programmatically? I want to estimate (as accurately as is reasonably possible) how much memory has been new or malloc since startup, minus the memory that has been free or delete
the scope of the question is linux and possibly other linux environments. The language is either C or C++
EDIT
It is enough for my purposes to know the actual number (and size) of allocated/held blocks by any malloc implementation, i don't need the detail of actual malloc memory minus the the freed memory
Assuming new uses malloc look here.
For more details of a processes memory allocation, look at the /proc/[pid]/maps.
Also note that linux implements copy-on-write. This means that sometimes processes can share memory. This is especially true if the process was forked without calling exec afterwards.
You can use mallinfo for an estimation. I've just found this, not sure whether this is process or system.. :/
I'm not totally sure what you are asking, malloc minus freed is less than the actual usage because of the memory fragmentation, if you really need that number you have to use custom allocators (which are tiny wrappers around existing ones) everywhere in your code which is going to be painful.
Have you considered reading from /proc/u/stat? (where "u" is your pid)
If you use valgrind and run your program to completion, it gives you a report on memory usage.
http://valgrind.org/
You can get quite a bit of information about your heap usage by linking against tcmalloc from Google Perftools. It is designed to locate memory leaks and determine "who the heck allocated all that RAM", but it provides enough instrumentation to answer most questions you might have about your heap.
Is there a libc function (or equivalent) to know the current size of the heap?
I have a memory problem in my application, and it seems being able to monitor the heap when I want to would help me find the problem. So is there a way to know the current size of the heap?
No.
As the functionality you want is for debugging, it would make a lot more sense for you to use your debugger or your operating system's resource accounting to monitor the process's memory usage, instead of trying to code that into your program.
If you really want your program to keep track of its own memory usage, the only portable way to do this is to avoid using malloc and free directly and instead call them through wrappers that increment/decrement a counter. This will not account for memory fragmentation, but if your interest is in the logical memory usage of your program and not the impact on physical resources, a counter implemented this way might actually be more informative than looking at the operating system's resource accounting.
If you only care about a particular target platform or family of platforms, there may also be functions above and beyond the C standard which do what you want. On POSIX, lookup getrusage.
What you need is http://valgrind.org/
Still wouldn't help.
Do you want to know :
The total address space.
The space available to user programs
The space unallocated, to this process, including swap or not
The biggest free chunk available
etc.
You can use tools like Purify to debug memory issues. This article from IBM contains a lot of details about the sources of such problems and pointers to solutions.