How to find out, how much RAM and CPU "eats" certain process in Linux? And how to find out all runned processes (including daemons and system ones)? =)
UPD: using C language
Use top or ps.
For example, ps aux will list all processes along with their owner, state, memory used, etc.
EDIT: To do that with C under Linux, you need to read the process files in the proc filesystem. For instance, /proc/1/status contains information about your init process (which always has PID 1):
char buf[512];
unsigned long vmsize;
const char *token = "VmSize:";
FILE *status = fopen("/proc/1/status", "r");
if (status != NULL) {
while (fgets(buf, sizeof(buf), status)) {
if (strncmp(buf, token, strlen(token)) == 0) {
sscanf(buf, "%*s %lu", &vmsize);
printf("The INIT process' VM size is %lu kilobytes.\n", vmsize);
break;
}
}
fclose(status);
}
Measuring how much ram a process uses is nearly impossible. The difficulty is, that each piece of ram is not used by exactly one process, and not all ram a process is using is actually "owned" by it.
For example, two processes can have shared mappings of the same file, in which case any pages which are in core for the mapping, would "belong" to both processes. But what if only one of these processes was using it?
Private pages can also be copy-on-write if the process has forked, or if they have been mapped but not used yet (consider the case where a process has malloc'd a huge area but not touched most of it yet). In this case, which process "owns" those pages?
Processes can also be effectively using parts of the buffer cache and lots of other kinds of kernel buffers, which aren't "owned" by them.
There are two measurements which are available, which are VM Size, (how much memory the process has mapped just now) and resident set size (RSS). Neither of them really tells you much about how much memory a process is using, because they both count shared pages and neither counts non-mapped pages.
So is there an answer? Some of these can be measured by examining the page maps structures which are now available in /proc (/proc/pid/pagemap), but there isn't necessarily a trivial way of sharing out the "ownership" of shared pages.
See Linux's Documentation/vm/pagemap.txt for a discussion of this.
Related
I am currently editing a program I inherited to be able to work with 23 GB files. As such, to maintain low memory, I am using mmap to load arrays which I had created in a previous program. However, I load these arrays, and then enter into a function and the shared and reserved memory spikes, even though I do not believe I ever allocate anything. When running, the memory starts at 0 and then quickly increases to 90% (~36GB as I have 40GB of ram) and stays there. Eventually, I start needing memory (less than 30GB) and the program then gets killed.
Usually, I would suspect that this issue would be due to allocation, or that I was somehow allocating memory. However, I am not allocating any memory (although I am reading in mmaped files).
The curious thing is that the memory reserved is equal to the amount of memory shared (see attached screenshot).
The functions that I wrote to access mmaped arrays:
double* loadArrayDouble(ssize_t size, char* backupFile, int *filedestination) {
*filedestination = open(backupFile, O_RDWR | O_CREAT, 0644);
if (*filedestination < 0) {
perror("open failed");
exit(1);
}
// make sure file is big enough
if (lseek(*filedestination,size*sizeof(double), SEEK_SET) == -1) {
perror("seek to len failed");
exit(1);
}
if (lseek(*filedestination, 0, SEEK_SET) == -1) {
perror("seek to 0 failed");
exit(1);
}
double *array1 = mmap(NULL, size*sizeof(double), PROT_READ | PROT_WRITE, MAP_SHARED, *filedestination, 0);
if (array1 == MAP_FAILED) {
perror("mmap failed");
exit(1);
}
return array1;
}
Please let me know if there is any other code to include.. It appears that the memory increases significantly even though double* file1 = loadArrayDouble(SeqSize, "/home/HonoredTarget/file1", &fileIT); is called multiple times (for each of the 6 arrays)
"Res" is short for "resident", not "reserved". Resident memory refers to the process memory which the kernel happens to have resident at the moment; the virtual memory system might drop a resident page at any moment, so it's not in any way a limitation. However, the kernel attempts not to swap out pages which seem to be active. The OOM killer will act if your process is churning too many pages in and out of memory. If you use data sequentially, then it usually doesn't matter how much you have mmap'ed, because only the recent pages will be resident. But if you skip around in the memory, reading a bit here and writing a bit there, then you'll create more churn. That seems like what is happening.
"shr" (shared) memory does in fact refer to memory which could be shared with another process (whether or not it actually is shared with another process). The fact that you use MAP_SHARED means that it is not surprising that all of your mmap'ed pages are shared. You need MAP_SHARED if your program modifies the data in the file, which I guess it does.
The "virt" (virtual) column measures how much of your address spaced you've actually mapped (including memory mapped to anonymous backing storage by whatever dynamic allocation library you're using.) 170G seems a bit high to me. If you have six 23GB files mapped simultaneously, that would be 138GB. But perhaps those numbers were just estimates. Anyway, it doesn't matter that much, as long as you're within the virtual memory limits you've set. (Although page tables do occupy real memory, so there is some effect.)
Memory mapping does not save you memory, really. When you mmap a file, the contents of the file still need to be read into memory in order for your program to use the data. The big advantage to mmap is that you don't have to futz around with allocating buffers and issuing read calls. Also, there is no need to copy data from the kernel buffer into which the file is read. So it can be a lot easier and more efficient, but not always; it depends a lot on the precise access pattern.
One thing to note: the following snippet does not do what the comment says it does:
// make sure file is big enough
if (lseek(*filedestination,size*sizeof(double), SEEK_SET) == -1) {
perror("seek to len failed");
exit(1);
}
lseek only sets the file position for the next read or write operation. If the file does not extend to that point, you'll get an EOF indication when you read, or the file will be extended (sparsely) if you write. So there's really not much point. If you want to check the file size, use stat. Or make sure you read at least one byte after doing the seek.
There's also not a lot of point using O_CREAT in the open call, since if the file doesn't exist and thus gets created, it will have size 0, which is presumably an error. Leaving O_CREAT off means the open call will fail if the file doesn't exist, which is likely what you want.
Finally, if you are not actually modifying the file contents, don't mmap with PROT_WRITE. PROT_READ pages are a lot easier for the kernel to deal with, because they can just be dropped and read back in later. (For writable pages, the kernel keeps track of the fact that the page has been modified, but if you aren't planning on writing and you don't allow modification, that makes the kernel's task a bit easier.)
Since you are (apparently) getting your process killed by the OOM killer, even though the memory you are using is MAP_SHARED (so never requires backing store -- it is auotmatically backed by the file), it would appear you are running your linux with no swap space, which is a bad idea if you have large mapped files like this, as it will cause processes to get killed whenever your resident memory approaches your physical memory. So the obvious solution would be to add a swap file -- even a small amount (1-2GB) will avoid the OOM killer problem. There are lots of tutorials online about how to add a swapfile in linux. You can look here or here or search for yourself.
If for some reason you don't want to add a swapfile, you may be able to reduce the frequency of getting killed by increasing the "swappiness" of your system -- this will cause the kernel to drop pages of your mmapped files more readily, thus reducing the likelyhood of getting into an OOM situation. You do this by increasing the vm.swappiness parameter either in your sysctl.conf file (for bootup) or by writing a new value to your /proc/sys/vm/swappiness file.
Assume that I have a page of memory that is read-only (e.g., set through mmap/mprotect). How do I modify one word (8 bytes) on this page at the lowest possible overhead?
Some context: I assume x86-64, Linux as my runtime environment. The modifications happen rarely but frequently enough so that I have to worry about overhead. The page is read only to protect some important data that must be read by the program frequently against rogue/illegal modifications. There are only few places that are allowed to modify the data on the page and I know all the locations of these places and the address of the page statically. The problem I'm trying to solve is protecting some data against memory safety bugs in the program with a few authorized places where I need to make modifications to the data. The modifications are not frequent but frequent enough so that several kernel-roundtrips (through system calls) are too costly.
So far, I thought of the following solutions:
mprotect
ptrace
shared memory
new system call
mprotect
mprotect(addr, 4096, PROT_WRITE | PROT_READ);
addr[12] = 0xc0fec0fe;
mprotect(addr, 4096, PROT_READ);
The mprotect solution is clean, simple, and straight-forward. Unfortunately, it involves two round trips into the kernel and will result in some overhead. In addition, the whole page will be writable during that time frame, allowing for some other thread to modify that memory area concurrently.
ptrace
Unfortunately, ptraceing yourself is no longer possible (as a ptraced-process needs to be stopped. So the solution is to fork, ptrace the child process, then use PTRACE_POKETEXT to write to the child processes memory.
This option has the drawback of spawning a parent process and will result in problems if the tracee uses multiple processes. The overhead per write is at least one system call for PTRACE plus the required synchronization between the processes.
shared memory
Shared memory is similar to the ptrace solution except that it reduces the system call. Both processes set up shared memory with different permissions (RW in the child, R in the parent). The two processes still need to synchronize on each write that is then carried out by the parent. Shared memory has similar drawbacks in complexity as the ptrace solution and incompatibilities with multiple communicating processes.
new system call
Adding a new system call to the kernel would solve the problem and would only require a single system call to modify one word in the process without having to change the page tables or the requirement to set up multiple communicating processes.
Is there anything that is faster than the 4 discussed/sketched solutions? Could I rely on any debug features? Are there any other neat low-level systems tricks?
If I wish to access data directly with virtual address from the other application, Can I access proc/[pid]/mem with virtual address? would the offset be the same as virtual address?
Code-wise, if I fseeko(the-proc/[pid]/mem-File, virtual_address, SEEK_SET);
and fread the amount from the virtual_address, Would I get the data from that virtual address of the application?
Thank you!
You cannot exactly do that. But I believe what you are looking for is shared memory. Any two processes can read and write to a common memory using shared memory. The virtual addresses for this shared memory between the processes might not be same. And it has to be accessed in controlled manner by the user, no natural locking mechanism is given for that.
This link from linuxgazette has a really good explanation with examples.
/proc/[pid] is just that, it holds the process information for the given PID. Remember that each process has its own PID which is stored globally by the linux kernel.
Here's an example of how to read from proc for the statm part of proc. This code could be easily adapted to /proc/[pid]/mem according to the appropriate manpage.
const char *statm_path = "/proc/[pid]/statm";
proc_f = fopen(statm_path, "r");
if (proc_f == NULL)
{
perror("Error opening proc file");
return;
}
// Writing the info from /proc/[pid]/statm to a struct.
if (7 != fscanf(proc_f, "%ld %ld %ld %ld %ld %ld %ld", &result.size,
&result.resident, &result.share, &result.text, &result.lib,
&result.data, &result.dt))
{
perror(statm_path);
return;
}
fclose(proc_f);
For the fields in /mem, I recommend this page. From here you can access any stats listed on the page. I don't think you could explicitly access particular variables or anything like that because each process has its own virtual memory space maintained individually by the kernel. At best it could get quite messy.
I'm on Linux 2.6 and I have a weird problem. I have 3 concurrent processes (forked from the same process) which need to obtain 3 DIFFERENT shared memory segments, one for each process. Each of the process executes this code (please note that 'message' type is user-defined)
message *m;
int fd = shm_open("message", O_CREAT|O_RDWR, S_IRUSR|S_IWUSR);
ftruncate(fd, sizeof(message));
m = mmap(NULL, sizeof(message), PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0);
char messagename[16];
snprintf(messagename, sizeof(messagename), "%p", m);
char path[32] = "/dev/shm/";
strcat(path, messagename);
rename("/dev/shm/message", path);
Let me explain a bit: I want every process to allocate a shared memory zone which contains a message. To make sure another process (the message receiver) can access the same shm, I then rename my shm file from "message" to a string named after the message pointer (this because the process which receives the message already knows the pointer).
When executing the program, though, I tried to print (for debugging purpose) the pointers that every process received when mmapping the fd obtained with shm_open, and I noticed that all of them got the SAME pointer. How is it possible? I thought that maybe other processes did the shm_open() after the first one did and before it renamed the segment, so I also tried to make these lines of code an atomic operation by using a process shared mutex, but the problem persists.
I would really appreciate any kind of help or suggestion.
Your processes all started with identical address space layouts at the moment of forking, and then followed very similar code paths. It is therefore not surprising that they all end up with the same value of m.
However, once they became separate processes, their address spaces became independent, so having the same value of m does not imply that all of the ms are pointing to the same thing.
Furthermore, I am not sure that your idea of renaming the /dev/shm entry after creating the shared memory block is safe or portable. If you want each process's shared memory block to have a unique name, why not base the name on the process ID (which is guaranteed to be unique at a given point in time) and pass it directly to shm_open, rather than going to the bother of renaming it afterwards?
The same virtual address in different processes can (and usually does) map to different physical pages in memory. You might want to read the wikipedia article on virtual memory.
I solved a similar problem simply by making the mmap before forking. So after forking the same area is shared between all processes. I then put my semaphores and mutexes on defined positions. It works perfectly.
I've been trying to understand how to read the memory of other processes on Mac OS X, but I'm not having much luck. I've seen many examples online using ptrace with PEEKDATA and such, however it doesn't have that option on BSD [man ptrace].
int pid = fork();
if (pid > 0) {
// mess around with child-process's memory
}
How is it possible to read from and write to the memory of another process on Mac OS X?
Use task_for_pid() or other methods to obtain the target process’s task port. Thereafter, you can directly manipulate the process’s address space using vm_read(), vm_write(), and others.
Matasano Chargen had a good post a while back on porting some debugging code to OS X, which included learning how to read and write memory in another process (among other things).
It has to work, otherwise GDB wouldn't:
It turns out Apple, in their infinite wisdom, had gutted ptrace(). The OS X man page lists the following request codes:
PT_ATTACH — to pick a process to debug
PT_DENY_ATTACH — so processes can stop themselves from being debugged
[...]
No mention of reading or writing memory or registers. Which would have been discouraging if the man page had not also mentioned PT_GETREGS, PT_SETREGS, PT_GETFPREGS, and PT_SETFPREGS in the error codes section. So, I checked ptrace.h. There I found:
PT_READ_I — to read instruction words
PT_READ_D — to read data words
PT_READ_U — to read U area data if you’re old enough to remember what the U area is
[...]
There’s one problem solved. I can read and write memory for breakpoints. But I still can’t get access to registers, and I need to be able to mess with EIP.
I know this thread is 100 years old, but for people coming here from a search engine:
xnumem does exactly what you are looking for, manipulate and read inter-process memory.
// Create new xnu_proc instance
xnu_proc *Process = new xnu_proc();
// Attach to pid (or process name)
Process->Attach(getpid());
// Manipulate memory
int i = 1337, i2 = 0;
i2 = process->memory().Read<int>((uintptr_t)&i);
// Detach from process
Process->Detach();
It you're looking to be able to share chunks of memory between processes, you should check out shm_open(2) and mmap(2). It's pretty easy to allocate a chunk of memory in one process and pass the path (for shm_open) to another and both can then go crazy together. This is a lot safer than poking around in another process's address space as Chris Hanson mentions. Of course, if you don't have control over both processes, this won't do you much good.
(Be aware that the max path length for shm_open appears to be 26 bytes, although this doesn't seem to be documented anywhere.)
// Create shared memory block
void* sharedMemory = NULL;
size_t shmemSize = 123456;
const char* shmName = "mySharedMemPath";
int shFD = shm_open(shmName, (O_CREAT | O_EXCL | O_RDWR), 0600);
if (shFD >= 0) {
if (ftruncate(shFD, shmemSize) == 0) {
sharedMemory = mmap(NULL, shmemSize, (PROT_READ | PROT_WRITE), MAP_SHARED, shFD, 0);
if (sharedMemory != MAP_FAILED) {
// Initialize shared memory if needed
// Send 'shmemSize' & 'shmemSize' to other process(es)
} else handle error
} else handle error
close(shFD); // Note: sharedMemory still valid until munmap() called
} else handle error
...
Do stuff with shared memory
...
// Tear down shared memory
if (sharedMemory != NULL) munmap(sharedMemory, shmemSize);
if (shFD >= 0) shm_unlink(shmName);
// Get the shared memory block from another process
void* sharedMemory = NULL;
size_t shmemSize = 123456; // Or fetched via some other form of IPC
const char* shmName = "mySharedMemPath";// Or fetched via some other form of IPC
int shFD = shm_open(shmName, (O_RDONLY), 0600); // Can be R/W if you want
if (shFD >= 0) {
data = mmap(NULL, shmemSize, PROT_READ, MAP_SHARED, shFD, 0);
if (data != MAP_FAILED) {
// Check shared memory for validity
} else handle error
close(shFD); // Note: sharedMemory still valid until munmap() called
} else handle error
...
Do stuff with shared memory
...
// Tear down shared memory
if (sharedMemory != NULL) munmap(sharedMemory, shmemSize);
// Only the creator should shm_unlink()
You want to do Inter-Process-Communication with the shared memory method. For a summary of other commons method, see here
It didn't take me long to find what you need in this book which contains all the APIs which are common to all UNIXes today (which many more than I thought). You should buy it in the future. This book is a set of (several hundred) printed man pages which are rarely installed on modern machines.
Each man page details a C function.
It didn't take me long to find shmat() shmctl(); shmdt() and shmget() in it. I didn't search extensively, maybe there's more.
It looked a bit outdated, but: YES, the base user-space API of modern UNIX OS back to the old 80's.
Update: most functions described in the book are part of the POSIX C headers, you don't need to install anything. There are few exceptions, like with "curses", the original library.
I have definitely found a short implementation of what you need (only one source file (main.c)).
It is specially designed for XNU.
It is in the top ten result of Google search with the following keywords « dump process memory os x »
The source code is here
but from a strict point of virtual address space point de vue, you should be more interested with this question: OS X: Generate core dump without bringing down the process? (look also this)
When you look at gcore source code, it is quite complex to do this since you need to deal with treads and their state...
On most Linux distributions, the gcore program is now part of the GDB package. I think the OSX version is installed with xcode/the development tools.
UPDATE: wxHexEditor is an editor which can edit devices. IT CAN also edit process memory the same way it does for regular files. It work on all UNIX machines.
Manipulating a process's memory behind its back is a Bad Thing and is fraught with peril. That's why Mac OS X (like any Unix system) has protected memory, and keeps processes isolated from one another.
Of course it can be done: There are facilities for shared memory between processes that explicitly cooperate. There are also ways to manipulate other processes' address spaces as long as the process doing so has explicit right to do so (as granted by the security framework). But that's there for people who are writing debugging tools to use. It's not something that should be a normal — or even rare — occurrence for the vast majority of development on Mac OS X.
In general, I would recommend that you use regular open() to open a temporary file. Once it's open in both processes, you can unlink() it from the filesystem and you'll be set up much like you would be if you'd used shm_open. The procedure is extremely similar to the one specified by Scott Marcy for shm_open.
The disadvantage to this approach is that if the process that will be doing the unlink() crashes, you end up with an unused file and no process has the responsibility of cleaning it up. This disadvantage is shared with shm_open, because if nothing shm_unlinks a given name, the name remains in the shared memory space, available to be shm_opened by future processes.