Multiple apache root processes - apache2

Question

I noticed today that when making requests from our web server, things were rather slow.
I started looking into it and I've found a load of root owned apache processes.
I don't know for sure that this is actually what's causing things to be slow, but none the less, it doesn't look good.
problem is, I don't know what to do from here?
How do I find out why there are so many root processes?
Could some recommend a set of tests? I've tried stracing a few of them, and they appear to be doing something, but the output of strace is beyond me.
root 30918 1.8 1.3 84284 52296 ? Ss 14:11 0:01 /usr/sbin/apache2 -k restart
root 30919 0.0 1.1 84420 45612 ? S 14:11 0:00 /usr/sbin/apache2 -k restart
root 30920 0.0 1.1 84420 45604 ? S 14:11 0:00 /usr/sbin/apache2 -k restart
root 30921 0.0 1.1 84420 45612 ? S 14:11 0:00 /usr/sbin/apache2 -k restart
root 30922 0.1 1.1 84420 45612 ? S 14:11 0:00 /usr/sbin/apache2 -k restart
root 30923 0.0 1.1 84420 45612 ? S 14:11 0:00 /usr/sbin/apache2 -k restart
www-data 30926 6.6 1.5 104964 61336 ? S 14:12 0:03 /usr/sbin/apache2 -k restart
root 30930 0.1 1.1 84420 45616 ? S 14:12 0:00 /usr/sbin/apache2 -k restart
root 30933 0.0 1.1 84420 45616 ? S 14:12 0:00 /usr/sbin/apache2 -k restart
root 30935 0.0 1.1 84420 45616 ? S 14:12 0:00 /usr/sbin/apache2 -k restart
root 30936 0.0 1.1 84420 45616 ? S 14:12 0:00 /usr/sbin/apache2 -k restart
root 30937 0.0 1.1 84420 45616 ? S 14:12 0:00 /usr/sbin/apache2 -k restart
root 30938 0.0 1.1 84420 45616 ? S 14:12 0:00 /usr/sbin/apache2 -k restart
root 30961 0.0 1.1 84420 45612 ? S 14:12 0:00 /usr/sbin/apache2 -k restart
root 30989 0.0 1.1 84420 45612 ? S 14:12 0:00 /usr/sbin/apache2 -k restart
root 30990 0.0 1.1 84420 45612 ? S 14:12 0:00 /usr/sbin/apache2 -k restart
root 31011 0.1 1.1 84420 45612 ? S 14:12 0:00 /usr/sbin/apache2 -k restart
root 31013 0.1 1.1 84420 45612 ? S 14:12 0:00 /usr/sbin/apache2 -k restart
root 31014 0.0 1.1 84420 45612 ? S 14:12 0:00 /usr/sbin/apache2 -k restart
www-data 31175 2.5 1.5 104168 60524 ? S 14:12 0:00 /usr/sbin/apache2 -k restart
www-data 31189 2.3 1.4 102360 58920 ? S 14:12 0:00 /usr/sbin/apache2 -k restart
www-data 31190 1.5 1.4 101904 58356 ? S 14:12 0:00 /usr/sbin/apache2 -k restart
www-data 31191 0.3 1.1 84556 46760 ? S 14:12 0:00 /usr/sbin/apache2 -k restart
www-data 31192 1.4 1.4 101916 58384 ? S 14:12 0:00 /usr/sbin/apache2 -k restart
www-data 31193 1.5 1.4 101916 58376 ? S 14:12 0:00 /usr/sbin/apache2 -k restart
root 31240 0.1 1.1 84420 45612 ? S 14:12 0:00 /usr/sbin/apache2 -k restart
This is an example of the output from strace from one of the processes.
--- SIGCHLD (Child exited) # 0 (0) ---
read(6, 0xff87f6ef, 1) = -1 EAGAIN (Resource temporarily unavailable)
getuid32() = 0
close(17) = 0
gettimeofday({1354109303, 670988}, NULL) = 0
semop(5668864, {{0, -1, SEM_UNDO}}, 1) = 0
accept(4, {sa_family=AF_INET, sin_port=htons(48107), sin_addr=inet_addr("192.168.16.12")}, [16]) = 17
fcntl64(17, F_GETFD) = 0
fcntl64(17, F_SETFD, FD_CLOEXEC) = 0
semop(5668864, {{0, 1, SEM_UNDO}}, 1) = 0
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0xf74a2768) = 1949
waitpid(1949, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0) = 1949
--- SIGCHLD (Child exited) # 0 (0) ---
read(6, 0xff87f6ef, 1) = -1 EAGAIN (Resource temporarily unavailable)
getuid32() = 0
close(17) = 0
gettimeofday({1354109305, 724358}, NULL) = 0
semop(5668864, {{0, -1, SEM_UNDO}}, 1) = 0
accept(4, {sa_family=AF_INET, sin_port=htons(48132), sin_addr=inet_addr("192.168.16.12")}, [16]) = 17
fcntl64(17, F_GETFD) = 0
fcntl64(17, F_SETFD, FD_CLOEXEC) = 0
semop(5668864, {{0, 1, SEM_UNDO}}, 1) = 0
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0xf74a2768) = 1974
waitpid(1974, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0) = 1974
--- SIGCHLD (Child exited) # 0 (0) ---
I've disabled all of the modules in mods-enabled except for essential ones like auth, env, siteenv and alias and started the server. In this case I still get 6 root apache processes and 1 www-data owned apache process.
I've made sure all the modules are up2date.
There are no obvious errors in the logs.
config follow;
ServerRoot "/etc/apache2"
LockFile /var/lock/apache2/accept.lock
PidFile ${APACHE_PID_FILE}
Timeout 300
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 15
<IfModule mpm_worker_module>
StartServers 2
MaxClients 150
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 25
MaxRequestsPerChild 0
</IfModule>
User ${APACHE_RUN_USER}
Group ${APACHE_RUN_GROUP}
AccessFileName .htaccess
<Files ~ "^\.ht">
Order allow,deny
Deny from all
</Files>
DefaultType text/plain
HostnameLookups Off
ErrorLog /var/log/apache2/error.log
LogLevel warn
Include /etc/apache2/mods-enabled/*.load
Include /etc/apache2/mods-enabled/*.conf
Include /etc/apache2/httpd.conf
Include /etc/apache2/ports.conf
LogFormat "%v:%p %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" vhost_combined
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %b" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent
CustomLog /var/log/apache2/other_vhosts_access.log vhost_combined
Include /etc/apache2/conf.d/
Include /etc/apache2/sites-enabled/
The compiled in modules are:
Compiled in modules:
core.c
mod_log_config.c
mod_logio.c
itk.c
http_core.c
mod_so.c
So I'm only running the mpm_worker config now.
DEBUG UPDATER
When I restart apache, and ps, I get something like this;
root 26921 0.5 1.3 80008 52452 ? Ss 21:27 0:02 /usr/sbin/apache2 -k start
root 27114 0.0 1.1 80144 44804 ? S 21:34 0:00 /usr/sbin/apache2 -k start
root 27115 0.0 1.1 80144 44820 ? S 21:34 0:00 /usr/sbin/apache2 -k start
root 27116 0.0 1.1 80144 44804 ? S 21:34 0:00 /usr/sbin/apache2 -k start
root 27117 0.0 1.1 80144 44804 ? S 21:34 0:00 /usr/sbin/apache2 -k start
root 27119 0.0 1.1 80144 44804 ? S 21:34 0:00 /usr/sbin/apache2 -k start
If I put LogLevel to debug and restart, then I see these messages from mod_proxy
[Thu Nov 29 21:34:01 2012] [info] Server built: Sep 9 2012 21:17:36
[Thu Nov 29 21:34:01 2012] [debug] itk.c(1100): AcceptMutex: sysvsem (default: sysvsem)
[Thu Nov 29 21:34:01 2012] [debug] proxy_util.c(1818): proxy: grabbed scoreboard slot 0 in child 27115 for worker proxy:reverse
[Thu Nov 29 21:34:01 2012] [debug] proxy_util.c(1818): proxy: grabbed scoreboard slot 0 in child 27114 for worker proxy:reverse
[Thu Nov 29 21:34:01 2012] [debug] proxy_util.c(1934): proxy: initialized single connection worker 0 in child 27115 for (*)
[Thu Nov 29 21:34:01 2012] [debug] proxy_util.c(1837): proxy: worker proxy:reverse already initialized
[Thu Nov 29 21:34:01 2012] [debug] proxy_util.c(1934): proxy: initialized single connection worker 0 in child 27114 for (*)
[Thu Nov 29 21:34:01 2012] [debug] proxy_util.c(1818): proxy: grabbed scoreboard slot 0 in child 27117 for worker proxy:reverse
[Thu Nov 29 21:34:01 2012] [debug] proxy_util.c(1837): proxy: worker proxy:reverse already initialized
[Thu Nov 29 21:34:01 2012] [debug] proxy_util.c(1934): proxy: initialized single connection worker 0 in child 27117 for (*)
[Thu Nov 29 21:34:01 2012] [debug] proxy_util.c(1818): proxy: grabbed scoreboard slot 0 in child 27119 for worker proxy:reverse
[Thu Nov 29 21:34:01 2012] [debug] proxy_util.c(1837): proxy: worker proxy:reverse already initialized
[Thu Nov 29 21:34:01 2012] [debug] proxy_util.c(1934): proxy: initialized single connection worker 0 in child 27119 for (*)
[Thu Nov 29 21:34:01 2012] [debug] proxy_util.c(1818): proxy: grabbed scoreboard slot 0 in child 27116 for worker proxy:reverse
[Thu Nov 29 21:34:01 2012] [debug] proxy_util.c(1837): proxy: worker proxy:reverse already initialized
[Thu Nov 29 21:34:01 2012] [debug] proxy_util.c(1934): proxy: initialized single connection worker 0 in child 27116 for (*)
[Thu Nov 29 21:36:20 2012] [notice] SIGHUP received. Attempting to restart
Notice the pids match. However, if I disable mod_proxy, then these message disappear, but I still get the same number of root processes starting, so I believe this is a symptom not a cause.

Answer 1

This is absolutely normal for Apache. Each process processes one request at a time. So if there was only one process (it is called worker) then it would be really slow if there are lots of users.
The issue I see is that these should not be root owned processes. Depending on your platform it should have it's own user. Like in Debian user would be www-data. Then only one process would be owned by root and rest would be owned by that user.
However speed is defined by several factors - hardware, web server, and web application.
Make sure that hardware you are running on fits requirements (enough ram and CPU)
Lower number of workers in case of poor hardware capabilities or increase if it is super good.
Make sure that web application (if there's is one, and often it is php app) is not a bottleneck for performance.
PS: sorry for poor formatting, typed clamsily from phone.

Answer 2

Know I'm a bit late to the game but I ran into the same issue and was going nuts trying to figure out what's going on. I'm on apache 2.4.7 so a bit newer than you but the general premise is the same.
I had to look in /etc/apache2/mods-enabled/mpm_prefork.conf to find my mpm configuration but you have it right here:
<IfModule mpm_worker_module>
StartServers 2
MaxClients 150
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 25
MaxRequestsPerChild 0
</IfModule>
Looks like a valid config, which it is. However, your MaxRequestsPerChild, like mine was, is set to 0. I've adjusted it to approximately 10 (can probably go higher but am just testing now) and I think that's solved my problem. Hope this helps!