[conspire] How much memory each process is taking up

Rick Moen rick at linuxmafia.com
Mon Feb 23 15:55:38 PST 2009


It occurs to me that many Linux users have absolutely no idea how to
fruitfully use the "ps" (process status) and "top" commands -- or others
-- to examine the use of memory.  So, I thought I'd show a few tricks.

You can get a stratospheric view of system memory using the "free"
command, or looking at the /proc/meminfo pseudofile:

rmoen at borgia:~$ free
             total       used       free     shared    buffers
cached
Mem:       2074864    2044428      30436          0      80300
1357888
-/+ buffers/cache:     606240    1468624
Swap:      1951888         88    1951800

rmoen at borgia:~$ cat /proc/meminfo 
MemTotal:      2074864 kB
MemFree:         30536 kB
Buffers:         80312 kB
Cached:        1357876 kB
SwapCached:         12 kB
Active:         589072 kB
Inactive:      1328784 kB
HighTotal:     1178136 kB
HighFree:         7400 kB
LowTotal:       896728 kB
LowFree:         23136 kB
SwapTotal:     1951888 kB
SwapFree:      1951800 kB
Dirty:               0 kB
Writeback:           0 kB
AnonPages:      479588 kB
Mapped:          59692 kB
Slab:           106924 kB
SReclaimable:   100592 kB
SUnreclaim:       6332 kB
PageTables:       1968 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
WritebackTmp:        0 kB
CommitLimit:   2989320 kB
Committed_AS:   718068 kB
VmallocTotal:   114680 kB
VmallocUsed:     44220 kB
VmallocChunk:    62964 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
HugePages_Surp:      0
Hugepagesize:     4096 kB
rmoen at borgia:~$ 


This is a system with 2GB physical RAM ("MemTotal: 2074864 kB").  The
fact that it reports "MemFree: 30536kB" (30 MB) is _not_ a bad thing:
Unix does its level best to make sure RAM is put to productive use.  If
there's nothing better for it to do, the RAM will get allocated to disk
cache, which is mostly what's happening in this case.

It's a Debian "squeeze" (new Debian testing track) P4 workstation with the
xdm display manager running the Window Maker window manager:
Post-installation, I reduced the number of "getty" processes from six to
two by commenting out lines in /etc/inittab[1], then disabling SysVInit
startup of the Avahi mDNS/DNS-SD daemon, the OpenBSD inetd, and the
kerneloops daemon, none of which I currently need.

(Running "telinit Q" makes the system init process re-parse
/etc/inittab, which is handy for putting into effect changes to that
file without rebooting.)

Likewise, the fact that the swap partition is actually in use (if
barely) does not indicate, in any way, system stress.  Some swap use
always happens if you provide swap to the kernel.  This happens to be an
edge case of a workstation with generous RAM that's running fairly
lightweight processes, hence is barely touching swap.  There are
effective ways to measure the "swappiness" of a system, e.g., using the
iostat utility, but "free" isn't really useful for that.

Getting down to the "ps" and "top" commands:  "ps" has a default output
that includes a number of default fields (columns) and omits others, but
you can easily customise its reporting, using command options.  Here's 
a custom "ps" report highlighting process memory stats:

$ ps -eo pid,user,group,%mem,rss,vsz,args

The option "e" is the SysV-style way of saying "include all processes".
Option "o" is "include user-specified columns" as a comma-separated
list.  That list then follows:

"pid" = the command's Process ID.
"user" = what user the process is running as.
"group" = what group the process is running as.
"%mem" = What percentage the process's resident set size (to be 
         explained below) is relative to total physical memory.
"rss" =  "the non-swapped physical memory that a task has used (in kiloBytes)".
"vsz" =  "virtual memory size of the process in KiB (1024-byte units)".
"arg" =  the process name plus all the arguments it started with.

In this case, we get:

ps -eo pid,user,group,%mem,rss,vsz,args 
  PID USER     GROUP    %MEM   RSS    VSZ COMMAND
    1 root     root      0.0   692   2100 init [2]  
    2 root     root      0.0     0      0 [kthreadd]
    3 root     root      0.0     0      0 [migration/0]
    4 root     root      0.0     0      0 [ksoftirqd/0]
    5 root     root      0.0     0      0 [watchdog/0]
    6 root     root      0.0     0      0 [migration/1]
    7 root     root      0.0     0      0 [ksoftirqd/1]
    8 root     root      0.0     0      0 [watchdog/1]
    9 root     root      0.0     0      0 [events/0]
   10 root     root      0.0     0      0 [events/1]
   11 root     root      0.0     0      0 [khelper]
   44 root     root      0.0     0      0 [kblockd/0]
   45 root     root      0.0     0      0 [kblockd/1]
   47 root     root      0.0     0      0 [kacpid]
   48 root     root      0.0     0      0 [kacpi_notify]
  121 root     root      0.0     0      0 [kseriod]
  160 root     root      0.0     0      0 [pdflush]
  161 root     root      0.0     0      0 [pdflush]
  162 root     root      0.0     0      0 [kswapd0]
  163 root     root      0.0     0      0 [aio/0]
  164 root     root      0.0     0      0 [aio/1]
  624 root     root      0.0     0      0 [ksuspend_usbd]
  625 root     root      0.0     0      0 [khubd]
  700 root     root      0.0     0      0 [ata/0]
  706 root     root      0.0     0      0 [ata/1]
  708 root     root      0.0     0      0 [ata_aux]
  788 root     root      0.0     0      0 [scsi_eh_0]
  789 root     root      0.0     0      0 [scsi_eh_1]
  932 root     root      0.0     0      0 [kjournald]
 1008 root     root      0.0  1532   2936 udevd --daemon
 1544 root     root      0.0     0      0 [kpsmoused]
 1776 root     root      0.0     0      0 [kjournald]
 1838 daemon   daemon    0.0   512   1892 /sbin/portmap
 1849 statd    root      0.0   724   1956 /sbin/rpc.statd
 2112 root     root      0.0     0      0 [kondemand/0]
 2114 root     root      0.0     0      0 [kondemand/1]
 2155 root     root      0.0  1516  28164 /usr/sbin/rsyslogd -c3
 2166 root     root      0.0   648   1764 /usr/sbin/acpid
 2176 103      108       0.0   888   2620 /usr/bin/dbus-daemon --system
 2223 root     root      0.1  2948   6544 /usr/sbin/cupsd
 2490 101      105       0.0   936   6296 /usr/sbin/exim4 -bd -q30m
 2526 daemon   daemon    0.0   440   2048 /usr/sbin/atd
 2546 root     root      0.0   988   3456 /usr/sbin/cron
 2569 root     root      0.0   920   5432 /usr/bin/xdm
 2575 root     root      0.0   504   1764 /sbin/getty 38400 tty1
 2578 root     root      0.0   504   1764 /sbin/getty 38400 tty2
 2593 root     root      1.8 37600 306540 /usr/bin/X :0 vt7 -nolisten tcp -auth /var/lib/xdm/authdir/authfiles/A:0-PoA
 2598 root     root      0.1  3752   6740 -:0         
 2619 rmoen    rmoen     0.0  1208   6816 /usr/lib/WindowMaker/WindowMaker
 2640 rmoen    rmoen     0.0   604   4756 /usr/bin/ssh-agent /usr/bin/dbus-launch --exit-with-session x-window-manager
 2643 rmoen    rmoen     0.0   672   3100 /usr/bin/dbus-launch --exit-with-session x-window-manager
 2644 rmoen    rmoen     0.0   860   2620 /usr/bin/dbus-daemon --fork --print-pid 5 --print-address 7 --session
 2662 rmoen    rmoen     0.2  5204   9856 /usr/lib/WindowMaker/WindowMaker --for-real
 2669 rmoen    rmoen     0.1  2228   5308 /usr/lib/libgconf2-4/gconfd-2
11
 2698 root     root      0.0   612   2180 dhclient3 -pf /var/run/dhclient.eth0.pid -lf /var/lib/dhcp3/dhclient.eth0.le
 2760 rmoen    rmoen     0.0  1256   3868 /bin/sh /usr/bin/icedove
 2772 rmoen    rmoen     0.0  1276   3908 /bin/sh /usr/lib/icedove/run-mozilla.sh /usr/lib/icedove/icedove-bin
 2777 rmoen    rmoen     6.1 127228 230032 /usr/lib/icedove/icedove-bin
 3526 rmoen    utmp      0.2  4372  10656 xterm
 3527 rmoen    rmoen     0.1  2944   5668 bash
 3538 rmoen    rmoen     0.0  1412   4364 /bin/sh /usr/local/bin/irc
 3539 rmoen    rmoen     0.2  4652   9692 irssi --port=7000 --connect=irc.example.com --nick=rmoen --password=hahyouwish
 3931 rmoen    rmoen    15.4 321364 440068 /usr/lib/iceweasel/firefox-bin
 5843 rmoen    utmp      0.2  4280  10648 xterm
 5844 rmoen    rmoen     0.1  2976   5672 bash
10055 rmoen    rmoen     0.1  2236   5260 ssh rick at 198.144.195.186
10102 rmoen    utmp      0.2  4864  11276 xterm
10103 rmoen    rmoen     0.1  2988   5680 bash
10150 root     root      0.0  1132   3772 su -
10151 root     root      0.0  1728   4228 -su
10327 root     root      0.0  1000   3716 ps -eo pid,user,group,%mem,rss,vsz,args

So, before I go further, "resident set size" (RSS) and "virtual size"
(VSZ) are crucial concepts in understanding RAM usage.  RSS is the
active parts of an executable -- allocated pages of RAM being actively
used within the process's RAM segment -- as opposed to portions of the
program that were never loaded into RAM in the first place, or portions
that used to be present but were swapped out to disk by the kernel.  (A
"page" is the smallest contiguous block of memory that the kernel can
allocate for storing data and code.)

The VSZ is the total "virtual" size of the process and its shared
libraries:  resident in-use memory, currently used swap (swapped-out
portions of code or data), memory-mapped files, and portions of the
program that reside currently only on-disk and have not been loaded.
Basically, physical RAM gets taken up by loaded-from-disk executable
code, plus stack size, plus data roaded into RAM.  Those comprise the
RSS.  RSS plus pages currently in swap equals VSZ.

There's a third crucial concept:  shared memory size (SHR), but we'll
encounter that only when we look at "top".


Notice the first process, init.  It has process ID 1; in fact, that's
one of its defining characteristics:  When your system starts, it loads
the kernel as process number zero.  The kernel forks, and the new
process (#1) execs "/bin/init".  init then starts up the rest of your
system.  

All of the couple-dozen processes in brackets are kernel service
processes.  To my knowledge, there's not much you can (or should) do
about deciding which of those to run.

udevd is the 2.6-kernel-ish daemon that manages what device nodes exist
on your system.  Notice it has an RSS of about 1.5 megs.  

I waste about 1.2MB on two service daemons for NFS (portmapper and
nfs.statd).  

rsyslogd (1.5 MB RSS) is a modern revamp of the original BSD system
logging daemon.  It's an essential system feature.  Like syslog-ng, 
it supports across networks (optionally), and encryption.

acpid (1/2 MB RSS) listens for system hardware events (ACPI = Advanced
Control Peripheral Interfae) and can take actions based on them.  Looks
like the only things acpid is listening for (/etc/acpi/events/*) are 
"power button" signals to shut off system power upon shutdown.  I might
consider disabling acpid, as that seems a bit stupid to waste half a meg
of RAM on.  Of course, ACPI can potentially do many other things,
including taking scripted actions when you press particular buttons on
the keyboard, and lots of other things.

dbus-daemon is an ugly GNOME-ism that's been allowed to worm its way
into Linux system space, D-Bus being a system facility, running both as
a system daemon and as a per-user daemon, and a related library
(libdbus) to manage dynamic system "events" such as adding/removing a
printer or adding/removing casual storage (e.g., flash drives), and
providing a general inter-process communication mechanism for talking
about system "events" between apps.

I didn't actually want an overengineered "middleware layer" created by a
bunch of desktop-software types, but that's what D-Bus is -- and it's
the camel's nose in the tent for HAL
(http://www.freedesktop.org/wiki/Software/hal), too.  Since I actually
don't _want_ something rummaging around in the background mounting and
umounting drives for me, I will probably disable the damned thing.
Meanwhile, at least it's reasonably small, about 2.5 MB scattered across
four processes (including the dbus-launch stuff).

cupds is the CUPS (Common Unix Printing System) daemon, which I need.
3 MB RSS.

exim4 is the system stub MTA for handling local mail, e.g., reports
generated for the admin user.  1MB RSS.

atd is for timed future execution of one-time (as opposed to recurring)
processes.  1/2 MB RSS.  crond runs recurring processes, 1 MB RSS.

xdm is the old-school display manager, providing graphical login.  1MB
RSS.  

Two getty ("get tty") processes furnish virtual consoles.  You need
minimum one.  1/2 MB RSS each.

The next process, "X", is the one where you have to be wary about
overinterpreting memory-usage stats, because physical RAM on the video
card is included.  So, the 37MB of RSS reported is vastly overstated.
How much _real_ RSS, then?  I'm not sure.

Process "-:0" is somehow related to the "session" of the X server that
the window manager is talking to.  There's some X Window System arcana
that I've largely forgotten relating to ":0" (session #0) or ":1"
(session #1), etc.  Like, you can start up a second instance on a
different virtual console, like this:

startx -- :1 &

Process WindowMaker is of course my favourite window manager, a mere 1.2
MB of RSS.

Process ssh-agent is a nice little tool to hold private SSH keys and
used for authentication.  I haven't yet looked into how it's set up on
this workstation.  1/2MB RSS.

Process gconfd-2 is a poxy bit of GNOME that got implicitly started by
the Debian build of Window Maker.  I probably cannot get rid of it
unless I compile Window Maker locally without "GNOME support" hooks.
2 MB of RSS wasted (almost twice as big as my window manager).

Process dhclient3 is, of course, the DHCP client, accepting "leased" IP
addresses and related information.

Finally, we get to serious memory usage:

"icedove" (Mozilla Thunderbird):  about 130 MB of RSS across three
processes.

"iceweasel" (Mozilla Firefox):  320 MB of RSS.

And there are a few miscellaneous proceses:  Each instance of bash is 
about 3 MB of RSS, xterm is 4-5 MB, and my preferred IRC client (irssi)
is 6MB including the little shell script that runs it.


Moving along:  "top" defaults to a dynamic display of processes,
refreshing every second and sorting by percentage of CPU consumed.
Once it's doing that, typing "M" will make top re-sort by percentage of
_memory_ consumed, and "c" will toggle between program name (default)
and command-line.  

If you just want to see the output once (instead of refreshed every
second), and want command lines (instead of program names), you can do:

# top -b -n 1 -c
top - 15:34:58 up 7 days,  1:10,  5 users,  load average: 0.04, 0.01,
0.00
Tasks:  74 total,   1 running,  73 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.4%us,  0.1%sy,  0.0%ni, 99.5%id,  0.0%wa,  0.0%hi,  0.0%si,
0.0%st
Mem:   2074864k total,  1991960k used,    82904k free,    85852k buffers
Swap:  1951888k total,       84k used,  1951804k free,  1330932k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND 
    1 root      20   0  2100  692  588 S    0  0.0   0:04.48 init [2]
    2 root      15  -5     0    0    0 S    0  0.0   0:00.00 [kthreadd]
    3 root      RT  -5     0    0    0 S    0  0.0   0:00.38 [migration/0]
    4 root      15  -5     0    0    0 S    0  0.0   0:01.98 [ksoftirqd/0]
    5 root      RT  -5     0    0    0 S    0  0.0   0:00.00 [watchdog/0]
    6 root      RT  -5     0    0    0 S    0  0.0   0:00.00 [migration/1]
    7 root      15  -5     0    0    0 S    0  0.0   0:01.82 [ksoftirqd/1]
    8 root      RT  -5     0    0    0 S    0  0.0   0:00.00 [watchdog/1]
    9 root      15  -5     0    0    0 S    0  0.0   0:15.72 [events/0]
   10 root      15  -5     0    0    0 S    0  0.0   0:10.56 [events/1]
   11 root      15  -5     0    0    0 S    0  0.0   0:00.00 [khelper]
   44 root      15  -5     0    0    0 S    0  0.0   0:00.48 [kblockd/0]
   45 root      15  -5     0    0    0 S    0  0.0   0:00.32 [kblockd/1]
   47 root      15  -5     0    0    0 S    0  0.0   0:00.00 [kacpid]
   48 root      15  -5     0    0    0 S    0  0.0   0:00.00 [kacpi_notify]
  121 root      15  -5     0    0    0 S    0  0.0   0:00.00 [kseriod]
  160 root      20   0     0    0    0 S    0  0.0   0:00.22 [pdflush]
  161 root      20   0     0    0    0 S    0  0.0   0:04.02 [pdflush]
  162 root      15  -5     0    0    0 S    0  0.0   0:01.16 [kswapd0]
  163 root      15  -5     0    0    0 S    0  0.0   0:00.00 [aio/0]
  164 root      15  -5     0    0    0 S    0  0.0   0:00.00 [aio/1]
  624 root      15  -5     0    0    0 S    0  0.0   0:00.38 [ksuspend_usbd]
  625 root      15  -5     0    0    0 S    0  0.0   0:00.06 [khubd]
  700 root      15  -5     0    0    0 S    0  0.0   0:00.00 [ata/0]
  706 root      15  -5     0    0    0 S    0  0.0   0:00.00 [ata/1]
  708 root      15  -5     0    0    0 S    0  0.0   0:00.00 [ata_aux]
  788 root      15  -5     0    0    0 S    0  0.0   0:00.00 [scsi_eh_0]
  789 root      15  -5     0    0    0 S    0  0.0   0:00.00 [scsi_eh_1]
  932 root      15  -5     0    0    0 S    0  0.0   0:06.66 [kjournald]
 1008 root      16  -4  2936 1532  488 S    0  0.1   0:00.54 udevd --daemon
 1544 root      15  -5     0    0    0 S    0  0.0   0:00.00 [kpsmoused]
 1776 root      15  -5     0    0    0 S    0  0.0   0:00.54 [kjournald]
 1838 daemon    20   0  1892  512  416 S    0  0.0   0:00.00 /sbin/portmap
 1849 statd     20   0  1956  724  624 S    0  0.0   0:00.00 /sbin/rpc.statd
 2112 root      15  -5     0    0    0 S    0  0.0   0:00.00 [kondemand/0]
 2114 root      15  -5     0    0    0 S    0  0.0   0:00.00 [kondemand/1]
 2155 root      20   0 28164 1516  948 S    0  0.1   0:00.47 /usr/sbin/rsyslogd -c3
 2166 root      20   0  1764  648  548 S    0  0.0   0:00.00 /usr/sbin/acpid
 2176 messageb  20   0  2620  888  704 S    0  0.0   0:00.00 /usr/bin/dbus-daemon --system
 2223 root      20   0  6544 2948 2168 S    0  0.1   0:01.02 /usr/sbin/cupsd
 2490 Debian-e  20   0  6296  936  616 S    0  0.0   0:00.14 /usr/sbin/exim4 -bd -q30m
 2526 daemon    20   0  2048  440  316 S    0  0.0   0:00.00 /usr/sbin/atd
 2546 root      20   0  3456  988  804 S    0  0.0   0:02.42 /usr/sbin/cron
 2569 root      20   0  5432  920  612 S    0  0.0   0:00.00 /usr/bin/xdm
 2575 root      20   0  1764  504  436 S    0  0.0   0:00.00 /sbin/getty 38400 tty1
 2578 root      20   0  1764  504  436 S    0  0.0   0:00.00 /sbin/getty 38400 tty2
2593 root      20   0  477m  37m  11m S    0  1.8  28:49.27 /usr/bin/X :0 vt7 -nolisten tcp -auth /var/lib/xdm/authdir/authfiles/A:0-PoAm0q
 2598 root      20   0  6740 3752 2908 S    0  0.2   0:00.48 -:0
 2619 rmoen     20   0  6816 1208  976 S    0  0.1   0:00.12 /usr/lib/WindowMaker/WindowMaker
 2640 rmoen     20   0  4756  604  304 S    0  0.0   0:00.78 /usr/bin/ssh-agent /usr/bin/dbus-launch --exit-with-session x-window-manager
 2643 rmoen     20   0  3100  672  456 S    0  0.0   0:00.00 /usr/bin/dbus-launch --exit-with-session x-window-manager
 2644 rmoen     20   0  2620  860  696 S    0  0.0   0:00.00 /usr/bin/dbus-daemon --fork --print-pid 5 --print-address 7 --session
 2662 rmoen     20   0  9856 5204 3276 S    0  0.3   4:30.69 /usr/lib/WindowMaker/WindowMaker --for-real
 2669 rmoen     20   0  5308 2228 1856 S    0  0.1   0:01.12 /usr/lib/libgconf2-4/gconfd-2 11
 2698 root      18  -2  2180  612  488 S    0  0.0   0:00.00 dhclient3 -pf /var/run/dhclient.eth0.pid -lf /var/lib/dhcp3/dhclient.eth0.leases eth0
 2760 rmoen     20   0  3868 1256 1084 S    0  0.1   0:00.00 /bin/sh /usr/bin/icedove
 2772 rmoen     20   0  3908 1276 1064 S    0  0.1   0:00.00 /bin/sh /usr/lib/icedove/run-mozilla.sh /usr/lib/icedove/icedove-bin
 2777 rmoen     20   0  225m 124m  25m S    0  6.1  26:35.85 /usr/lib/icedove/icedove-bin
 3526 rmoen     20   0 10656 4372 2404 S    0  0.2   0:04.34 xterm
 3527 rmoen     20   0  5668 2944 1404 S    0  0.1   0:00.12 bash
 3538 rmoen     20   0  4364 1412  996 S    0  0.1   0:00.00 /bin/sh /usr/local/bin/irc
 3539 rmoen     20   0  9692 4652 3368 S    0  0.2   1:19.65 irssi --port=7000 --connect=irc.example.com --nick=rmoen --password=hahyouwish
 3931 rmoen     20   0  430m 313m  23m S    0 15.5 132:53.25 /usr/lib/iceweasel/firefox-bin
 5843 rmoen     20   0 10648 4284 2400 S    0  0.2   1:05.20 xterm
 5844 rmoen     20   0  5672 2976 1432 S    0  0.1   0:00.20 bash
10055 rmoen     20   0  5260 2236 1808 S    0  0.1   0:02.74 ssh rick at 198.144.195.186
10102 rmoen     20   0 12036 5752 2400 S    0  0.3   0:04.02 xterm
10103 rmoen     20   0  5680 2988 1436 S    0  0.1   0:00.40 bash
10601 root      20   0  2388 1016  800 R    0  0.0   0:00.04 top -b -n 1 -c            
$

That's "-c" for the command-line display, "-b" for batch, and "-n 1" for 
display once.

Notice that batch mode makes "top" default to Process ID order.

The other main thing to note is that (aside from %MEM) there are now _three_ 
memory figures:  VIRT  RES  SHR

VIRT is VSZ again, just as RES is RSS.  The new one is SHR.  Manpage says:
"Shared Mem size (kb).  The amount of shared memory used by a task.  It
simply reflects memory that could be potentially shared with other
processes."  This concept is important for applications like
OpenOffice.org or Mozilla-family applications that rely on core libs
that will be shared among multiple instances of the app and its
relatives.  For example, the two icedove-bin (Mozilla Thunderbird)
processes shown have RSS ("RES") of 1.2MB and 124MB, respectively, but
also SHR figures of 1MB and 25MB, respectively.  A third process,
"icedove", has figures almost identical to the first icedove-bin.

The point?  About a meg of code is almost certainly near-100% common
among all three processes, reducing the real memory consumption by that
amount for two of the three.  That's not a huge effect in Thunderbird's
case, but OpenOffice.org shows it in spades.

I hope the above is handy in better assessing where your RAM is _really_
going.





More information about the conspire mailing list