Articles

News

Vendors

    Whitepapers

    Write for Us

    About Us

    It’s the Little Things

    vmstat

    One of those *nix commands that gets no respect is vmstat. However, it can be an extremely useful command, particularly for HPC. Vmstat reports Linux system virtual memory statistics. Although it has several “modes,” I find the default mode to be extremely useful. Listing 2 is a quick snapshot of a Linux laptop.

    Listing 2: vmstat on a Laptop

    [laytonjb@laytonjb-Lenovo-G50-45 ~]$ vmstat 1 5
    procs  -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
     r  b   swpd    free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
     1  0      0 5279852   2256 668972    0    0  1724    25  965 1042 17  9 71  2  0
     1  0      0 5269008   2256 669004    0    0     0     0 2667 1679 28  3 69  0  0
     1  0      0 5260976   2256 669004    0    0     0   504 1916  933 25  1 74  0  0
     2  0      0 5266288   2256 668980    0    0     0    36 4523 2941 29  4 67  0  0
     0  0      0 5276056   2256 668960    0    0     4     4 9104 6262 36  5 58  0  0

    Each line of output corresponds to a system snapshot at a particular time (Table 1), and you can control the amount of time between snapshots. The first line of numbers are the metrics since the system was rebooted. Lines of output after that correspond to their current values. A number of system metrics are very important. The first thing is to look at is the number of processes (r and b). If these numbers start moving up, something unusual might be happening on the node, such as processes waiting for run time or sleeping.

    Table 1: vmstat Output

    vmstat Column Meaning
    procs  
      b No. of processes waiting for run time
      r No. of processes in uninterruptible sleep
    memory  
      swpd Amount of virtual memory used
      free Amount of idle memory
      buff Amount of memory used as buffers
      cache Amount of memory used as cache
    swap  
      si Amount of memory swapped in from disk (blocks/sec)
      so Amount of memory swapped out to disk (blocks/sec)
    io  
      bi No. of blocks received from a block device (blocks/sec)
      bo No. of blocks sent to a block device (blocks/sec)
    system  
      in No. of interrupts per second, including the clock
      cs No. of context switches per second
    cpu  
      us Time spent running non-kernel code (=user time + nice time)
      sy Time spent running kernel code (=system time)
      id Time spent idle
      wa Time spent waiting for I/O
      st Time stolen from a virtual machine

    The metrics listed as memory can be useful, particularly as the kernel grabs and releases memory. You shouldn't be too worried about these values unless the values in the next section (swap), are non-zero. If you see non-zero si and so values, excluding the first row, you should be concerned, because it indicates that the system is swapping, and swapping memory to disk and can really kill performance. If a user is complaining about performance and you see a node running really slowly with a very large load, then it’s a good possibility that the node is swapping.

    The metrics listed in the io section are also good to watch. They list either blocks sent to a block device or blocks received from a block device. If these numbers are both large, the application running on the nodes is likely doing something unusual by reading and writing to the device at the same time. This situation too can hurt performance.

    The other metrics can be very useful, but I tend to focus on those mentioned first before scanning the others. You can also send this data to a file for later postprocessing or plotting – for example, for debugging user problems on nodes.