carlessanagustin
11/17/2015 - 11:33 AM

Troubleshooting linux MEMORY usage

Troubleshooting linux MEMORY usage

Troubleshooting linux MEMORY usage

This entry will hopefully show you how to troubleshoot memory used by Linux and application.

Linux System Memory usage

Display amount of free and used memory in the system (-m = in megas)

$ free -m
             total       used       free     shared    buffers     cached
Mem:           993        249        744          0         15        108
-/+ buffers/cache:        124        869
Swap:            0          0          0

Taken from...

$ cat /proc/meminfo
MemTotal:        1017816 kB
MemFree:          760000 kB
Buffers:           17060 kB
Cached:           112728 kB
SwapCached:            0 kB
Active:           125712 kB
Inactive:          93112 kB
Active(anon):      89136 kB
Inactive(anon):      584 kB
Active(file):      36576 kB
Inactive(file):    92528 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:         89032 kB
Mapped:            10820 kB
Shmem:               688 kB
Slab:              23552 kB
SReclaimable:      15548 kB
SUnreclaim:         8004 kB
KernelStack:         704 kB
PageTables:         2696 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:      508908 kB
Committed_AS:     173068 kB
VmallocTotal:   34359738367 kB
VmallocUsed:        8488 kB
VmallocChunk:   34359724028 kB
HardwareCorrupted:     0 kB
AnonHugePages:      2048 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:       32704 kB
DirectMap2M:     1015808 kB
  • MemTotal — Total amount of physical RAM, in kilobytes.
  • MemFree — The amount of physical RAM, in kilobytes, left unused by the system.
  • Buffers — The amount of physical RAM, in kilobytes, used for file buffers.
  • Cached — The amount of physical RAM, in kilobytes, used as cache memory.
  • SwapCached — The amount of swap, in kilobytes, used as cache memory.
  • Active — The total amount of buffer or page cache memory, in kilobytes, that is in active use. This is memory that has been recently used and is usually not reclaimed for other purposes.
  • Inactive — The total amount of buffer or page cache memory, in kilobytes, that are free and available. This is memory that has not been recently used and can be reclaimed for other purposes.
  • HighTotal and HighFree — The total and free amount of memory, in kilobytes, that is not directly mapped into kernel space. The HighTotal value can vary based on the type of kernel used.
  • LowTotal and LowFree — The total and free amount of memory, in kilobytes, that is directly mapped into kernel space. The LowTotal value can vary based on the type of kernel used.
  • SwapTotal — The total amount of swap available, in kilobytes.
  • SwapFree — The total amount of swap free, in kilobytes.
  • Dirty — The total amount of memory, in kilobytes, waiting to be written back to the disk. Writeback — The total amount of memory, in kilobytes, actively being written back to the disk.
  • Mapped — The total amount of memory, in kilobytes, which have been used to map devices, files, or libraries using the mmap command.
  • Slab — The total amount of memory, in kilobytes, used by the kernel to cache data structures for its own use.
  • Committed_AS — The total amount of memory, in kilobytes, estimated to complete the workload. This value represents the worst case scenario value, and also includes swap memory. PageTables — The total amount of memory, in kilobytes, dedicated to the lowest page table level.
  • VMallocTotal — The total amount of memory, in kilobytes, of total allocated virtual address space.
  • VMallocUsed — The total amount of memory, in kilobytes, of used virtual address space.
  • VMallocChunk — The largest contiguous block of memory, in kilobytes, of available virtual address space.
  • HugePages_Total — The total number of hugepages for the system. The number is derived by dividing Hugepagesize by the megabytes set aside for hugepages specified in /proc/sys/vm/hugetlb_pool. This statistic only appears on the x86, Itanium, and AMD64 architectures.
  • HugePages_Free — The total number of hugepages available for the system. This statistic only appears on the x86, Itanium, and AMD64 architectures.
  • Hugepagesize — The size for each hugepages unit in kilobytes. By default, the value is 4096 KB on uniprocessor kernels for 32 bit architectures. For SMP, hugemem kernels, and AMD64, the default is 2048 KB. For Itanium architectures, the default is 262144 KB. This statistic only appears on the x86, Itanium, and AMD64 architectures.

Flushing the cache

$ echo 1 > /proc/sys/vm/drop_caches

Write cache to disk

$ sync

Finding linux process memory usage

$ htop
  • RSS : Stands for Resident Set Size and displays the amount of mem ory used by the process.
  • SHR : Stands for Shared memory and displays the memory shared with other processes.

In order to find the exact memory usage you can you use this tool:

$ git clone https://github.com/pixelb/ps_mem.git
$ cd ps_mem
$ sudo ./ps_mem.py
Private  +   Shared  =  RAM used       Program

34.6 MiB +   1.0 MiB =  35.7 MiB       gnome-terminal
139.8 MiB +   2.3 MiB = 142.1 MiB       firefox
291.8 MiB +   2.5 MiB = 294.3 MiB       gnome-shell
272.2 MiB +  43.9 MiB = 316.1 MiB       chrome (12)
913.9 MiB +   3.2 MiB = 917.1 MiB       thunderbird
---------------------------------
1.9 GiB
=================================

Troubleshooting slow application issues in Linux

The Linux kernel moves out pages which are not active or being used at the moment to swap space on the disk. This process is known as swappiness. Since swap space is on the hard drive fetching data will be slower as compared to your ram, This may cause your application to take a hit in terms of speed. You have the option to turn off swaping by changing t he value in /proc/sys/vm/swappiness to 0. The value ranges from 0 to 100 whe re 100 means aggressive swapping.

$ cat /proc/sys/vm/swappiness
60

Try 'vmstat -n 1' and check the 'si', 'so' field. If 'si' and 'so' (stands for swapin and swapout) fields are always 0, then the system is currently not swapping. Some application, has used the swap but somehow its not cleaned the swap space.

$ vmstat -n 1
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
0  0      0 759280  17080 113060    0    0   121     7   29   86  1  1 99  0  0
0  0      0 759280  17080 113092    0    0     0     0   19   29  0  0 100  0  0
0  0      0 759280  17080 113092    0    0     0     0   19   25  0  0 100  0  0
0  0      0 759280  17080 113092    0    0     0     0   20   33  0  0 100  0  0

$ sudo slabtop
Active / Total Objects (% used)    : 93687 / 94741 (98.9%)
Active / Total Slabs (% used)      : 3988 / 3988 (100.0%)
Active / Total Caches (% used)     : 65 / 103 (63.1%)
Active / Total Size (% used)       : 22980.43K / 23436.93K (98.1%)
Minimum / Average / Maximum Object : 0.01K / 0.25K / 8.00K
OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME
25557  25557 100%    0.19K   1217       21      4868K dentry
13932  13932 100%    0.11K    387       36      1548K sysfs_dir_cache
7336   7336 100%    0.57K    524       14      4192K inode_cache
5184   4924  94%    0.06K     81       64       324K kmalloc-64
4446   4446 100%    0.10K    114       39       456K buffer_head
4216   4216 100%    0.96K    527        8      4216K ext4_inode_cache
3969   3557  89%    0.19K    189       21       756K kmalloc-192
3366   3366 100%    0.04K     33      102       132K ext4_extent_status

$ top -n 1 -o RES
top - 11:31:14 up 40 min,  1 user,  load average: 0.00, 0.01, 0.05
Tasks:  75 total,   1 running,  74 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.3 us,  0.3 sy,  0.0 ni, 99.4 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:   1017816 total,   258940 used,   758876 free,    17232 buffers
KiB Swap:        0 total,        0 used,        0 free.   113360 cached Mem
    PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
    1203 root      20   0  112668  34544   1104 S  0.0  3.4   0:00.00 ruby
    1111 root      20   0  182496  34428   2644 S  0.0  3.4   0:00.14 puppet
    1219 root      20   0  265568  11808   2736 S  0.0  1.2   0:00.55 fail2ban-server
    1740 root      20   0  107696   4220   3236 S  0.0  0.4   0:00.01 sshd
    1823 vagrant   20   0   21560   4040   1764 S  0.0  0.4   0:00.05 bash
    1545 root      20   0   61364   3056   2380 S  0.0  0.3   0:00.00 sshd
    1 root      20   0   33616   2952   1492 S  0.0  0.3   0:01.01 init
    595 root      20   0   10220   2432    120 S  0.0  0.2   0:00.00 dhclient
    1822 vagrant   20   0  107696   2144   1156 S  0.0  0.2   0:00.15 sshd
    2017 ntp       20   0   33512   2096   1488 S  0.0  0.2   0:00.18 ntpd
    906 root      20   0   43448   1756   1420 S  0.0  0.2   0:00.00 systemd-logind
    418 root      20   0   49880   1652   1008 S  0.0  0.2   0:00.03 systemd-udevd
    2211 vagrant   20   0   23536   1480   1072 R  0.0  0.1   0:00.00 top
    931 syslog    20   0  255840   1376    692 S  0.0  0.1   0:00.02 rsyslogd
    744 statd     20   0   21540   1372    916 S  0.0  0.1   0:00.00 rpc.statd
    709 root      20   0   23416   1108    800 S  0.0  0.1   0:00.01 rpcbind
    1170 root      20   0  232796   1092    756 S  0.0  0.1   0:00.72 VBoxService
    1057 root      20   0   23652   1040    792 S  0.0  0.1   0:00.01 cron
    828 message+  20   0   39212   1012    676 S  0.0  0.1   0:00.12 dbus-daemon
    1005 root      20   0   14540    960    800 S  0.0  0.1   0:00.00 getty
    1010 root      20   0   14540    960    800 S  0.0  0.1   0:00.00 getty
    1009 root      20   0   14540    956    800 S  0.0  0.1   0:00.00 getty
    1012 root      20   0   14540    956    800 S  0.0  0.1   0:00.00 getty
    1264 root      20   0   14540    956    800 S  0.0  0.1   0:00.00 getty
    1002 root      20   0   14540    952    800 S  0.0  0.1   0:00.00 getty
    403 root      20   0   19472    912    612 S  0.0  0.1   0:00.09 upstart-udev-br