[Linux] used value is different from the result of free command

[kyet]

Summary

• OS: Linux
• Architecture: 64bit
• Psutil version: 7.0.1 (installed from the latest source code)
• Python version: Python 3.12.3
• Type: core

Description

The comment for the virtual_memory function states that it returns values identical to the free command. However, there is a difference in the calculation of the used value.

def virtual_memory():
    """ .. snip ..
    The returned values are supposed to match both "free" and "vmstat -s"
    CLI tools.
    """

The psutil library calculates the used value as follows:

used = total - free - cached - buffers

On the other hand, the free command calculates the used value as:

mem_used = mHr(MemTotal) - mHr(MemAvailable);

• https://gitlab.com/procps-ng/procps/-/blame/master/library/meminfo.c#L736

According to the kernel, available != (cached + buffers).

• https://github.com/torvalds/linux/blob/0ff41df1cb268fc69e703a08a57ee14ae967d0ca/mm/show_mem.c#L32

Therefore, it seems that the used value should be corrected to match the calculation of the free command.

DRAFT patch

diff --git a/psutil/_pslinux.py b/psutil/_pslinux.py
index ac0d1d6f..dd227acc 100644
--- a/psutil/_pslinux.py
+++ b/psutil/_pslinux.py
@@ -420,12 +420,6 @@ def virtual_memory():
     except KeyError:
         slab = 0

-    used = total - free - cached - buffers
-    if used < 0:
-        # May be symptomatic of running within a LCX container where such
-        # values will be dramatically distorted over those of the host.
-        used = total - free
-
     # - starting from 4.4.0 we match free's "available" column.
     #   Before 4.4.0 we calculated it as (free + buffers + cached)
     #   which matched htop.
@@ -456,6 +450,8 @@ def virtual_memory():
         #     24fd2605c51fccc375ab0287cec33aa767f06718/proc/sysinfo.c#L764
         avail = free

+    used = total - avail
+
    percent = usage_percent((total - avail), total, round_=1)

    # Warn about missing metrics which are set to 0.

Problem scenario

For instance, when using the PyTorch library to allocate a tensor buffer on the CPU and pinning it, shared memory is used. The free command shows an increase in used, shared, and cached (not page cache), and a decrease in available. But, psutil does not report an increase in used.

Before patch

• percent increases, but used does not increase

(psutil_origin) (Supermicro-EMR) ~/ws/psutil_origin $ uv pip install torch numpy
.. snip ..

(psutil_origin) (Supermicro-EMR) ~/ws/psutil_origin $ uv pip install psutil .
.. snip ..
 + psutil==7.0.1 (from file:///home/kwangjin/ws/psutil_origin)

(psutil_origin) (Supermicro-EMR) ~/ws/psutil_origin $ cd ~
(psutil_origin) (Supermicro-EMR) ~ $ python
Python 3.12.3 (main, Feb  4 2025, 14:48:35) [GCC 13.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch, psutil
>>> psutil.virtual_memory()
svmem(total=1081845010432, available=913956642816, percent=15.5, used=24018743296, free=814759215104, active=175978917888, inactive=79222304768, buffers=1501523968, cached=241565528064, shared=137461260288, slab=5935333376)
>>> buffer = torch.empty(16 * 1024**3, dtype=torch.uint8, pin_memory=True)
>>> psutil.virtual_memory()
svmem(total=1081845010432, available=896386371584, percent=17.1, used=24400781312, free=797149900800, active=193165127680, inactive=79222366208, buffers=1501560832, cached=258792767488, shared=154649518080, slab=6045192192)

After patch

• both percent and used increase

.. snip installation of torch, numpy, psutil ..
(psutil) (Supermicro-EMR) ~ $ python
.. snip ..
>>> psutil.virtual_memory()
svmem(total=1081845010432, available=915627560960, percent=15.4, used=166217449472, free=816512339968, active=174438731776, inactive=79149432832, buffers=1494491136, cached=241490489344, shared=137461243904, slab=5922525184)
>>> buffer = torch.empty(16 * 1024**3, dtype=torch.uint8, pin_memory=True)
>>> psutil.virtual_memory()
svmem(total=1081845010432, available=897884762112, percent=17.0, used=183960248320, free=798730485760, active=191647444992, inactive=79149588480, buffers=1494732800, cached=258717679616, shared=154649501696, slab=6034608128)