How to measure GPU usage?

Question

Using the top command in the terminal lists processes, sorted by their CPU usage (and you can change it to sort by another parameter)

Is there an equivalent for the GPU?

This fellow is asking about RAM used by GPU

Deppending, if you are using a radeon you can use radeontop, for nvidia there's another tool but I don't have the name at hand. — Braiam, Dec 07 '13 at 19:39
For nvidia it's the commandline tool nvidia-smi, except for the jetsons, where it is tegrastats. — Jus, Feb 17 '20 at 07:49
For a nvidia gpu, you can use nvidia-smi -l 5, which will provide an update every 5 seconds. (Change this number to update at a different interval.) — mikey, Apr 13 '21 at 11:14

score 160 · Accepted Answer · edited Dec 08 '23 at 12:00

160

For Nvidia GPUs there is a tool nvidia-smi that can show memory usage, GPU utilization and temperature of GPU.
For Intel GPU's you can use the intel-gpu-tools.
AMD has two options
1. fglrx (closed source drivers):
```
 aticonfig --odgc --odgt
```
2. And for mesa (open source drivers), you can use RadeonTop
```
sudo apt install radeontop
radeontop
```

^{Source:GPU usage monitoring}

edited Dec 08 '23 at 12:00

Ciro Santilli OurBigBook.com

28,474

answered Dec 07 '13 at 19:39

Mitch

107,631

87

Use watch nvidia-smi for real-time updates. – Lenar Hoyt Nov 20 '14 at 22:00
1

aticonfig won't work over SSH. Claims it needs an X server running to work (there is one running). However, RadeonTop (sudo apt-get radeontop) does work with the fglrx (needs root). Hurrah! Sadly RadeonTop doesn't provide any temperature readings. – Ken Sharp Jan 05 '15 at 09:58
aticonfig WILL work over SSH, but an X server with tcp enabled needs to be running. This can be done by configuring lightdm via xserver-allow-tcp=true. Searching around this site with these keywords should lead to the result. – jyalim Feb 18 '16 at 00:20
2

Not sure why but watch -n 1 nvidia-smi gave me real-time updates. watch nvidia-smi has a 2 sec update delay. – markroxor Aug 06 '18 at 07:58
10

sudo intel_gpu_top should give you real time updates for intel gpus. – George D Aug 07 '18 at 12:07
I like the radeontop. – Melroy van den Berg Dec 21 '19 at 01:12
You can use nvidia-smi -l 1 for real-time updates. Source: GPU usage monitoring (CUDA) – ekl1pse Jun 06 '21 at 14:35

score 51 · Answer 2 · answered Oct 01 '16 at 19:45

51

You can use gpustat, which is a simple command-line script (wrapper for nvidia-smi) for querying and monitoring GPU status:

answered Oct 01 '16 at 19:45

Franck Dernoncourt

3,460

11

NOTE: Only for Nvidia – Mr. Developerdude Apr 22 '19 at 22:32
7

pip install gpustat – Shital Shah Sep 17 '19 at 08:09
1

Works perfect with proprietary driver, thank you. – Smeterlink Apr 26 '21 at 12:02

score 48 · Answer 3 · edited Sep 02 '18 at 16:35

48

For Intel:

Install intel-gpu-tools (its likely that they are installed already)
```
sudo apt-get install intel-gpu-tools 
```
Start the top like utility with
```
sudo intel_gpu_top
```
Check your stats and then exit with Ctrl+C

Thats what you get:

Thanks @Mitch! :)

edited Sep 02 '18 at 16:35

abu_bua

10,783

answered Apr 22 '17 at 09:10

Thomas Venturini

739

1

Nope, it doesn't seem to be installed by default. I'm using Skylake GT2 (Intel HD Graphics 520). – wyphan Feb 24 '21 at 20:54

score 37 · Answer 4 · edited Apr 13 '17 at 12:37

Nvidia: to continuously update the output of nvidia-smi, you can use nvidia-smi --loop=1 (refresh interval of 1 second) or nvidia-smi --loop-ms=1000 (refresh interval of 1000 milliseconds).

   -l SEC, --loop=SEC
       Continuously  report  query data at the specified interval, rather than
       the default of  just  once.   The  application  will  sleep  in-between
       queries.   Note  that on Linux ECC error or XID error events will print
       out during the sleep period if the -x flag was not specified.  Pressing
       Ctrl+C at any time will abort the loop, which will otherwise run indef‐
       initely.  If no argument is specified for the -l form a default  inter‐
       val of 5 seconds is used.

   -lms ms, --loop-ms=ms
       Same as -l,--loop but in milliseconds.

FYI:

score 21 · Answer 5 · edited Apr 21 '23 at 10:26

21

You can use the monitoring program glances with its GPU monitoring plug-in:

open source
to install: sudo apt-get install -y python-pip; sudo pip install glances
to launch: sudo glances

It also monitors the CPU, disk IO, disk space, network, and a few other things:

edited Apr 21 '23 at 10:26

Lampe2020

51

answered Feb 07 '17 at 00:06

Franck Dernoncourt

3,460

You also need to do pip install nvidia-ml-py3 – Paul Williams Aug 18 '18 at 04:40
1

It also works without sudo and when istalled for user only: pip install --user glances[gpu] and then run glances. – Ivan Kovtun May 10 '19 at 22:10

WinEunuuchs2Unix · Answer 6 · 2019-07-08T10:35:06.627

Conky

I like to use conky as a real-time monitor for both CPU and GPU. Installation is straightforward:

sudo apt install conky

Intel i7-6700HQ iGPU HD 530

In this instance I've booted using the integrated GPU rather than the nVidia GTX 970M:

The conky code adapts depending on if booted with prime-select intel or prime-select nvidia:

nVidia GPU GTX 970M

In this instance I've booted using the nVidia GTX 970M rather than the integrated GPU:

Conky code

The conky code was recently modified to auto-sense the GPU. Now it doesn't have to be hand modified when rebooting to a different GPU:

#------------+
# Intel iGPU |
#------------+
${color orange}${hr 1}${if_existing /sys/class/drm/card0/gt_cur_freq_mhz}
${color2}${voffset 5}Intel® Skylake GT2 HD 530 iGPU @${alignr}${color green}
${execpi .001 (cat /sys/class/drm/card0/gt_cur_freq_mhz)} MHz
${color}${goto 13}Min. Freq:${goto 120}${color green}${execpi .001 (cat /sys/class/drm/card0/gt_min_freq_mhz)} MHz${color}${goto 210}Max. Freq:${alignr}${color green}${execpi .001 (cat /sys/class/drm/card0/gt_max_freq_mhz)} MHz
${color orange}${hr 1}${else}
#------------+
# Nvidia GPU |
#------------+
#${color orange}${hr 1}${if_match "${lsmod | grep nvidia_uvm}">""}
${color2}${voffset 5}${execpi .001 (nvidia-smi --query-gpu=gpu_name --format=csv,noheader)} ${color1}@ ${color green}${execpi .001 (nvidia-smi --query-gpu=clocks.sm --format=csv,noheader)} ${alignr}${color1}Temp: ${color green}${execpi .001 (nvidia-smi --query-gpu=temperature.gpu --format=csv,noheader)}°C
${color1}${voffset 5}Ver: ${color green}${execpi .001 (nvidia-smi --query-gpu=driver_version --format=csv,noheader)} ${color1} P-State: ${color green}${execpi .001 (nvidia-smi --query-gpu=pstate --format=csv,noheader)} ${alignr}${color1}BIOS: ${color green}${execpi .001 (nvidia-smi --query-gpu=vbios_version --format=csv,noheader)}
${color1}${voffset 5}GPU:${color green}${execpi .001 (nvidia-smi --query-gpu=utilization.gpu --format=csv,noheader)} ${color1}Ram:${color green}${execpi .001 (nvidia-smi --query-gpu=utilization.memory --format=csv,noheader)} ${color1}Pwr:${color green}${execpi .001 (nvidia-smi --query-gpu=power.draw --format=csv,noheader)} ${alignr}${color1}Freq: ${color green}${execpi .001 (nvidia-smi --query-gpu=clocks.mem --format=csv,noheader)}
${color orange}${hr 1}${endif}

Different versions of the full code listing can be found in these answers:

Could you provide the steps to install conky and setup the theme to be like your Nvidia example please? — Tak, Jul 08 '19 at 08:14
@Tak I've updated answer with installation instructions and existing links to code. — WinEunuuchs2Unix, Jul 08 '19 at 10:36

Tshilidzi Mudau · Answer 7 · 2016-11-10T21:18:41.440

20

I use the following command:

nvidia-smi -l 2

and it gives me updates every 2 seconds.

Or :

watch -n0.1 "nvidia-settings -q GPUUtilization -q useddedicatedgpumemory"

And on AMD, use:

aticonfig --odgc --odgt

edited Nov 10 '16 at 21:18

answered Nov 10 '16 at 21:11

Tshilidzi Mudau

4,173

useddedicatedgpumemory is that the tensorcores utilization? – Fadwa Oct 28 '20 at 12:29
If you installed your nvidia driver, it's probably already installed. Take a look with whereis nvidia-smi to find its location. – bvdb Oct 30 '20 at 16:41

score 11 · Answer 8 · answered May 28 '20 at 19:42

11

I just found this command:

nvidia-smi --query-gpu=utilization.gpu --format=csv --loop=1

Here is a demo:

answered May 28 '20 at 19:42

singrium

6,880

score 10 · Answer 9 · answered Mar 03 '23 at 15:55

No one mentioned the nvtop ? A nice tool similar to the htop we used to. Installation is easy, simply do:

$ sudo apt install nvtop
or
$ snap install nvtop

The version from snap is usually a lot newer. The attractive feature is that it shows which process uses GPU by how much. A typical screenshot looks like the following:

score 5 · Answer 10 · answered Feb 24 '21 at 20:43

In my case nvidia-smi did not show the GPU load %, only the memory (guess my GTX 650 is too old).

What did work for me was the NVIDIA X Server Settings GUI app (shipped with the driver I believe). Navigate to the section named GPU 0 - (Your Model) - it shows the detailed status info of your GPU usage, updating every 2 seconds:

score 1 · Answer 11 · answered Dec 17 '23 at 20:00

1

For AMD/ATi/Radeon cards on Linux Systems, CoreCtrl seems to do the job well.

I am using Ubuntu 20.04 LTS.

https://gitlab.com/corectrl/corectrl

answered Dec 17 '23 at 20:00

utdream

229
2
3

score 0 · Answer 12 · answered Nov 11 '23 at 07:04

0

I'm surprised that no one mentioned this command:

watch -n 1 nvidia-smi

answered Nov 11 '23 at 07:04

DevLoverUmar

117

How to measure GPU usage?

12 Answers12

Conky

Intel i7-6700HQ iGPU HD 530

nVidia GPU GTX 970M

Conky code

Linked

Related