1

I have a list of cron jobs that i can see when I type htop in at the cli. These jobs have been running for about 3 hours at 100% cpu and i have no clue what they are or what they are doing. They are all run by root.

Is there a way to find out what a specific PID is doing

HTOP output

MCP_infiltrator
  • 441

1 Answers1

3

Assuming you are using a recent version of Ubuntu, you can use systemctl to get a broad overview or list of error messages. For example:

$ systemctl status cron.service
● cron.service - Regular background program processing daemon
   Loaded: loaded (/lib/systemd/system/cron.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2019-07-09 07:50:56 CDT; 6 days ago
     Docs: man:cron(8)
 Main PID: 884 (cron)
    Tasks: 1 (limit: 4915)
   CGroup: /system.slice/cron.service
           └─884 /usr/sbin/cron -f

Jul 15 09:25:01 opal CRON[20797]: pam_unix(cron:session): session opened for user root by (uid=0)
Jul 15 09:25:01 opal CRON[20798]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Jul 15 09:25:01 opal CRON[20797]: pam_unix(cron:session): session closed for user root
Jul 15 09:35:01 opal CRON[21319]: pam_unix(cron:session): session opened for user root by (uid=0)
Jul 15 09:35:01 opal CRON[21319]: pam_unix(cron:session): session closed for user root
Jul 15 09:39:01 opal CRON[21517]: pam_unix(cron:session): session opened for user root by (uid=0)
Jul 15 09:39:01 opal CRON[21518]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Jul 15 09:39:01 opal CRON[21517]: pam_unix(cron:session): session closed for user root
Jul 15 09:45:01 opal CRON[21982]: pam_unix(cron:session): session opened for user root by (uid=0)
Jul 15 09:45:01 opal CRON[21982]: pam_unix(cron:session): session closed for user root

Next, you could run strace on individual processes. For example:

$ pgrep -x cron
884
$ sudo strace -p 884
strace: Process 884 attached
restart_syscall(<... resuming interrupted nanosleep ...>^Cstrace: Process 884 detached
 <detached ...>

Here I've used Ctrl-C to detach once I've seen enough. You can also pipe the output to less.

$ sudo strace -p 884 2>&1 | less -c

In my case, the cron daemon is idle, so it is running the nanosleep syscall. To see just a summary of what syscalls are using CPU, use the -c flag:

$ sudo strace -c -p 884
strace: Process 884 attached
^Cstrace: Process 884 detached
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 61.11    0.000253         253         1           clone
 16.18    0.000067          34         2         1 wait4
  9.66    0.000040           3        12           stat
  6.28    0.000026          26         1           restart_syscall
  5.80    0.000024          24         1         1 nanosleep
  0.97    0.000004           4         1         1 rt_sigreturn
------ ----------- ----------- --------- --------- ----------------
100.00    0.000414                    18         3 total

You could also use gdb to get a backtrace.

$ sudo apt install cron-dbgsym
$ sudo gdb -p 884
(gdb) backtrace
#0  0x00007f84cdf399a4 in __GI___nanosleep (requested_time=0x7ffdcdd18d10, remaining=0x7ffdcdd18d10)
    at ../sysdeps/unix/sysv/linux/nanosleep.c:28
#1  0x00007f84cdf398aa in __sleep (seconds=0) at ../sysdeps/posix/sleep.c:55
#2  0x000055af6a6206ab in cron_sleep (target=<optimized out>) at cron.c:415
#3  main (argc=<optimized out>, argv=<optimized out>) at cron.c:173

This again tells me it is sleeping while idle, but yours may be more informative.

I would also recommend you alter cron.service to limit the CPU and I/O priority:

$ sudo systemctl edit cron.service
# edits go to /etc/systemd/system/cron.service.d/override.conf

I use something like this:

[Service]
Nice=19
CPUSchedulingPolicy=idle
IOSchedulingClass=idle
Nathaniel M. Beaver
  • 1,518
  • 12
  • 33