Several cron jobs keep failing to execute because the kernel cannot fork. It's been happening for weeks (I think since I upgraded to Ubuntu 16 but possibly earlier).
Error messages
$ grep fork myscript.log
/usr/bin/groovy: 24: /usr/bin/groovy: Cannot fork
$ tail /var/log/syslog
Oct 19 14:25:01 nuc CRON[19939]: (CRON) error (can't fork)
Oct 19 14:25:01 nuc cron[719]: /usr/sbin/sendmail: Resource temporarily unavailable
Note: I don't care about sendmail specifically.
Environment info
ulimit -a
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) unlimited
stack(kbytes) 8192
coredump(blocks) 0
memory(kbytes) unlimited
locked memory(kbytes) 64
process 999999
nofiles 999999
vmemory(kbytes) unlimited
locks unlimited
rtprio 0
cat /proc/sys/fs/file-nr
7936 0 793377
cat /proc/sys/kernel/pid_max
32768
htop
1 [||| 1.9%] Tasks: 196, 114 kthr; 1 running
2 [||||| 4.2%] Load average: 0.43 0.20 0.16
3 [| 0.7%] Uptime: 2 days, 18:22:08
4 [ 0.0%]
Mem[|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| 5.68G/7.62G]
Swp[||||||| 218M/3.82G]
uname -a
Linux nuc 4.4.0-21-generic #37-Ubuntu SMP Mon Apr 18 18:33:37 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
Other relevant information
- It mostly seems to happen in cron. When I run it interactively (usually in
zsh) the job runs fine. - I'm running dozens of groovy microservices (and each thread opens its own file handle to each jar file).
- I started using Docker out of desperation, but it's just making it more difficult to see which ports are currently being listened on.
