2

I am attempting to spin up 1000x Docker containers via:

nohup docker run --rm --name n-$1-$2 n-v2 bash -c "/root/sw/run.sh $1 $2" 2> ~/logs/log-$1-$2.log &

But after about 500x containers I start getting the following error:

fork: retry: No child processes

The server I am running this on has 48x cores and 192 GB of ram ... I should be able to spin up several thousand of these containers with no problem ... is there a way to increase this limit?

After doing some googling I found that I can check the number of max forks via ulimit -u ... is there a way to change this?

Soren A's answer seems legit, however it doesnt seem like this solved the issue. I can get to ~12287 threads (checed via cat /sys/fs/cgroup/pids/user.slice/user-1000.slice/pids.current) before I start hitting the same issue ... both my hard and soft limits in /etc/security/limits.conf are set to 99999999 yet I am still being capped at around 12287

David Foerster
  • 36,264
  • 56
  • 94
  • 147
CaffeineAddiction
  • 175
  • See here – Doug Smythies Feb 17 '17 at 23:20
  • So, increase /sys/fs/cgroup/pids/user.slice/user-1000.slice/pids.max as explained in the answer I pointed you to. I just had 118205 simultaneous processes running on my computer, although it took some doing. – Doug Smythies Feb 18 '17 at 00:41
  • @DougSmythies Thank You, your answer solved it ... 1000x nodes 15095 threads – CaffeineAddiction Feb 18 '17 at 00:53
  • I rolled back the addition of the answer to the question post. If you solved your problem based on an existing answer in a linked question please vote for that answer to increase its visibility. Don’t put the answer in your question or the comments! :-) – David Foerster Mar 06 '17 at 22:24

1 Answers1

2

You can set the values temporary with ulimit -u <value> Or permanently in /etc/security/limits.conf with lines like

*    hard    nproc  64000
*    soft    nproc  64000

Or the same in a ,conf file in /etc/security/limits.d, ex. /etc/security/limits.d/90-nproc.conf if you like.

Instead of setting it for all users (*) you could do it only for the user starting your processes, so that other users can't grab that many resources on your system.

Soren A
  • 6,799
  • I tried this with and without sudo and got errors both times -bash: ulimit: max user processes: cannot modify limit: Operation not permitted & sudo: ulimit: command not found – CaffeineAddiction Feb 17 '17 at 23:17
  • Yes, a normal user can't go above the 'hard limit' .. you will have to set nproc in the limit.conf file .. and reboot. And ulimit can'tbe set in sudo. – Soren A Feb 17 '17 at 23:24
  • still getting this issue when I go above 12287 threads (checked via cat /sys/fs/cgroup/pids/user.slice/user-1000.slice/pids.current) – CaffeineAddiction Feb 18 '17 at 00:22
  • I can't see what limit you are hitting, but for all ulimit limits the above method works. – Soren A Feb 20 '17 at 12:02