91

When running tail -f filename, I got the following message:

tail: inotify cannot be used, reverting to polling: Too many open files

Is that a potential problem?

How do I diagnose what's responsible for all the open files? I have a list of suspect processes, but if they don't turn out to be the culprits, instructions that don't rely on knowing which process to check would be useful.

Andrew Grimm
  • 1,314

3 Answers3

87

You can use lsof to understand who's opening so many files. Usually it's a (web)server that opens so many files, but lsof will surely help you identify the cause.

Once you understand who's the bad guy you can

  • kill the process/stop the program
  • raise the ulimit

If output from lsof is quite huge try redirecting it to a file and then open the file

Example (you might have to Ctrl+C the first command)

lsof > ~/Desktop/lsof.log
cat ~/Desktop/lsof.log | awk '{ print $2 " " $1; }' | sort -rn | uniq -c | sort -rn | head -20
vim ~/Desktop/lsof.log
gatorback
  • 5,785
  • 9
  • 40
  • 65
Andrea Olivato
  • 1,313
  • 49
    For the lazy: lsof | awk '{ print $2; }' | uniq -c | sort -rn | head – itsadok Nov 27 '12 at 06:10
  • 1
    I got the same error and using ulimit doesn't work. The tail -F command still returns the error. I increased the limit from 1024 to 3000 so I would imagine that I have enough room by then... I guess I'd have to reboot! – Alexis Wilke Jan 26 '13 at 11:41
  • 18
    I found itsadok's line helpful, but I think you should sort first (because uniq only works with adjacent lines), run uniq, then sort again. So lsof | awk '{ print $2; }' | sort -rn | uniq -c | sort -rn | head. – Tyler Collier Feb 02 '13 at 00:45
  • 27
    Sorting and counting up the most files open is definitely the best thing to do. Show process name as well as pid: '''lsof | awk '{ print $2 " " $1; }' | sort -rn | uniq -c | sort -rn | head -20''' – gaoithe Dec 01 '14 at 11:34
  • 2
    @gaoithe if you put that into an answer I will gladly upvote it :) – Matt Ball Jul 27 '15 at 17:56
  • 1
    Use lsof +c 0 to show full process names on macOS. – vaughan Jul 12 '18 at 11:51
  • Newcomer here, what does the first and 2nd column of numbers represent? One is probably the process id but which one? The first column seems to be what it's ordered by, but no way there are 14000 opened files from VSCode... – Douglas Gaskell Oct 26 '18 at 21:49
  • ps $(lsof | awk '{print $2}' | uniq -c | sort -rn | head -n5 | awk '{print $2}') – kenorb Feb 02 '19 at 23:59
  • 1
    Nice command, but what does the first and second column represent? Edit: Oh, my past self already asked this... – Douglas Gaskell Apr 08 '19 at 23:10
  • 1
    first column is number of files, second is PID – QkiZ Apr 22 '20 at 12:01
35

In case anyone else needs it...

ulimit -a

Will display all current limits. Specifically ulimit -n 70000 will set the file descriptor limit.

Also...

cat /proc/sys/fs/file-max

Will display/set the kernel limit if edited.

sudo echo 200000 > /proc/sys/fs/file-max

A much more detailed explanation can be found at...

How do I increase the open files limit for a non-root user?

Community
  • 1
kervin
  • 1,451
  • 4
    ulimits are there so you can keep tight control over resource use in your system. It is best to increase them individually for processes you know will need to use many file descriptors. By keeping them tight in testing you will find processes which might steadily leak file descriptors over time. Note also that file handles are used for any device access in unix/linux. e.g. every network socket open by a process uses a file handle. That explains why you can hit the "too many open files" in case of regular file-system files as well as any device files such as network connections. – gaoithe Dec 01 '14 at 11:51
  • "Too many files" (EMFILE) has nothing to do with /proc/sys/fs/file-max. The english translation for that error "Too many files for systerm" (ENFILE), indicating the system-wide limit. "Too many files" always refers to the per-process file descriptor limit. – Remember Monica Sep 10 '21 at 02:00
7

While ulimit can be used to determine how many files are allowed to be open per process you may want to find the culprit.

@itsadok @Tyler Collier @gaoithe in comments to other answers highlight that sorting and counting which process has the most files open is the best thing to do here:

sudo lsof | head -1 | awk '{ print "COUNT " $2 " " $1; }' && sudo lsof +c 0 | awk '{ print $2 " " $1; }' | sort -rn | uniq -c | sort -rn | head -20

Above command:

  • Gives output a header
  • lsof gives list of open files
  • +c 0 option instructs lsof to print full command
  • awk '{ print $2 " " $1; }' prints PID and COMMAND column in results
  • sort -rn sorts results so identical entries are next to each other (needed for uniq -c to work properly)
  • uniq -c counts files open by PID/command
  • sort -rn sorts results by count
  • head -20 shows top 20 files open by PID/command

Note: This will count includes "files" that don't count towards limits

You may want to investigate even further by looking at limits for a PID, # of files open for a specific PID, and limit lsof to only count files that count towards limit - see https://serverfault.com/a/964752/152562.

bbodenmiller
  • 209