4

My ubuntu server (20.04 headless) freezes almost twice a week.

The first time, the last message in journalctl was

systemd: Finished Message of the Day.

and the second time, one from jellyfin. The weird thing is (I think), the computer isn't really crashing. It freezes and the fans keep spinning. To get it working again, I always have to turn it off by holding down the power button and then turn it back on. Maybe that's obvious, but I have also seen a lot of post where the user was still able to ssh into the machine. This isn't the case. Even though, the light on my LAN switch is blinking (on) the router/modem says it's offline.

I'm using:

  • CPU: Intel Core i5-6500 (4)
  • Ram: 16GB with Swap: 4GB

Does anyone know how to troubleshoot this accordingly?

Output of: $ journaclctl -b -1 -e

Sep 06 19:53:58 vind-media-server jellyfin[5074]: [19:53:58] [INF] Executing Scan Media Library
Sep 06 19:53:58 vind-media-server jellyfin[5074]: [19:53:58] [INF] Stopping directory watching for path /mnt/hdd/
Sep 06 19:53:58 vind-media-server jellyfin[5074]: [19:53:58] [INF] Stopping directory watching for path /mnt/hdd/
Sep 06 19:53:58 vind-media-server jellyfin[5074]: [19:53:58] [INF] Stopping directory watching for path /mnt/hdd/
Sep 06 19:53:58 vind-media-server jellyfin[5074]: [19:53:58] [INF] Stopping directory watching for path /mnt/hdd/
Sep 06 19:53:58 vind-media-server jellyfin[5074]: [19:53:58] [INF] Validating media library
Sep 06 19:54:07 vind-media-server jellyfin[335115]: [matroska,webm @ 0x55b5c247b600] Could not find codec parameters for stream 3 (Subtitle: hdmv_pgs_subtitle (pgssub)): unspecified size
Sep 06 19:54:07 vind-media-server jellyfin[335115]: Consider increasing the value for the 'analyzeduration' and 'probesize' options
Sep 06 19:54:07 vind-media-server jellyfin[335115]: [matroska,webm @ 0x55b5c247b600] Could not find codec parameters for stream 4 (Subtitle: hdmv_pgs_subtitle (pgssub)): unspecified size
Sep 06 19:54:07 vind-media-server jellyfin[335115]: Consider increasing the value for the 'analyzeduration' and 'probesize' options
Sep 06 19:54:07 vind-media-server jellyfin[335115]: [matroska,webm @ 0x55b5c247b600] Could not find codec parameters for stream 5 (Subtitle: hdmv_pgs_subtitle (pgssub)): unspecified size
Sep 06 19:54:07 vind-media-server jellyfin[335115]: Consider increasing the value for the 'analyzeduration' and 'probesize' options
Sep 06 19:54:07 vind-media-server jellyfin[335115]: [matroska,webm @ 0x55b5c247b600] Could not find codec parameters for stream 6 (Subtitle: hdmv_pgs_subtitle (pgssub)): unspecified size
Sep 06 19:54:07 vind-media-server jellyfin[335115]: Consider increasing the value for the 'analyzeduration' and 'probesize' options
Sep 06 19:54:18 vind-media-server jellyfin[5074]: [19:54:18] [INF] Scan Media Library Completed after 0 minute(s) and 19 seconds
Sep 06 19:54:18 vind-media-server jellyfin[5074]: [19:54:18] [INF] ExecuteQueuedTasks
Sep 06 19:54:18 vind-media-server jellyfin[5074]: [19:54:18] [INF] Watching directory /mnt/hdd/
Sep 06 19:54:18 vind-media-server jellyfin[5074]: [19:54:18] [INF] Watching directory /mnt/hdd/
Sep 06 19:54:19 vind-media-server jellyfin[5074]: [19:54:19] [INF] Watching directory /mnt/hdd/
Sep 06 19:54:19 vind-media-server jellyfin[5074]: [19:54:19] [INF] Watching directory /mnt/hdd/
Sep 06 19:54:23 vind-media-server jellyfin[5074]: [19:54:23] [WRN] Slow HTTP Response from DONT/Items/6d43d5be1bcbfcf47c0a00a573543faf/Images/Primary?fillHeight=670&fillWidth=446&quality=96&tag=97f0105f1268e219>
Sep 06 19:54:23 vind-media-server jellyfin[5074]: [19:54:23] [WRN] Slow HTTP Response from DONT/Items/fc45e94a2764bf3d5bdb53dfdfdc701c/Images/Primary?fillHeight=670&fillWidth=446&quality=96&tag=b465bdd42db9a771>
Sep 06 19:54:24 vind-media-server jellyfin[5074]: [19:54:24] [WRN] Slow HTTP Response from DONT/Items/90a6e3463d797f491b6b2cca511d4922/Images/Primary?fillHeight=1440&fillWidth=960&quality=96&tag=d09daa02958181b>
Sep 06 19:54:24 vind-media-server jellyfin[5074]: [19:54:24] [WRN] Slow HTTP Response from DONT/Items/90a6e3463d797f491b6b2cca511d4922/Images/Backdrop/0?maxWidth=3840&tag=1182e96a248b54b0ed77dd3e18a425ba&qualit>
Sep 06 19:54:24 vind-media-server jellyfin[5074]: [19:54:24] [WRN] Slow HTTP Response from DONT/Items/685c6bb9649da6cba69b39772a767696/Images/Primary?fillHeight=670&fillWidth=446&quality=96&tag=58a3ee89ce80b31e>
Sep 06 19:54:50 vind-media-server jellyfin[5074]: [19:54:50] [INF] WS 193.29.106.188 closed
Sep 06 19:55:14 vind-media-server sshd[334989]: Received disconnect from IP port PORT:11: disconnected by user
Sep 06 19:55:14 vind-media-server sshd[334989]: Disconnected from user death IP port PORT
Sep 06 19:55:14 vind-media-server systemd[1]: session-633.scope: Succeeded.
Sep 06 19:55:14 vind-media-server sshd[334843]: pam_unix(sshd:session): session closed for user death
Sep 06 19:55:14 vind-media-server systemd-logind[930]: Session 633 logged out. Waiting for processes to exit.
Sep 06 19:55:14 vind-media-server systemd-logind[930]: Removed session 633.
Sep 06 20:09:01 vind-media-server CRON[335497]: pam_unix(cron:session): session opened for user root by (uid=0)
Sep 06 20:09:01 vind-media-server CRON[335498]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Sep 06 20:09:01 vind-media-server CRON[335497]: pam_unix(cron:session): session closed for user root
Sep 06 20:09:09 vind-media-server systemd[1]: Starting Clean php session files...
Sep 06 20:09:10 vind-media-server systemd[1]: phpsessionclean.service: Succeeded.
Sep 06 20:09:10 vind-media-server systemd[1]: Finished Clean php session files.
Sep 06 20:17:01 vind-media-server CRON[335698]: pam_unix(cron:session): session opened for user root by (uid=0)
Sep 06 20:17:01 vind-media-server CRON[335699]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Sep 06 20:17:01 vind-media-server CRON[335698]: pam_unix(cron:session): session closed for user root
Sep 06 20:39:01 vind-media-server CRON[336116]: pam_unix(cron:session): session opened for user root by (uid=0)
Sep 06 20:39:01 vind-media-server CRON[336117]: (root) CMD (  [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Sep 06 20:39:01 vind-media-server CRON[336116]: pam_unix(cron:session): session closed for user root
Sep 06 20:39:09 vind-media-server systemd[1]: Starting Clean php session files...
Sep 06 20:39:10 vind-media-server systemd[1]: phpsessionclean.service: Succeeded.
Sep 06 20:39:10 vind-media-server systemd[1]: Finished Clean php session files.
Sep 06 20:55:05 vind-media-server systemd[1]: Starting Message of the Day...
Sep 06 20:55:10 vind-media-server 50-motd-news[336477]:  * Super-optimized for small spaces - read how we shrank the memory
Sep 06 20:55:10 vind-media-server 50-motd-news[336477]:    footprint of MicroK8s to make it the smallest full K8s around.
Sep 06 20:55:10 vind-media-server 50-motd-news[336477]:    https://ubuntu.com/blog/microk8s-memory-optimisation
Sep 06 20:55:10 vind-media-server systemd[1]: motd-news.service: Succeeded.
Sep 06 20:55:10 vind-media-server systemd[1]: Finished Message of the Day.

Pastebin.com

Should be the dmesg log file of the crash:

Pastebin.com

I was able to evade any freezes over the past days by just rebooting once a day. This proves it's not some kind of thermal issue.

Thank you @thomas-ward for the tip. I have applied the recommended changes from this command: https://askubuntu.com/a/45009/1028839. Unfortunately, I couldn't find the old log from the freeze-up anymore. I will reboot once more to make the changes take effect and then wait for the next freeze-up.

Uptime: 6 days, 3 hours, 53 mins

Looks like that was the fix. Thank you very much @thomas-ward. If you'd like to submit the answer, I can tag it as the solution.

Update

After a freeze during reboot, I have decided to blame these problems on hardware. I replaced the harddrive with an ssd, switched to debian (unrelated) and updated the bios. Will add coming information about this in the future. Could take some time though.

Uptime: 9 days, 13 hours, 37 mins

I think it's pretty save to say the problem is fixed now.

Notes for users with similar issues: try to 'disable' 5% of your RAM as explained in EDIT 4. If that doesn't work, try switching SATA ports on your mainboard or get a new SATA cable. In my case, logs were impossible to show up due to the HDD straight up failing/disconnecting while the computer is running.

Artur Meinild
  • 26,018
Vernox Vernax
  • 43
  • 4
  • Before the server crashes check dmesg logs. A general maintainance would be good. It may also likely be a hardware problem see if your processor is overheating and needs repasting. Do a memtest to check if your memory is still ok. – kleo Sep 07 '21 at 23:45
  • After a "sudden shutdown", aka "system crash", and reboot, the terminal command sudo journalctl -b -1 -e will show you the end of the previous boot's log's. If there is no hint there, suspect power/ overheating.

    Read man journalctl to see how to extract information from the logs.

    You can find how I make use of journalctl easier at https://askubuntu.com/users/25618/waltinator.

     – waltinator Sep 08 '21 at 02:52
  • @admin Hm, thank you for the suggestions, but I dont think it's an overheating issue. I have been using cpu encoding for three days straight, but the crash was almost 10hours after that. Also, how would I go about logging dmesg before the crash? I found a log file of dmesg but I can't see any problems. (EDIT 2) – Vernox Vernax Sep 08 '21 at 15:42
  • Check syslog for any output. It might be RAM exhaustion which means you're doing things that your system just doesn't have the resources for, which can lead to these types of freezes. – Thomas Ward Sep 14 '21 at 17:10
  • 1
    I’m voting to close this question because it appears to be a hardware issue. – BeastOfCaerbannog Oct 24 '23 at 08:47

0 Answers0