In case if you don't want to install a bunch of sub-dependencies for apport-retrace tool, you can unpack apport format into separate files and to use only CoreDump dump with gdb as usual.
apport-unpack systemGeneratedCrashReportPath.crash yourNewUnpackDirectoryHere
cd yourNewUnpackDirectoryHere/
gdb `cat ExecutablePath` CoreDump (pay attention to backticks here!)
bt (output actual back-trace)
Note: apport-unpack will sometimes crash itself on unpack operation (apport seems broken all around... xD), but your CoreDump and other files will be there, just ignore it and delete all .crash files in /var/crash after you move them elsewhere in order to allow system to output new crash reports from same apps there.
There is a tool called apport-retrace that reads the .crash files and allows you to either fill it with a fully-symbolic stack trace or run a gdb session using the core dump. To start a gdb session, run apport-retrace -g CRASHFILE.crash. Note that you need to have the -dbg packages installed to get a good stack trace.
That being said (I'm not an expert on PHP), it might actually be something that you wrote in one of your files that is causing the crash.
t seem to work:# apport-retrace -g _usr_sbin_php5-fpm.1002.crash
ERROR: report file does not contain one of the required fields: CoreDump DistroRelease Package ExecutablePath--rebuild-package-info should also work as long as you are on the same system the crash happened, I think? I get errors because of missing deb-src URIs and can't be bothered to get it to work, I'll try the apport-unpack version.
– mxmlnkn
Dec 05 '20 at 23:29
How to also obtain source code on GDB
Tested on Ubuntu 23.10, I managed to get GDB working with stamster's answer: https://askubuntu.com/a/947532/52975 however there was no source code visible for the packages. Since I had a recent crash I wanted to debug, I'll take this chance to document how to do it.
First set things up so that you are able to do a package debugging with source hello world and properly debug gdb ls with source as explained at: How to install debug symbols for installed packages?
Next, after doing:
gdb `cat ExecutablePath` CoreDump
this is what I saw:
>>> bt
#0 __pthread_kill_implementation (no_tid=0, signo=5, threadid=<optimized out>) at ./nptl/pthread_kill.c:44
#1 __pthread_kill_internal (signo=5, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
#2 __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=5) at ./nptl/pthread_kill.c:89
#3 0x00007f62b5842866 in __GI_raise (sig=5) at ../sysdeps/posix/raise.c:26
#4 0x0000562470b03bda in ??? ()
#5 0x00007f62b5842910 in <signal handler called> () at /lib/x86_64-linux-gnu/libc.so.6
#6 0x00007f62b62e5867 in g_log_structured_array () at /lib/x86_64-linux-gnu/libglib-2.0.so.0
#7 0x00007f62b62e5b52 in g_log_default_handler () at /lib/x86_64-linux-gnu/libglib-2.0.so.0
#8 0x00007f62b62e69e2 in g_logv () at /lib/x86_64-linux-gnu/libglib-2.0.so.0
#9 0x00007f62b62e6ca3 in g_log () at /lib/x86_64-linux-gnu/libglib-2.0.so.0
#10 0x00007f62b68ab59e in ??? () at /usr/lib/x86_64-linux-gnu/mutter-13/libmutter-mtk-13.so.0
#11 0x00007f62b5704a5b in _XError () at /lib/x86_64-linux-gnu/libX11.so.6
#12 0x00007f62b1f63adb in ??? () at /lib/x86_64-linux-gnu/libGLX.so.0
#13 0x00007f62b5f0962f in ??? () at /usr/lib/x86_64-linux-gnu/mutter-13/libmutter-cogl-13.so.0
but no frame had source.
From the function names, the first possibly interesting frame was #10 0x00007f62b68ab59e in ??? () at /usr/lib/x86_64-linux-gnu/mutter-13/libmutter-mtk-13.so.0 so let's try to get its source code.
First I identify the package:
apt-file search /usr/lib/x86_64-linux-gnu/mutter-13/libmutter-mtk-13.so.0
which gives:
libmutter-13-0: /usr/lib/x86_64-linux-gnu/mutter-13/libmutter-mtk-13.so.0
libmutter-13-0: /usr/lib/x86_64-linux-gnu/mutter-13/libmutter-mtk-13.so.0.0.0
and so we obtain the debug symbols and source code with:
sudo apt install libmutter-13-0-dbgsym
but TODO hmmm that fails with:
The following packages have unmet dependencies:
libmutter-13-0-dbgsym : Depends: libmutter-13-0 (= 45.2-0ubuntu3) but 45.2-0ubuntu2~really45.0 is to be installed
E: Unable to correct problems, you have held broken packages.
Edit: it later worked when I tried again some time later. Go figure.
OK let's try another interesting frame at #11 0x00007f62b5704a5b in _XError () at /lib/x86_64-linux-gnu/libX11.so.6:
apt-file search /lib/x86_64-linux-gnu/libX11.so.6
which gives:
libx11-6: /usr/lib/x86_64-linux-gnu/libX11.so.6
libx11-6: /usr/lib/x86_64-linux-gnu/libX11.so.6.4.0
and so we get the debug symbols with:
sudo apt install libx11-6-dbgsym
and the sources with:
cd /usr/src
sudo apt source libx11-6
Then on GDB:
frame 11
shows:
#11 0x00007f62b5704a5b in _XError (dpy=0x562471d040d0, rep=<optimized out>) at ../../src/XlibInt.c:1503
1503 ../../src/XlibInt.c: No such file or directory.
so we do:
set substitute-path ../.. /usr/src/libx11-1.8.6
and then:
list
gives us source:
1498 dpy->error_threads = &thread_info;
1499 if (dpy->lock)
1500 (*dpy->lock->user_lock_display)(dpy);
1501 UnlockDisplay(dpy);
1502 #endif
1503 rtn_val = (*_XErrorFunction)(dpy, (XErrorEvent *)&event); /* upcall */
1504 #ifdef XTHREADS
1505 LockDisplay(dpy);
1506 if (dpy->lock)
1507 (*dpy->lock->user_unlock_display)(dpy);
OK let's try another one on #12 0x00007f62b1f63adb in ??? () at /lib/x86_64-linux-gnu/libGLX.so.0:
apt-file search /lib/x86_64-linux-gnu/libGLX.so.0
gives:
libglx0: /usr/lib/x86_64-linux-gnu/libGLX.so.0
libglx0: /usr/lib/x86_64-linux-gnu/libGLX.so.0.0.0
so:
sudo apt install libglx0-dbgsym
cd /usr/src
sudo apt source libglx0
then in GDB:
frame 12
gives:
#12 0x00007f62b1f63adb in __glXSendError (dpy=0x562471d040d0, errorCode=<optimized out>, resourceID=0, minorCode=<optimized out>, coreX11error=0) at ../src/GLX/libglx.c:805
805 ../src/GLX/libglx.c: No such file or directory.
so:
set substitute-path .. /usr/src/libglvnd-1.6.0
and now list gives:
>>> list
800 error.majorCode = dpyInfo->glxMajorOpcode;
801 if (!coreX11error) {
802 error.errorCode += dpyInfo->glxFirstError;
803 }
804
805 _XError(dpy, &error);
806
807 UnlockDisplay(dpy);
808 }
809
OK, it worked for two packages, we clearly see that library calling the _XError from before!
This makes me wonder: TODO how to use a different substitute-path for each separate library? Because there will be conflicts.
Related:
ExecutablePath. I thought the reader must fill the executable path there. – digital_infinity Feb 28 '19 at 08:27