Why is gdb looking for assembly files?

Question

I am attempting to learn about gdb using Ubuntu 13.10. I am using the gcc compiler and following example:

https://en.wikipedia.org/wiki/Gdb#An_example_session

Instead of the output shown in the example, I get:

Program received signal SIGSEGV, Segmentation fault.
__strlen_sse2_pminub () at ../sysdeps/x86_64/multiarch/strlen-sse2-pminub.S:38
38      ../sysdeps/x86_64/multiarch/strlen-sse2-pminub.S: No such file or directory.

Why is the debugger looking for what I assume is an assembly source file, that I apparently don't have installed?

Answer 1

Why is it looking for an assembly source file?

Because the code in question is library code that is written in assembler, and the debug information for it references that file. (In particular, it's one of [e]glibc's optimized strlen() implementations for x86-64 (aka amd64).)

Presumably, the debuggee has passed an incorrect argument to strlen(), and this has made it crash. (Theoretically it could be a problem with the implementation of strlen(), but such a fundamental function seems quite unlikely to have bugs, especially given the popularity of amd64 and the fact that [e]glibc only has 3 choices of strlen() on that arch

Why isn't that file installed?

Mostly because it is not easy to arrange for that to happen.

What follows is a fairly technical summary of the issues involved, and may expect a lot of knowledge that you do not, in fact, have; if you have questions, you can ask in #debian-mentors on OFTC.

Key facts:

• Ubuntu uses largely the same collection of source packages as Debian, with a relatively small number of packages that are not taken straight from Debian.

• Most of Ubuntu's debug info is in -dbgsym packages generated automatically at dh_strip time by pkg-create-dbgsym (which installs a wrapper around dh_strip). (This doesn't happen on Debian.) This works because 99% of packages use dh_strip to strip their binaries.

• A few key source packages explicitly generate their own -dbg packages, primarily so that these will be available on Debian, but potentially also because they need to install more than just separate DWARF info.

• eglibc is one of these key source packages.

Due to the almost limitless freedom that Debian source packages have in how they can build, it is decidedly non-trivial for a general-purpose tool to identify which files, exactly, are the relevant source files that should be bundled into a debugging-source package.

You might think that just unpacking the source package would suffice, but that wouldn't work terribly well with packages like gcc which actually contain a tarball inside their so-called .orig tarball, or even with packages that merely use a patch system more complicated than what you get with the 3.0 (quilt) format. (Another thing that would screw things up is if the upstream package generates any sources during the build that either lack #line directives, or refer to themselves in those #line directives.)

However, it seems that the binaries do contain absolute paths to all source files, so if they are still intact at this time they could be packaged. Unfortunately, I seem to have hallucinated the objcopy flags that would allow you to rewrite the source paths in the DWARF to match the install paths, so GDB still would not be able to find them.

Answer 2

Note that the wiki page says your output may be different. You can follow the discussion about this example that led to that warning being added to the Wiki page at LinuxQuestions.

In general, these differences are based on how things like libc (and maybe gdb) are compiled on different systems. Note that the example is run on Fedora, not Ubuntu; so the base system may have some key differences in how it's setup.

I think this is the main reason for the difference. In fact, I ran the example on Fedora and received output that was more or less identical to the output in the example.

The larger point is that the output you are receiving is the normal output for Ubuntu. I got the same output. You can issue the bt command from that point and the next set of output will be very similar to the output in the example.

So why is gdb looking for assembly files when you are working in C? Well, keep in mind that it's libc that is looking for the assembly file. GDB is just reporting what happened.

And libc is looking for the assembly file essentially because it is supposed to: parts of the library are implemented in assembly.

I'm not a guru in the inner workings of the the C standard library or assembly, so I want to be careful not to say something that is incorrect.

But note 2 things about this example:

1. It's about length, which by definition will get into system dependent definitions.

   If you look at the message, you will see that indeed the lib is looking into it's own internal sysdeps directory, where system dependent directories are stored . This directory is part of the compiled library and is not part of the visible filesystem on the computer.

   According to the GNU C Library documentation this structure is created based on the result of the configure utility, which determines how the library is compiled.

   If you look at the source code for libc you will see that strlen calls hidden built-in functions based on system architecture, which in turn calls another hidden built-in system dependent assembly strlen.S (I count 18 files with that name, all for different archs), which in turn calls (or may call?) an additional hidden built-in .S function for different processors. The sse and pminub relate to processor details.

2. And why isn't it finding the file? I imagine it doesn't find it because it is not there. The compiled binary doesn't include it (I presume) because it is not needed on this system and configure did not say it should be created during the build.

   The point here is that the example is passing a NULL to a function that expects a string, which is a big error. Unlike in higher-level languages, the compiler does not catch this. The result is undefined behavior, so anything can happen.

   My guess is that the library is trying to find the the version of the function that it thinks makes the most sense; but since the NULL is meaningless here, it guesses wrong and looks for a version that is not there.

   Either way it will fail with a core dump. I would also guess that the library compiled for Fedora does not exhibit exactly the same behavior because of various system differences and because any behavior is legal at this point.

Answer 3

I found the answer on this website

Here's the short version

1. From the terminal, run strace -o gdboutput.txt gdb example and go through the wiki example you linked to

2. Open gdboutput.txt and look for the text No such file or directory. There you'll find a path for strlen-sse2-pminub.S

3. Download the most recent glibc library using sudo apt-get source glibc

4. Look for strlen-sse2-pminub.S in the directory you just downloaded using find -iname strlen-sse2-pminub.S

5. Move the glibc to the location found in step 2. you might need to create the directory. Or you can change where gdb looks for those files (I don't know enough to do that yet)

Now running the gdb example won't return a file not found error. But it still didn't give me the wikipedia result. I'm using Ubuntu. It worked on an cent-os as shown in wikipedia.

I'll copy the full discussion below in case the site goes off.

---

Debugging in Ubuntu 12.10 – missing file syscall-template.S NOTE: There is an update at the bottom of this post. Please read it!

After upgrading my development machine to Ubuntu 12.04, I was disappointed to find that debugging suddenly stopped working properly. Every time I tried I got the error:

Cannot find file ‘../sysdeps/unix/syscall-template.S’

After that none of the debugging made much sense (and the repeated warnings were just getting in the way).

Life kind of got in the way and I didn’t do any C development for a good while – so much so that the 12.10 upgrade came around before I’d even thought about it any more. Sadly the problem hadn’t gone away, so after another period of inactivity I’m now fixing it.

I discovered that the file in question is in the sources for libc6. These can be obtained using the following command:

sudo apt-get source libc6

This downloads the sources to the current directory. Back to the debugger, and it still couldn’t find the files – not really a surprise, but I didn’t know where to put them. In desperation I ran strace on the debugger (nemiver, in this case) to see where it was looking for our friend. This is what I got:

stat(“/build/buildd/eglibc-2.15/misc/../sysdeps/unix/syscall-template.S”, 0x7fff0ba361f0) = -1 ENOENT (No such file or directory)
stat(“/home/test/../sysdeps/unix/syscall-template.S”, 0x7fff0ba361f0) = -1 ENOENT (No such file or directory)
stat(“/build/buildd/eglibc-2.15/misc/syscall-template.S”, 0x7fff0ba36200) = -1 ENOENT (No such file or directory)
stat(“/home/test/syscall-template.S”, 0x7fff0ba36200) = -1 ENOENT (No such file or directory)

Not knowing the inner workings of debuggers, and wanting to keep the files visible to all users, I decided to create the directory /build/buildd and put the libc6 sources in there:

sudo mkdir -p /build/buildd
cd /build/buildd
sudo mv ~/eglic* .

Success! Nemiver found the files and was happy evermore. And since nemiver is a wrapper on gdb, I checked and confirmed that that’s happy too, so if you’re one of those crazy people who can understand gdb* then this will work for you.

By the way, I have no idea whether or not this is the ‘right’ way to do this kind of thing, and I strongly suspect it isn’t. But it works so I’m happy. If someone knows what I should have done I’ll be all ears.

*My only experience of debugging prior to learning C was Java debugging, which is the nicest debugging experience you can have. After that, gdb is a scary nightmare.

UPDATE 6th May 2014: It appears that the libc6 sources are available in newer Ubuntu releases as an installable package. You should be able to just do

sudo apt-get install eglibc-source

and everything will get installed in the right place and be visible to the debugger.

This information came to me from Adventures in Code.