CUDA nvprof profiler in 12.04

Question

I have the latest CUDA toolkit and drivers installed on a 12.04 server. The drivers are working fine: all the NVIDIA sample code compiles and runs and I've written, compiled, and run several CUDA programs. The other day I went to use the new nvprof command line profiler and was greeted with the following error:

Error: unable to locate profiling library libcuinj64.so.5.0.35.

I double checked the CUDA libraries and that specific library is in fact included in the LD_LIBRARY_PATH. (I used the method described here). Running ldconfig -v shows that all the Cuda libraries are loaded into LD_LIBRARY_PATH:

/usr/local/cuda-5.0/lib:
    libcurand.so.5.0 -> libcurand.so.5.0.35
    libnpp.so.5.0 -> libnpp.so.5.0.35
    libcusparse.so.5.0 -> libcusparse.so.5.0.35
    libcufft.so.5.0 -> libcufft.so.5.0.35
    **libcuinj32.so.5.0 -> libcuinj32.so.5.0.35**
    libcudart.so.5.0 -> libcudart.so.5.0.35
    libnvToolsExt.so.5.0 -> libnvToolsExt.so.5.0.35
    libcublas.so.5.0 -> libcublas.so.5.0.35
/usr/local/cuda-5.0/lib64:
    libcurand.so.5.0 -> libcurand.so.5.0.35
    libcuinj64.so.5.0 -> libcuinj64.so.5.0.35
    libnpp.so.5.0 -> libnpp.so.5.0.35
    libcusparse.so.5.0 -> libcusparse.so.5.0.35
    libcufft.so.5.0 -> libcufft.so.5.0.35
    libcudart.so.5.0 -> libcudart.so.5.0.35
    libnvToolsExt.so.5.0 -> libnvToolsExt.so.5.0.35
    libcublas.so.5.0 -> libcublas.so.5.0.35

The offending library is in bold. At this point the old command line compiler works fine, the compiler itself works fine, so it seems to be nvprof specific.

I'm not sure if this is Ubuntu or the nvprof build provided by NVIDIA. Does anyone have an experience with running nvprof under 12.04 or even 12.10? Anyone seen this problem before? I realize this is a fairly niche question but you never know.

UPDATE: This problem seems to persist on Ubuntu 12.04 with CUDA 5.5 (released on August 1, 2013). The fix still works, you just have to use the updated library (see below).

Update for CUDA 6.0: This issue appears to be fixed for the 6.0 release candidate :). I installed the RC version using the deb provided by NVIDIA. — Logan Mayfield, Mar 15 '14 at 12:59

score 2 · Accepted Answer · edited Aug 01 '13 at 22:23

I know this is old, but this is still the first hit on google for this problem and I wanted to let people know how to fix it. Looking at strace told me that it wasn't finding the library in any of the paths (looks like it is looking through a different set of folders for some reason). Regardless, I just linked my copy of the library into one of the directories that it was looking at, and it worked perfectly fine. I ran this code:

sudo ln -s `locate libcuinj64.so.5.0.35` /usr/lib/x86_64-linux-gnu/libcuinj64.so.5.0.35

You can just run the locate command on its own if you want to put it in manually, but if you only have one copy of the library this will work fine. Create the upper level directory first if you don't have it for some reason:

sudo mkdir -p /usr/lib/x86_64-linux-gnu/

Hope this helps!

Update for CUDA 5.5: The library for CUDA 5.5 is libcuinj64.so.5.5.22. So, the one line command is:

sudo ln -s `locate libcuinj64.so.5.5.22` /usr/lib/x86_64-linux-gnu/libcuinj64.so.5.5.22

score 1 · Answer 2 · answered Apr 10 '13 at 16:16

1

It's probably due to a not correctly set $LD_LIBRARY_PATH. This environment variable should include the path to the CUDA library. Example:

setenv QUDA_INSTALL_PATH /usr/local/cuda/lib64 

setenv LD_LIBRARY_PATH $QUDA_INSTALL_PATH/lib64:$LD_LIBRARY_PATH

answered Apr 10 '13 at 16:16

ritter

353

Like I said, I'm pretty sure that is not the problem. I added some details to the question to show how I ruled that out. – Logan Mayfield Apr 10 '13 at 17:27
Maybe it's a 32bit vs 64bit issue? Also you could try using the LD_PRELOAD trick – ritter Apr 10 '13 at 18:49
Tried $ LD_PRELOAD=/path/to/lib nvprof myprogram and got the same error. So if I did the trick right, that's a no go as well. – Logan Mayfield Apr 11 '13 at 15:23

CUDA nvprof profiler in 12.04

2 Answers2