I have the same exact versions of numpy (1.9.3) and scipy (0.17.0.dev0+7dd2b91) installed on my laptop and on a computing cluster.
When I run scipy.test on my laptop, it completes without any failures. But when I run scipy.test on the computing cluster, it completes with a single failure, reported in this question.
I've traced the cause of this failure to the file scipy/linalg/_decomp_update.so, which is a C file (I believe . . . or C++?), not a Python file.
Hence, I've concluded that the C software on my laptop differs from that on the cluster.
My question is, what is the relevant C software? Which compiler does scipy use by default? How do I check what version of C I have installed?
Update #1
Note that the .so file is compiled. The original files in the Git repo from which I installed scipy are _decomp_update.c, _decomp_update.pyx, and _decomp_update.pyx.in.
Perhaps the relevant difference between my laptop and the cluster isn't in the C code, but in the Python package that translates between C and Python (which in this case appears to be cython)?
My laptop has cython version 0.23.2.
The cluster has cython version 0.22.1.
I am currently updating the cluster's version and re-running the tests.
Update #2
I now have cython version 0.23.3 installed on both my laptop and the cluster.
The failure persists on the cluster; it continues not to occur on my laptop.
Hence, the difference seems to be in the C implementation itself, not in Python or cython.
Because the cython docs mention gcc as the standard C compiler it uses, it makes sense to me to check this.
On the cluster, I have gcc version 4.4.7.
On my laptop, I have 4.8.4.
In the future I may want to update the cluster's version and re-run the tests.
Update #3
I aborted the update to gcc in order to invetigate whether the cluster's versions of LAPACK and BLAS differ from those on my laptop (see the comment below). I followed this answer.
This is what I see for my laptop:
$ cd /usr/local/lib/python2.7/dist-packages/scipy/linalg/
$ ldd cython_lapack.so
linux-gate.so.1 => (0xb7792000)
liblapack.so.3 => /usr/lib/liblapack.so.3 (0xb7159000)
libblas.so.3 => /usr/lib/libblas.so.3 (0xb6a7e000)
libgfortran.so.3 => /usr/lib/i386-linux-gnu/libgfortran.so.3 (0xb697f000)
libm.so.6 => /lib/i386-linux-gnu/libm.so.6 (0xb6939000)
libgcc_s.so.1 => /lib/i386-linux-gnu/libgcc_s.so.1 (0xb691c000)
libc.so.6 => /lib/i386-linux-gnu/libc.so.6 (0xb676e000)
libpthread.so.0 => /lib/i386-linux-gnu/libpthread.so.0 (0xb6752000)
libquadmath.so.0 => /usr/lib/i386-linux-gnu/libquadmath.so.0 (0xb66d6000)
/lib/ld-linux.so.2 (0xb7793000)
This is what I see on the cluster:
dbliss#nx3[~]> cd lib/python2.7/site-packages/scipy/linalg/
dbliss#nx3[linalg]> ldd cython_lapack.so
linux-vdso.so.1 => (0x00007ffe6bbec000)
liblapack.so.3 => /usr/lib64/atlas/liblapack.so.3 (0x00007fceb3f35000)
libblas.so.3 => /usr/lib64/libblas.so.3 (0x00007fceb3cdd000)
libpython2.7.so.1.0 => not found
libgfortran.so.3 => /usr/lib64/libgfortran.so.3 (0x00007fceb39eb000)
libm.so.6 => /lib64/libm.so.6 (0x00007fceb3766000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fceb3550000)
libc.so.6 => /lib64/libc.so.6 (0x00007fceb31bc000)
libf77blas.so.3 => /usr/lib64/atlas/libf77blas.so.3 (0x00007fceb2f9c000)
libcblas.so.3 => /usr/lib64/atlas/libcblas.so.3 (0x00007fceb2d7c000)
/lib64/ld-linux-x86-64.so.2 (0x000000377cc00000)
libatlas.so.3 => /usr/lib64/atlas/libatlas.so.3 (0x00007fceb2720000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fceb2502000)
There are differences here for sure, but are there any important differences?
Related
I am trying to cross compile Gstreamer 18.06 for my embedded board which is currently using Gstreamer 14.04. After I have managed to cross compiled the library, I copied over the library to /usr/lib of my embedded board. After which I have recompiled my application with the new library on the host but using the cross tool chain with the newly updated Gstreamer library. Upon running the Application, the terminal output this command
Illegal instruction (core dumped)
Steps I did to discover what went wrong:
I did an ldd Appsto find out the dependencies which my application is running
linux-vdso.so.1 (0x0000ffff93ac4000)
libutil.so.1 => /lib/libutil.so.1 (0x0000ffff93a86000)
libxml2.so.2 => /usr/lib/libxml2.so.2 (0x0000ffff93919000)
libdl.so.2 => /lib/libdl.so.2 (0x0000ffff93906000)
libgstpbutils-1.0.so.0 => /usr/lib/libgstpbutils-1.0.so.0 (0x0000ffff9388a000)
libgstsdp-1.0.so.0 => /usr/lib/libgstsdp-1.0.so.0 (0x0000ffff9384d000)
libgstrtsp-1.0.so.0 => /usr/lib/libgstrtsp-1.0.so.0 (0x0000ffff937fe000)
libgstapp-1.0.so.0 => /usr/lib/libgstapp-1.0.so.0 (0x0000ffff937cd000)
libgstreamer-1.0.so.0 => /usr/lib/libgstreamer-1.0.so.0 (0x0000ffff93550000)
libgobject-2.0.so.0 => /usr/lib/libgobject-2.0.so.0 (0x0000ffff934f4000)
libglib-2.0.so.0 => /usr/lib/libglib-2.0.so.0 (0x0000ffff933e9000)
libgstrtspserver-1.0.so.0 => /usr/lib/libgstrtspserver-1.0.so.0 (0x0000ffff93317000)
libpthread.so.0 => /lib/libpthread.so.0 (0x0000ffff932eb000)
libpng16.so.16 => /usr/lib/libpng16.so.16 (0x0000ffff932ae000)
libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x0000ffff9311b000)
libnsl.so.1 => /lib/libnsl.so.1 (0x0000ffff930f6000)
libm.so.6 => /lib/libm.so.6 (0x0000ffff9303d000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x0000ffff93019000)
libc.so.6 => /lib/libc.so.6 (0x0000ffff92ec5000)
libz.so.1 => /lib/libz.so.1 (0x0000ffff92e9e000)
/lib/ld-linux-aarch64.so.1 (0x0000ffff93a99000)
libgstvideo-1.0.so.0 => /usr/lib/libgstvideo-1.0.so.0 (0x0000ffff92d85000)
libgstbase-1.0.so.0 => /usr/lib/libgstbase-1.0.so.0 (0x0000ffff92c92000)
libgstaudio-1.0.so.0 => /usr/lib/libgstaudio-1.0.so.0 (0x0000ffff92bbc000)
libgsttag-1.0.so.0 => /usr/lib/libgsttag-1.0.so.0 (0x0000ffff92b42000)
libgstrtp-1.0.so.0 => /usr/lib/libgstrtp-1.0.so.0 (0x0000ffff92af2000)
libgio-2.0.so.0 => /usr/lib/libgio-2.0.so.0 (0x0000ffff9297d000)
libgmodule-2.0.so.0 => /usr/lib/libgmodule-2.0.so.0 (0x0000ffff92969000)
libunwind.so.8 => /usr/lib/libunwind.so.8 (0x0000ffff92916000)
libdw.so.1 => /usr/lib/libdw.so.1 (0x0000ffff928ca000)
libpcre.so.1 => /usr/lib/libpcre.so.1 (0x0000ffff9287b000)
libffi.so.6 => /usr/lib/libffi.so.6 (0x0000ffff92863000)
libgstnet-1.0.so.0 => /usr/lib/libgstnet-1.0.so.0 (0x0000ffff92827000)
libresolv.so.2 => /lib/libresolv.so.2 (0x0000ffff92803000)
libmount.so.1 => /lib/libmount.so.1 (0x0000ffff927a6000)
libelf.so.1 => /usr/lib/libelf.so.1 (0x0000ffff92780000)
libbz2.so.1 => /usr/lib/libbz2.so.1 (0x0000ffff9275e000)
libblkid.so.1 => /lib/libblkid.so.1 (0x0000ffff92709000)
libuuid.so.1 => /lib/libuuid.so.1 (0x0000ffff926f2000)
librt.so.1 => /lib/librt.so.1 (0x0000ffff926db000)
I did a sampling check on library (libgst****) and their link in the embedded
( using file libgst**** and found they were of 18.06.
For example
/usr/lib# file libgstreamer-1.0.so.0
libgstreamer-1.0.so.0: symbolic link to libgstreamer-1.0.so.0.1806.0
I tried to correct those with older links to 14.04 by doing, eg
/usr/lib# rm libgstreamer-1.0.so.0
/usr/lib# ln -s libgstreamer-1.0.so.0.1806.0 libgstreamer-1.0.so.0
I went back to my host machine and did the same as above to ensure that the libraries are linked to latest library and found they were basically correct
However the result were similar
Question
Is my steps of copying, done correctly?
What else have I missed?
How to use ldd in my host machine to find out the exact dependencies used
REgards
====================================================================
EDIT
I have tried to search around for Illegal instruction and found they were due likely due to wrong compiler set. I doublecheck my library and found they were similar build
(the updated version)
/usr/lib$ file libgstreamer-1.0.so.0.1806.0
libgstreamer-1.0.so.0.1806.0: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, BuildID[sha1]=9e9408daeaa2c048df31ae41d415ec28e8bc5755, stripped
(the previous version)
/usr/lib$ file libgstreamer-1.0.so.0.1404.0
libgstreamer-1.0.so.0.1404.0: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, BuildID[sha1]=70c4b48457b63c38e52fb503ce01095e9cdbab0b, stripped
I've developed a project in C/C++ for Linux environment and it works great on my machine. I decided to move this application to another machine (VPS). After first run I realized that the new machine doesn't have some dependencies of this application. Some might suggest linking your application statically but this doesn't work for me because of some technical problems I have. So my question is how can I move my application with all dependencies preferably packed like a setup program in Windows and install it on the new machine?
This is my program's dependency list :
linux-vdso.so.1 (0x00007ffdcf123000)
libmicrohttpd.so.12 => /usr/local/lib/libmicrohttpd.so.12 (0x00007f653074a000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f6530359000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f653013a000)
libgnutls.so.30 => /usr/lib/x86_64-linux-gnu/libgnutls.so.30 (0x00007f652fdd4000)
/lib64/ld-linux-x86-64.so.2 (0x00007f6530b71000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f652fbb7000)
libp11-kit.so.0 => /usr/lib/x86_64-linux-gnu/libp11-kit.so.0 (0x00007f652f888000)
libidn2.so.0 => /usr/lib/x86_64-linux-gnu/libidn2.so.0 (0x00007f652f66b000)
libunistring.so.2 => /usr/lib/x86_64-linux-gnu/libunistring.so.2 (0x00007f652f2ed000)
libtasn1.so.6 => /usr/lib/x86_64-linux-gnu/libtasn1.so.6 (0x00007f652f0da000)
libnettle.so.6 => /usr/lib/x86_64-linux-gnu/libnettle.so.6 (0x00007f652eea4000)
libhogweed.so.4 => /usr/lib/x86_64-linux-gnu/libhogweed.so.4 (0x00007f652ec70000)
libgmp.so.10 => /usr/lib/x86_64-linux-gnu/libgmp.so.10 (0x00007f652e9ef000)
libffi.so.6 => /usr/lib/x86_64-linux-gnu/libffi.so.6 (0x00007f652e7e7000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f652e5e3000)
I am trying to run some Command line:
gst-launch-1.0 playbin uri=file:///home/xxx/VSCode/C++/Gstreamer/resource/traffic.avi
But there was some problem, it complained it couldn't find avi demuxer
Missing element: Audio Video Interleave (AVI) demuxer
I thought that it was missing and i tried to reinstall the program again using this link for Ubuntu. However everything was ok and the file was up-to-date
When I tried to see how many plugins I have using gst-inspect,
>>>>> gst-inspect-1.0
...............
Total count: 25 plugins, 253 features
Is it normal to have only 25 plugins? Seem to be missing quite a bit of plugins or is there some linking process you would need to do when installing gstreamer
Thanks
========================================================================
Edited: To present more diagnostic information as suggested
Location of gst-inspect-1.0
>> which gst-inspect-1.0
>> /home/xxx/anaconda3/bin/gst-inspect-1.0
It is quite surprising to me that gst-inspect-1.0 is over in this folder, anaconda
Version of gst inspect
>> gst-inspect-1.0 --version
gst-inspect-1.0 version 1.14.0
GStreamer 1.14.0
Unknown package origin
Dynamic link required
>> ldd $(which gst-inspect-1.0)
linux-vdso.so.1 (0x00007fff81cef000)
libgstreamer-1.0.so.0 => /home/xxx/anaconda3/bin/.. /lib/libgstreamer-1.0.so.0 (0x00007ff0bbcf5000)
libgobject-2.0.so.0 => /home/xxx/anaconda3/bin/../lib/libgobject-2.0.so.0 (0x00007ff0bc40d000)
libglib-2.0.so.0 => /home/xxx/anaconda3/bin/../lib/libglib-2.0.so.0 (0x00007ff0bc2f0000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007ff0bbad6000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ff0bb6e5000)
libffi.so.6 => /home/xxx/anaconda3/bin/../lib/./libffi.so.6 (0x00007ff0bb4dc000)
libpcre.so.1 => /home/xxx/anaconda3/bin/../lib/./libpcre.so.1 (0x00007ff0bc277000)
libgmodule-2.0.so.0 => /home/xxx/anaconda3/bin/../lib/./libgmodule-2.0.so.0 (0x00007ff0bc270000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007ff0bb13e000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007ff0baf36000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007ff0bad32000)
/lib64/ld-linux-x86-64.so.2 (0x00007ff0bc23e000)
I've wrote a tool using GTK for a company project, however it seems that the entire GTK package or git is around 250mb and is required to compile.
This isn't a portable solution, to move 250mb to a machine, just to compile and then get rid of the library, so how do I go about making my program portable?
*This isn't a repeat question, I'm yet to find a solution on Linux.
Credit to gengisdave in the question comments section; the solution is to use ldd.
In my case: ldd /dir/blablah/program_name - this produced an output like:
mikearmstrong#ubuntu:~/Desktop/linux_command_tool$ ldd ProntoCommandTool
linux-vdso.so.1 => (0x00007ffdb25f0000)
libgtk-3.so.0 => /usr/lib/x86_64-linux-gnu/libgtk-3.so.0 (0x00007ffa4717f000)
libgobject-2.0.so.0 => /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0 (0x00007ffa46f2e000)
libglib-2.0.so.0 => /lib/x86_64-linux-gnu/libglib-2.0.so.0 (0x00007ffa46c25000)
libcups.so.2 => /usr/lib/x86_64-linux-gnu/libcups.so.2 (0x00007ffa469b1000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007ffa46793000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ffa463cd000)
libgdk-3.so.0 => /usr/lib/x86_64-linux-gnu/libgdk-3.so.0 (0x00007ffa46126000)
libgmodule-2.0.so.0 => /usr/lib/x86_64-linux-gnu/libgmodule-2.0.so.0 (0x00007ffa45f22000)
libpangocairo-1.0.so.0 => /usr/lib/x86_64-linux-gnu/libpangocairo-1.0.so.0 (0x00007ffa45d14000)
libX11.so.6 => /usr/lib/x86_64-linux-gnu/libX11.so.6 (0x00007ffa459df000)
and you can see that each library is declared within the return string.
Odd situation here...
On Centos5, we have an app that uses 2 different, specific versions of openssl (1.0.1h and 0.9.8u to be specific) to make requests to servers. We build both versions ourselves and install them to a non-standard location (/opt/foo/first-openssl and /opt/foo/other-openssl for example)
The config/build/install commands look like this:
$ ./openssl-1.0.1h/Configure --prefix=/opt/foo/first-openssl --with-krb5-flavor=MIT --with-krb5-dir=/opt/foo/krb5 shared zlib enable-rc5 enable-camellia enable-capieng enable-cms enable-mdc2 enable-idea enable-experimental enable-seed no-asm linux-generic64 -m64 -Wl,-rpath=/opt/foo/zlib/lib -Wl,-rpath=/opt/foo/krb5/lib -Wl,-rpath=/opt/foo/first-openssl/lib
$ make depend && make
$ make install
$ ./openssl-0.9.8u/Configure --prefix=/opt/foo/other-openssl --with-krb5-flavor=MIT --with-krb5-dir=/opt/foo/krb5 shared zlib enable-rc5 enable-camellia enable-capieng enable-cms enable-mdc2 enable-idea enable-experimental enable-seed no-asm linux-generic64 -m64 -Wl,-rpath=/opt/foo/zlib/lib -Wl,-rpath=/opt/foo/krb5/lib -Wl,-rpath=/opt/foo/other-openssl/lib
$ make depend && make
$ make install
However, I'm hitting a segfault later on because the app calls an other-openssl function which in turn calls a function in the first openssl's compiled library for crpyto. From gdb:
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x2aaab6dfb940 (LWP 15836)]
0x00002aaab1905bd4 in engine_unlocked_finish () from /opt/foo/first-openssl/lib/libcrypto.so.1.0.0
(gdb) bt
#0 0x00002aaab1905bd4 in engine_unlocked_finish () from /opt/foo/first-openssl/lib/libcrypto.so.1.0.0
#1 0x00002aaab1905cc1 in ENGINE_finish () from /opt/foo/first-openssl/lib/libcrypto.so.1.0.0
#2 0x00002aaab1919895 in EVP_DigestInit_ex () from /opt/foo/first-openssl/lib/libcrypto.so.1.0.0
#3 0x00002aaab701f90b in ssl3_connect () from /opt/foo/other-openssl/lib/libssl.so
#[snip]
But everything looks to be linked properly in /opt/foo/other-openssl/lib...
# ldd /opt/foo/other-openssl/lib/libssl.so
linux-vdso.so.1 => (0x00007fffd11fd000)
libcrypto.so.0.9.8 => /opt/foo/other-openssl/lib/libcrypto.so.0.9.8 (0x00002ab91fbd3000)
libgssapi_krb5.so.2 => /opt/foo/lib/libgssapi_krb5.so.2 (0x00002ab91ff7f000)
libkrb5.so.3 => /opt/foo/lib/libkrb5.so.3 (0x00002ab9201d5000)
libcom_err.so.3 => /opt/foo/lib/libcom_err.so.3 (0x00002ab9204c0000)
libk5crypto.so.3 => /opt/foo/lib/libk5crypto.so.3 (0x00002ab9206c4000)
libdl.so.2 => /lib64/libdl.so.2 (0x00002ab920915000)
libz.so.1 => /opt/foo/lib/libz.so.1 (0x00002ab920b19000)
libc.so.6 => /lib64/libc.so.6 (0x00002ab920d35000)
libkrb5support.so.0 => /opt/foo/lib/libkrb5support.so.0 (0x00002ab92108e000)
libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00002ab92129a000)
libresolv.so.2 => /lib64/libresolv.so.2 (0x00002ab92149d000)
/lib64/ld-linux-x86-64.so.2 (0x00000036b6a00000)
I do see quite a bit of:
gcc: -rpath: linker input file unused because linking not done
gcc: /opt/foo/lib: linker input file unused because linking not done
gcc: -rpath: linker input file unused because linking not done
gcc: /opt/foo/other-openssl/lib: linker input file unused because linking not done
While running the commands for 'other-openssl' that I don't see for 'first-openssl'
Am I doing something stupid with how I build these libraries? We don't do any customization/patching, and I can't figure out for the life of me why the 'other-openssl' library is calling out to the 'first-openssl' library.