This is a linkage question rather than a uwsgi question. But I will explain the story.
I am using uwsgi to host my flask app. After running some weeks on production, I found that my app have some slight memory leak; its RSS size reported by uswgitop is increasing.
Although it is a not big deal as it is only around 50MiB to 60MiB, I want to find out why it is increasing.
After some work, I found that it might due to memory fragmentation of the malloc implementation in libc.
So I am considering using tcmalloc or jemalloc, which is famous for the memory fragmentation management.
Then I came to build uwsgi on my own. It produced the following linking command.
x86_64-linux-gnu-gcc -pthread -o uwsgi -L/usr/lib -Wl,-rpath,/usr/lib .......tons of object files
-lpthread -lm -rdynamic -ldl -lz -ltcmalloc
-L/home/alex/local/lib -lpcre -lssl -lcrypto -lxml2 -lpthread -ldl -lutil -lm -lpython2.7 -lcrypt
As you can see from the comand, it explicitly links against tcmalloc, which is a dynamic link not a static one.
As far as I know, tcmalloc implements the standard posix malloc interface, that is also why we can use LD_PRELOAD to hook into one application to replace its malloc implementation.
My question here is:
Why we need to explicitly specify to link against tcmalloc when we can use LD_PRELOAD to do so?
If you are interesting only in standard API, there is no difference.
But tcmalloc has more than that, like heap checking, heap profiling, allocation functions hooks etc. Natural way to use it is linking with libtcmalloc.
Related
I have a problem which I think I can solve if I manage to get from pkg-config the list in Requires.private without being recursive. First I'll post the problem, and then my try to workaround it.
Problem:
I had a static-only library (let's call it libfoo.a) with no pkg-config files, which I just included as a submodule of my programs. I had to take into account in the program makefile all the dependencies that my library might have, because as a static library, it didn't carry the information of which shared library it depended on. That meant that I had to dinamically link with a lot of libraries in my programs' makefiles, but that worked.
Example:
$(CC) $(OBJS) -o $# -L path/foo/ -l foo `pkg-config --libs opencv gsl ncurses`
It worked, because it could only find a static version of libfoo, and after that it had a list of what libfoo needed (instead of what the program actually needed dinamically).
Now I have improved that library so that it is installed in /usr/local/, provide pkg-config files, and both .a and .so files.
When I link dinamically to my library, everything is fine.
When I try to link statically, if the program doesn't use opencv at all, it is fine (I don't know if it's opencv that is broken when trying to link statically, or that I broke it installing some updates from the testing repo; if I had a clean installation I would know, but I don't). The problem comes when I try to link statically, and some opencv is used. There are some libraries not found, so what I thought as a solution is to replicate what happened before I installed my library.
But now I don't want to hardcode my library's dependencies in my program's makefile, so I use pkg-config.
My try:
Assuming that my program only depends directly on libfoo, and libfoo depends on opencv, gsl and ncurses directly (which my program ignores), this is what I would do:
$(CC) $(OBJS) -o $# -Wl,-Bstatic `pkg-config --libs foo` -Wl,-Bdynamic `pkg-config --static --libs foo`
the first pkg-config is to expose only libfoo to be linked statically, and the second one is to expose libfoo's dependencies to be linked dinamically. But the problem is that I'm exposing, not only libfoo's direct dependencies, but also all the recursive dependencies, which I don't want to.
Is there any way to expose only the direct dependencies? Or is there any other workaround to this situation?
In this case, it is needed to the problem with a broken openCV, and someone could say: hey, just solve that problem with openCV, and forget about that.
But this is also useful for someone who wants to link statically to a library which performance is critical, but still link dinamically to other heavier libraries which aren't as critical.
System:
Debian 10
GCC 8
OpenCV might have some packages or dependencies installed from testing.
I just bumped into this, and I think I'm going to have to write a bit of Python to just repeatedly call pkg-config.
I don't understand the difference between the two types of libraries, and many websites say that they are the same thing, but at school we use two different commands to create them
dynamic library
$ gcc -shared -o libsample.so lib.c
$ gcc -o main main.c -ldl
to execute:
$ ./main ./libsample.so
shared library
$ gcc -shared -o libsample.so lib.c
$ gcc -o main main.c -L. -lsample
to execute:
$ LD_LIBRARY_PATH=. ./main
Can someone help me in understanding the difference between the two "codes"?
Dynamic Linked Library (.DLL) is the terminology used by Microsoft Windows. Shared Object (.so) is the terminology used by Unix and Linux.
Other than that, conceptually they're the same.
Regarding your snippets of commands, I guess the difference (and I'm only guessing here, because you didn't show us the relevant parts) is how the library is loaded. There is "link time loading" where the library is tied to the executable by the linker¹. And there is "runtime loading", where the program sort of "ingests" the dynamic/shared library.
runtime loading is done in Windows with the LoadLibrary (there's an …A and a …W variant) function, and on Unix/Linux with dlopen (which is made available by libdl which is linked to by that -ldl library link statement).
1: The linker is the program that creates the actually executable file from the intermediary objects created by the various compiler stages.
Dynamic and shared libraries are usually the same. But in your case, it looks as if you are doing something special.
In the shared library case, you specify the shared library at compile-time. When the app is started, the operating system will load the shared library before the application starts.
In the dynamic libary case, the library is not specified at compile-time, so it's not loaded by the operating system. Instead, your application will contain some code to load the library.
The first case is the normal case. The second case is a special use and it's mainly relevant if your application support extensions such a plug-ins. The dynamic loading is required because there can be many plug-ins and they are built after your application. So their names are not available at compile-time.
I'm exploring using clang as the compiler for ARM embedded development. As clang doesn't have an equivalent of .spec files, I'm having trouble convincing clang to link against libc_nano. How could I either tell clang to not link against any libraries by default so I can specify the correct library, or rewrite the -lc command to -lc_nano?
The command I'm trying to run is:
clang -target arm-none-eabi -mcpu=cortex-a5 -mfpu=neon-vfpv4 -mfloat-abi=hard -march=armv7-a main.c
Currently I get this error message:
/usr/lib/llvm-6.0/bin/ld.lld: error: unable to find library -lc
EDIT: I've noticed that clang has a -fno-autolink which according to the help text: Disable generation of linker directives for automatic library linking. However it doesn't seem to do anything?
EDIT2: I'm aware I could abuse symlinks to achieve this. I would like to avoid symlinks in this case as it can make the build system brittle.
Upon further google-fu and grep-fu, it turns out the answer was staring at me the entire time. Clang has a -nodefaultlibs that does the trick and prevents default linker directives. Although strangely it wasn't documented in --help.
You can build fake libc.a without any functions inside and use it together with libc_nano.
I am in the process of updating an arm cross compiler from 4.3.3 to 4.9.4. One issue I am seeing is that the new compiler no longer automatically includes dependent libraries. For example:
gcc ... -L -l -lssl -lrt
works fine with the previous compiler. If libssl needed to reference something in libcrypto, then the linker would automatically find and link with libcrypto (no -lcrypto needed).
With the new compiler, this still works, but only if libssl does not reference anything in libcrypto. If it does, then the -lcyrpto is required. The same issue applies to -lpthread, -ldl, etc.
Is this a change in the behavior of gcc or is something not configured properly when building gcc?
Are you using static or dynamic libraries? For dynamic libraries, if libssl depends on libcrypto, you don't need to explicitly link -lcrypto as long as libssl itself was correctly linked, but if you want to make direct use in your program of symbols from libcrypto, then you have to explicitly link it. This is a change/intentional-regression in newer versions of binutils.
I need to statically link glibc to my project, because the target platform supports only a very old one ( but it works with statically linked glibc from my toolchain, I have checked it)
Unfortunately, this application has to make use of pthread library, but statically linked libpthread takes too much space.
I would like to statically link glibc, and dynamically pthread.
After running this command
powerpc-unknown-linux-gnu-gcc object_files -lrt -lpthread -Wl,-Bstatic -lc
I get:
/powerpc-unknown-linux-gnu/bin/ld: cannot find -lgcc_s
There is a -static-libgcc if that may help
You should be using -static, not -Wl,-static. The latter bypasses gcc's knowledge, and therefore gcc is still trying to link the shared libgcc_s.so rather than the static libgcc_eh.a.
If your aim is to link libc statically but libpthread dynamically, this is simply not going to work. You cannot mix and match different versions of libpthread; it's part of glibc, just a separate file, and the internals need to match. Even with the same version, I think linking libc statically and libpthread dynamically will be very broken.
If glibc is too big for your needs, you could try an alternate libc like uClibc or musl.