A while back I asked a question about this subject and "solved" it by using Cygwin instead with its XWin utility, but I've come back to this issue again since the Xwin utility does not use my GPU and creates a severe bottleneck in simulations as a result. MinGW/MSYS on the other hand DOES use my GPU for rendering, which is a huge help, but there are some rough areas that need smoothing over, specifically with readlink.
Basically, the src/makefile for rebound (https://github.com/hannorein/rebound) says this:
PREDEF+= -D$(shell basename `readlink gravity.c` '.c' | tr '[a-z]' '[A-Z]')
PREDEF+= -D$(shell basename `readlink boundaries.c` '.c' | tr '[a-z]' '[A-Z]')
PREDEF+= -D$(shell basename `readlink collisions.c` '.c' | tr '[a-z]' '[A-Z]')
If my understanding is correct, this is supposed to find which version of gravity, boundaries and collisions I specified, and adds that to PREDEFS so the compiler uses the right versions of gravity, boundaries and collisions. However, it does not seem to work in MSYS. What it ends up spitting out for predefs is this:
-DOPENGL -D.C -D.C -D.C
Obviously it did not get anything back from the code above. This results in a macronames must be identifiers error, of course. I can work around this by adding any of the special options in between readlink and the filename, like -f, for instance, but then it only spits out
-DOPENGL -DGRAVITY -DBOUNDARIES -DCOLLISIONS
Which is not right because it should have extra bits, like so:
-DOPENGL -DGRAVITY_DIRECT -DBOUNDARIES_OPEN -DCOLLISIONS_NONE
Now, if I don't want any special gravity, boundaries or collisions, the workaround is okay, but only because (I'm guessing) it defaults to those if there's nothing special specified after each macroname. But if I DO want something special, like the more efficient gravity tree code, or actual collisions, the shortened name resulting from the workaround will not help it find anything, and so it causes errors in compiling as certain functions it needed from the special files obviously are missing.
And so I'm pretty stuck at the moment. I would like very much to be able to use other codes than the defaults, but MSYS is acting funny with the readlink and not finding the right stuff. As I said, it worked fine in an X windows style compiler. I feel like there must be some library I'm missing or some hidden syntax disconnect I'm overlooking that needs to be accounted for between XWin and non-Xwin compiling, but I can't find anything.
Here's an example of the links it should be reading (at least I think this is what is being read, I'm still learning makefiles):
ln -fs gravity_tree.c ../../src/gravity.c
ln -fs boundaries_open.c ../../src/boundaries.c
ln -fs collisions_none.c ../../src/collisions.c
If anyone can tell me why this would work on an Xwin command line but not MSYS, I'd greatly appreciate it.
Why on earth do you expect readlink to work in MSYS? Where did you even get whatever readlink.exe is being invoked, (if that is what is being executed)? There is no readlink command in a standard MSYS installation. Perhaps you discovered it in MinGW.org's msys-coreutils-ext package? If this is the case, you should note the comment within the description of that package, (as seen via MinGW.org's mingw-get installer):
The msys-coreutils-bin subpackage contains those applications that were historically part of the standard MSYS installation. The associated msys-coreutils-ext subpackage contains the rest of the coreutils applications that have been (nominally) ported to MSYS -- usually these are less often used, and are not guaranteed to work: e.g. 'su.exe', 'chroot.exe' and 'mkfifo.exe' are known to be broken.
and, it seems that we may add readlink.exe to that list of "known to be broken" applications.
It may also be worth noting that readlink is not among the list of supporting tools, which a GNU Coding Standards conforming application is permitted to invoke from either its configure script, or its makefile. Thus, there is little incentive for the MinGW.org developers, (who maintain MSYS), to address the issue of making readlink.exe work, (although patches from an independent developer, with such an incentive, would be welcomed).
As a final qualification, and as one comment on the question notes, ln -s creates copies of files; it does not create symbolic links. How could it? MSYS itself dates from an era when windows didn't support symbolic links ... indeed, even today its support for them is flaky. At the time when MSYS was published, either copying the files, or creating NTFS hard links, was the best compromise MSYS could offer, in the situation where a script invoked ln -s. Consequently, it would become incumbent upon any developer submitting patches to make readlink.exe work, to also address the issue of updating ln.exe, such that it could create the symbolic links, (in an OS version dependent fashion), which readlink.exe would then read.
I'm sorry if this isn't the answer you hoped for, but unless someone devotes some effort into updating MSYS, so that it can make use of the (unreliable) symbolic link feature in more recent windows versions, then you need to find a different approach; current MSYS does not support symbolic links, even if the underlying OS now does.
Related
After making a mistake of replying to a post from 2010 (the link to it: What does it mean to 'touch' a target in make? ), i would like to understand something as a new student for C programming language, who recently learned about the makefile, make command an so on. i've got a question on class about what does the make -t command do, and why should i use it (or not, for the matter).
i understand that make -t only create the files (the relevant ones, from my makefile commands and full script). but why should i ever use it?
i mean, if it is just creating me the files, without actually doing anything with it - why bother?
(I'm using Ubuntu 20.04.4 LTS, gcc for compiling, and writing my code in C language)
As mentioned in comments you should pretty much never use this.
It is "useful" in very limited situations such as: you know for a fact that your entire build is is correct and up to date, then something happens such that the timestamps on your files get all messed up. Maybe some tool went in and tweaked a comment in every file (maybe something changed the copyright year in every file), or maybe you copied the build tree somewhere but forgot to preserve the modification time, or whatever.
Then you can run make -t to "bring back" the relative timestamps of your files so that make understands everything is up to date, without actually building anything.
Back in the day when builds were a lot slower and there were more opportunities to mess up timestamps maybe this was more useful.
These days it's better to just run make without -t: yes you'll have to rebuild a bunch of files but it's much safer than assuring make you know that everything is up to date, when you might be wrong.
Basically, I want to get a list of libraries a binary might load.
The unreliable way I came up with that seems to work (with possible false-positives):
comm -13 <(ldd elf_file | sed 's|\s*\([^ ]*\)\s.*|\1|'| sort -u) <(strings -a elf_file | egrep '^(|.*/)lib[^:/]*\.so(|\.[0-9]+)$' | sort -u)
This is not reliable. But it gives useful information, even if the binary was stripped.
Is there a reliable way to get this information without possible false-positives?
EDIT: More context.
Firefox is transitioning from using gstreamer to using ffmpeg.
I was wondering what versions of libavcodec.so will work.
libxul.so uses dlopen() for many optional features.
And the library names are hard-coded. So, the above command helps
in this case.
I also have a general interest in package management and binary dependencies.
I know you can get direct dependencies with readelf -d, dependencies of
dependencies with ldd. And I was wondering about optional dependencies, hence the question.
ldd tells you the libraries your binary has been linked against. These are not those that the program could open with dlopen.
The signature for dlopen is
void *dlopen(const char *filename, int flag);
So you could, still unreliably, run strings on the binary, but this could still fail if the library name is not a static string, but built or read from somewhere during program execution -- and this last situation means that the answer to your question is "no"... Not reliably. (The name of the library file could be read from the network, from a Unix socket, or even uncompressed on the fly, for example. Anything is possible! -- although I wouldn't recommend any of these ideas myself...)
edit: also, as John Bollinger mentioned, the library names could be read from a config file.
edit: you could also try substituting the dlopen system call with one of yours (this is done by the Boehm garbage collector with malloc, for example), so it would open the library, but also log its name somewhere. But if the program didn't open a specific library during execution, you still won't know about it.
(I am focusing on Linux; I guess that most of my answer fits for every POSIX systems; but on MacOSX dlopen wants .dylib dynamic library files, not .so shared objects)
A program could even emit some C code in some temporary file /tmp/foo1234.c, fork a compilation of that /tmp/foo1234.c into a shared library /tmp/foo1234.so by some gcc -O -shared -fPIC /tmp/foo1234.c -o /tmp/foo1234.so command -generated and executed at runtime of your program-, perhaps remove the /tmp/foo1234.c file -since it is not needed any more-, and dlopen that /tmp/foo1234.so (and perhaps even remove /tmp/foo1234.so after dlopen), all that in the same process. My GCC MELT plugin for gcc does exactly this, and so does Bigloo, and the GCCJIT library is doing something close.
So in general, your quest is impossible and even has no sense.
Is there a reliable way to get this information without possible false-positives?
No, there is no reliable way to get such information without false positives (you could prove that equivalent to the halting problem, or to some other undecidable problem). See also Rice's theorem.
In practice, most dlopen happens on plugins provided by some configuration. There might not be exactly named as such in a configuration file (e.g. some Foo programs might have a convention like a plugin named bar in some foo.conf configuration file is provided by foo-bar.so plugin).
However, you might find some heuristic approximation. Most programs doing some dlopen have some plugin convention requesting some particular symbol names in the plugin. You could search for shared objects defining these names. Of course you'll get false positives.
For example, the zsh shell accepts plugins called zsh modules. the example module shows that enables_,
boot_, features_ etc... functions are expected in zsh modules. You could use nm -D to find *.so files providing these (hence finding the plugins likely to be perhaps loadable by zsh)
(I am not convinced that such an approach is worthwhile; in fact you should usually know which plugins are useful on your system by which applications)
BTW, you could use strace(1) on the execution of some command to understand the syscalls it is doing, hence the plugins it is loading. You might also use ltrace(1), or pmap(1) (on some given process), or simply -for a process 1234- use cat /proc/1234/maps to understand its virtual address space, hence the plugins it has already loaded. See proc(5).
Notice that strace, ltrace, pmap exist on Linux, but many POSIX systems have similar programs.
Also, a program could generate some machine code at runtime and execute it (SBCL does that at every REPL interaction!). Your program could also use some JIT techniques (e.g. with libjit, llvm, asmjit, GCCJIT or with hand-written code...) to do likewise. So plugin-like behavior can happen without dlopen (and you might mimic dlopen with mmap calls and some ELF relocation processing).
Addenda:
If you are installing firefox from its packaged version (e.g. the iceweasel package on Debian), its package is likely to handle the dependencies
Just as a learning experience, I'm trying to code the following problems in C.
If two flash drives are inserted each having a folder (say,
Course_Notes), then they get synced. That is, data is copied from one
to the other and if there a file already exists, then the newer one is
retained.
I would do this in bash by:
#!/bin/bash
while $1
do
cp -ur /media/PD_1/Course_Notes /media/PD_2/Course_Notes
cp -r /media/PD_2/Course_Notes /media/PD_1/Course_Notes
done
How do I do this in C without too many system calls ?
In the real world, you'd probably use something like rsync for this.
You're probably looking for the stat() family of functions to get size and modification date, along with opendir()/readdir()/closedir() for directory listings. The standard way for copying a file itself is to open the source and destination, and write to the latter what you read from the former with the usual read and write functions.
Note also that the source code for GNU and BSD versions of the standard UNIX utilites, such as cp, is freely available. (likewise for rsync) You may want to read some of that source code to see what you may have missed in your own approach.
Since this is a programming Q&A site, you would use usual programming tools (instead of rewriting them yourself).
This sounds like a perfect application for a Distributed Version Control System. Some of the most popular choices today are Git and Mercurial.
I have some c code which provides libfoo.so and libfoo.a along with the header file foo.h. A large number of clients currently use these libraries from /old_location/lib and /old_location/include directories which is where they are disted.
Now I want to move this code to /new_location. Yet I am not in a position to inform the clients about this change. I would want the old clients to continue accessing the libs and headers from the /old_location.
For this, will creating symlinks to the libs/headers to the new locations work?
/old_location/lib/libfoo.so -> /new_location/lib/libnewfoo.so
/old_location/lib/libfoo.a -> /new_location/lib/libnewfoo.a
/old_location/inlcude/foo.h -> /new_location/inlcude/foo.h
[Note that I need to name the new lib as libnewfoo and not libfoo due to some constraints. Can this renaming cause any problem? Yet the C code that generates these has not changed.]
It seems to work for the few simple cases I tried. But can there be cases where clients are using the libs and headers in a way which may break as a result of this change. Please let me know what kind of intricacies can be involved in this. Sorry if this seems to be a novice question, I've hardly worked with c before and am a java person.
You have to differentiate between compile time and run time.
For compile time, clients need to update their Makefile and / or configure logic.
For run time, you simply tell ld.so via ld.so.conf about where to find the .so library (or tell your clients to adjust LD_LIBRARY_PATH, a second best choice). The static library does not matter as its code is already built into the executable.
And yes, by providing symbolic links you can make the move 'disappear' as well and provide all files via the old location.
And all this is pretty testable from your end before roll-out.
I don't see any reason why this would break, this is more a question about symlinks than C. To an unsuspecting user program (one which doesn't have special code to detect symlinks and complain), a symlink is transparent.
If you do experience errors feel free to post them and we'll do our best to advise. However I see nothing off the top of my head that would cause issues.
The only problem with the symlinks could be if some clients mount the new location with a different path, which is possible in a networked unix type environment. For example, you could have the location as:
/var/stuff/new_location/include/...
and the client could be mounting that as:
/auto/var/stuff/new_location/include/..
In which case a relative symlink might work better, i.e.:
old_location/include/foo.h -> ../new_location/include/foo.h
Another thing to consider is to replace old_location/foo.h with:
/*
* Please note that this library has moved to a new location...
*/
#include "new_location/include/foo.h"
The symlinks will work on any operating system and file system that supports symlinks.
Shouldn't be hard, right? Right?
I am currently trawling the OpenAFS codebase to find the header definition of pioctl. I've thrown everything I've got at it: checked ctags, grepped the source code for pioctl, etc. The closest I've got to a lead is the fact that there's a file pioctl_nt.h that contains the definition, except it's not actually what I want because none of the userspace code directly includes it, and it's Windows specific.
Now, I'm not expecting you to go and download the OpenAFS codebase and find the header file for me. I am curious, though: what are your techniques for finding the header file you need when everything else fails? What are the worst case scenarios that could cause a grep for pioctl in the codebase to not actually come up with anything that looks like a function definition?
I should also note that I have access to two independent userspace programs that have done it properly, so in theory I could do an O(n) search for the function. But none of the header files pop out to me, and n is large...
Edit: The immediate issue has been resolved: pioctl() is defined implicitly, as shown by this:
AFS.xs:2796: error: implicit declaration of function ‘pioctl’
If grep -r and ctags are failing, then it's probably being defined as the result of some nasty macro(s). You can try making the simplest possible file that calls pioctl() and compiles successfully, and then preprocessing it to see what happens:
gcc -E test.c -o test.i
grep pioctl -C10 test.i
There are compiler options to show the preprocessor output. Try those? In a horrible pinch where my head was completely empty of any possible definition the -E option (in most c compilers) does nothing but spew out the the preprocessed code.
Per requested information: Normally I just capture a compile of the file in question as it is output on the screen do a quick copy and paste and put the -E right after the compiler invocation. The result will spew preprocessor output to the screen so redirect it to a file. Look through that file as all of the macros and silly things are already taken care of.
Worst case scenarios:
K&R style prototypes
Macros are hiding the definition
Implicit Declaration (per your answer)
Have you considered using cscope (available from SourceForge)?
I use it on some fairly significant code sets (25,000+ files, ranging up to about 20,000 lines in a file) with good success. It takes a while to derive the file list (5-10 minutes) and longer (20-30 minutes) to build the cross-reference on an ancient Sun E450, but I find the results useful.
On an almost equally ancient Mac (dual 1GHz PPC 32-bit processors), cscope run on the OpenAFS (1.5.59) source code comes up with quite a lot of places where the function is declared, sometimes inline in code, sometimes in headers. It took a few minutes to scan the 4949 files, generating a 58 MB cscope.out file.
openafs-1.5.59/src/sys/sys_prototypes.h
openafs-1.5.59/src/aklog/aklog_main.c (along with comment "Why doesn't AFS provide these prototypes?")
openafs-1.5.59/src/sys/pioctl_nt.h
openafs-1.5.59/src/auth/ktc.c includes a define for PIOCTL
openafs-1.5.59/src/sys/pioctl_nt.c provides an implementation of it
openafs-1.5.59/src/sys/rmtsysc.c provides an implementation of it (and sometimes afs_pioctl() instead)
The rest of the 184 instances found seem to be uses of the function, or documentation references, or release notes, change logs, and the like.
The current working theory that we've decided on, after poking at the preprocessor and not finding anything either, is that OpenAFS is letting the compiler infer the prototype of the function, since it returns an integer and takes pointer, integer, pointer, integer as its parameters. I'll be dealing with this by merely defining it myself.
Edit: Excellent! I've found the smoking gun:
AFS.xs:2796: error: implicit declaration of function ‘pioctl’
While the original general question has been answered, if anyone arrives at this page wondering where to find a header file that defines pioctl:
In current releases of OpenAFS (1.6.7), a protoype for pioctl is defined in sys_prototypes.h. But that the time that this question was originally asked, that file did not exist, and there was no prototype for pioctl visible from outside the OpenAFS code tree.
However, most users of pioctl probably want, or are at least okay with using, lpioctl ("local" pioctl), which always issues a syscall on the local machine. There is a prototype for this in afssyscalls.h (and these days, also sys_prototypes.h).
The easiest option these days, though, is just to use libkopenafs. For that, include kopenafs.h, use the function k_pioctl, and link against -lkopenafs. That tends to be a much more convenient interface than trying to link with OpenAFS libsys and other stuff.
Doesn't it usually say in the man page synopsis?