I'm trying to learn how to use libraries when writing C code on my mac (not using Xcode). My understanding is that on macs, there is the Library/Frameworks folder where you can put common libraries that can be shared across different projects.
My goal at this point is to use the SDL library to open basic window and do nothing else, but I can't figure out how to utilize libraries on my mac. So to be very specific, I just want to have one file of application code that I have written called main.c and it will have this boilerplate SDL code:
#include "SDL2/SDL.h" // OR #include "SDL.h" (Not sure how the difference in path works)
int SCREEN_HEIGHT = 800;
int SCREEN_WIDTH = 600;
int main() {
SDL_Init(SDL_INIT_VIDEO);
SDL_Window *window = SDL_CreateWindow("SDL Game", 0, 0,
SCREEN_HEIGHT, SCREEN_WIDTH, SDL_WINDOW_HIDDEN);
SDL_ShowWindow(window);
SDL_Event event;
int running = 1;
while(running) {
while(SDL_PollEvent(&event)) {
if(event.type == SDL_QUIT) {
running = 0;
}
}
SDL_Delay( 32 );
}
SDL_DestroyWindow(window);
SDL_Quit();
return 0;
}
I downloaded the development library for mac from the SDL website (https://www.libsdl.org/download-2.0.php) and moved the download to the /Library/Frameworks folder on my machine, just as SDL instructed. However I don't know what to do in my Makefile for the library to be included and linked and then compiled with my main.c file.
Here are some specifics of my laptop/compiler:
MacOS Bug Sur
Version 11.3.1
MacBook Pro (16-inch, 2019)
Processor: 2.3 GHz 8-Core Intel Core i9
Memory: 32 GB 2667 MHz DDR4
Startup Disk: Macintosh HD
Graphics: AMD Radeon Pro 5500M 8 GB
Apple clang version 12.0.5 (clang-1205.0.22.9)
Target: x86_64-apple-darwin20.4.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin
Can someone show me what an example of a very simple Makefile would look like to accomplish this goal??
If you work from the command line or with a properly configured IDE, you link (In GCC) with the -l(library name) flag. The library you want to link needs to be in some directory like /usr/lib, /usr/local/lib or some Libraries folder MacOS might have. If it's not there, then use the -L(path/to/lib/directory) to tell GCC where to find it, and then use -l(library name) flag again.
The library name should start with lib and either end with a .a or .so suffix. So if you wish to link to your own SDL2 library build:
gcc source.c -o mygame -Llibraries/ -lSDL2
Assuming that the library is under libraries/libSDL2.so (Numbers in the end don't matter).
Is it possible to compile a C++ (or the like) program without generating the executable file but writing it and executing it directly from memory?
For example with GCC and clang, something that has a similar effect to:
c++ hello.cpp -o hello.x && ./hello.x $# && rm -f hello.x
In the command line.
But without the burden of writing an executable to disk to immediately load/rerun it.
(If possible, the procedure may not use disk space or at least not space in the current directory which might be read-only).
Possible? Not the way you seem to wish. The task has two parts:
1) How to get the binary into memory
When we specify /dev/stdout as output file in Linux we can then pipe into our program x0 that reads
an executable from stdin and executes it:
gcc -pipe YourFiles1.cpp YourFile2.cpp -o/dev/stdout -Wall | ./x0
In x0 we can just read from stdin until reaching the end of the file:
int main(int argc, const char ** argv)
{
const int stdin = 0;
size_t ntotal = 0;
char * buf = 0;
while(true)
{
/* increasing buffer size dynamically since we do not know how many bytes to read */
buf = (char*)realloc(buf, ntotal+4096*sizeof(char));
int nread = read(stdin, buf+ntotal, 4096);
if (nread<0) break;
ntotal += nread;
}
memexec(buf, ntotal, argv);
}
It would also be possible for x0 directly execute the compiler and read the output. This question has been answered here: Redirecting exec output to a buffer or file
Caveat: I just figured out that for some strange reason this does not work when I use pipe | but works when I use the x0 < foo.
Note: If you are willing to modify your compiler or you do JIT like LLVM, clang and other frameworks you could directly generate executable code. However for the rest of this discussion I assume you want to use an existing compiler.
Note: Execution via temporary file
Other programs such as UPX achieve a similar behavior by executing a temporary file, this is easier and more portable than the approach outlined below. On systems where /tmp is mapped to a RAM disk for example typical servers, the temporary file will be memory based anyway.
#include<cstring> // size_t
#include <fcntl.h>
#include <stdio.h> // perror
#include <stdlib.h> // mkostemp
#include <sys/stat.h> // O_WRONLY
#include <unistd.h> // read
int memexec(void * exe, size_t exe_size, const char * argv)
{
/* random temporary file name in /tmp */
char name[15] = "/tmp/fooXXXXXX";
/* creates temporary file, returns writeable file descriptor */
int fd_wr = mkostemp(name, O_WRONLY);
/* makes file executable and readonly */
chmod(name, S_IRUSR | S_IXUSR);
/* creates read-only file descriptor before deleting the file */
int fd_ro = open(name, O_RDONLY);
/* removes file from file system, kernel buffers content in memory until all fd closed */
unlink(name);
/* writes executable to file */
write(fd_wr, exe, exe_size);
/* fexecve will not work as long as there in a open writeable file descriptor */
close(fd_wr);
char *const newenviron[] = { NULL };
/* -fpermissive */
fexecve(fd_ro, argv, newenviron);
perror("failed");
}
Caveat: Error handling is left out for clarities sake. Includes for sake of brevity.
Note: By combining step main() and memexec() into a single function and using splice(2) for copying directly between stdin and fd_wr the program could be significantly optimized.
2) Execution directly from memory
One does not simply load and execute an ELF binary from memory. Some preparation, mostly related to dynamic linking, has to happen. There is a lot of material explaining the various steps of the ELF linking process and studying it makes me believe that theoretically possible. See for example this closely related question on SO however there seems not to exist a working solution.
Update UserModeExec seems to come very close.
Writing a working implementation would be very time consuming, and surely raise some interesting questions in its own right. I like to believe this is by design: for most applications it is strongly undesirable to (accidentially) execute its input data because it allows code injection.
What happens exactly when an ELF is executed? Normally the kernel receives a file name and then creates a process, loads and maps the different sections of the executable into memory, performs a lot of sanity checks and marks it as executable before passing control and a file name back to the run-time linker ld-linux.so (part of libc). The takes care of relocating functions, handling additional libraries, setting up global objects and jumping to the executables entry point. AIU this heavy lifting is done by dl_main() (implemented in libc/elf/rtld.c).
Even fexecve is implemented using a file in /proc and it is this need for a file name that leads us to reimplement parts of this linking process.
Libraries
UserModeExec
libelf -- read, modify, create ELF files
eresi -- play with elfes
OSKit (seems like a dead project though)
Reading
http://www.linuxjournal.com/article/1060?page=0,0 -- introduction
http://wiki.osdev.org/ELF -- good overview
http://s.eresi-project.org/inc/articles/elf-rtld.txt -- more detailed Linux-specific explanation
http://www.codeproject.com/Articles/33340/Code-Injection-into-Running-Linux-Application -- how to get to hello world
http://www.acsu.buffalo.edu/~charngda/elf.html -- nice reference of ELF structure
Loaders and Linkers by John Levine -- deeoer explanation of linking
Related Questions at SO
Linux user-space ELF loader
ELF Dynamic loader symbol lookup ordering
load-time ELF relocation
How do global variables get initialized by the elf loader
So it seems possible, you decide whether is also practical.
Yes, though doing it properly requires designing significant parts of the compiler with this in mind. The LLVM guys have done this, first with a kinda-separate JIT, and later with the MC subproject. I don't think there's a ready-made tool doing it. But in principle, it's just a matter of linking to clang and llvm, passing the source to clang, and passing the IR it creates to MCJIT. Maybe a demo does this (I vaguely recall a basic C interpreter that worked like this, though I think it was based on the legacy JIT).
Edit: Found the demo I recalled. Also, there's cling, which seems to do basically what I described, but better.
Linux can create virtual file systems in RAM using tempfs. For example, I have my tmp directory set up in my file system table like so:
tmpfs /tmp tmpfs nodev,nosuid 0 0
Using this, any files I put in /tmp are stored in my RAM.
Windows doesn't seem to have any "official" way of doing this, but has many third-party options.
Without this "RAM disk" concept, you would likely have to heavily modify a compiler and linker to operate completely in memory.
If you are not specifically tied to C++, you may also consider other JIT based solutions:
in Common Lisp SBCL is able to generate machine code on the fly
you could use TinyCC and its libtcc.a which emits quickly poor (i.e. unoptimized) machine code from C code in memory.
consider also any JITing library, e.g. libjit, GNU Lightning, LLVM, GCCJIT, asmjit
of course emitting C++ code on some tmpfs and compiling it...
But if you want good machine code, you'll need it to be optimized, and that is not fast (so the time to write to a filesystem is negligible).
If you are tied to C++ generated code, you need a good C++ optimizing compiler (e.g. g++ or clang++); they take significant time to compile C++ code to optimized binary, so you should generate to some file foo.cc (perhaps in a RAM file system like some tmpfs, but that would give a minor gain, since most of the time is spent inside g++ or clang++ optimization passes, not reading from disk), then compile that foo.cc to foo.so (using perhaps make, or at least forking g++ -Wall -shared -O2 foo.cc -o foo.so, perhaps with additional libraries). At last have your main program dlopen that generated foo.so. FWIW, MELT was doing exactly that, and on Linux workstation the manydl.c program shows that a process can generate then dlopen(3) many hundred thousands of temporary plugins, each one being obtained by generating a temporary C file and compiling it. For C++ read the C++ dlopen mini HOWTO.
Alternatively, generate a self-contained source program foobar.cc, compile it to an executable foobarbin e.g. with g++ -O2 foobar.cc -o foobarbin and execute with execve that foobarbin executable binary
When generating C++ code, you may want to avoid generating tiny C++ source files (e.g. a dozen lines only; if possible, generate C++ files of a few hundred lines at least; unless lots of template expansion happens thru extensive use of existing C++ containers, where generating a small C++ function combining them makes sense). For instance, try if possible to put several generated C++ functions in the same generated C++ file (but avoid having very big generated C++ functions, e.g. 10KLOC in a single function; they take a lot of time to be compiled by GCC). You could consider, if relevant, to have only one single #include in that generated C++ file, and pre-compile that commonly included header.
Jacques Pitrat's book Artificial Beings, the conscience of a conscious machine (ISBN 9781848211018) explains in details why generating code at runtime is useful (in symbolic artificial intelligence systems like his CAIA system). The RefPerSys project is trying to follow that idea and generate some C++ code (and hopefully, more and more of it) at runtime. Partial evaluation is a relevant concept.
Your software is likely to spend more CPU time in generating C++ code than GCC in compiling it.
tcc compiler "-run" option allows for exactly this, compile into memory, run there and finally discard the compiled stuff. No filesystem space needed. "tcc -run" can be used in shebang to allow for C script, from tcc man page:
#!/usr/local/bin/tcc -run
#include <stdio.h>
int main()
{
printf("Hello World\n");
return 0;
}
C scripts allow for mixed bash/C scripts, with "tcc -run" not needing any temporary space:
#!/bin/bash
echo "foo"
sed -n "/^\/\*\*$/,\$p" $0 | tcc -run -
exit
/**
*/
#include <stdio.h>
int main()
{
printf("bar\n");
return 0;
}
Execution output:
$ ./shtcc2
foo
bar
$
C scripts with gcc are possible as well, but need temporary space like others mentioned to store executable. This script produces same output as the previous one:
#!/bin/bash
exc=/tmp/`basename $0`
if [ $0 -nt $exc ]; then sed -n "/^\/\*\*$/,\$p" $0 | gcc -x c - -o $exc; fi
echo "foo"
$exc
exit
/**
*/
#include <stdio.h>
int main()
{
printf("bar\n");
return 0;
}
C scripts with suffix ".c" are nice, headtail.c was my first ".c" file that needed to be executable:
$ echo -e "1\n2\n3\n4\n5\n6\n7" | ./headtail.c
1
2
3
6
7
$
I like C scripts, because you just have one file, you can easily move around, and changes in bash or C part require no further action, they just work on next execution.
P.S:
The above shown "tcc -run" C script has a problem, C script stdin is not available for executed C code. Reason was that I passed extracted C code via pipe to "tcc -run". New gist run_from_memory_stdin.c does it correctly:
...
echo "foo"
tcc -run <(sed -n "/^\/\*\*$/,\$p" $0) 42
...
"foo" is printed by bash part, "bar 42" from C part (42 is passed argv[1]), and piped script input gets printed from C code then:
$ route -n | ./run_from_memory_stdin.c
foo
bar 42
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 172.29.58.98 0.0.0.0 UG 306 0 0 wlan1
10.0.0.0 0.0.0.0 255.255.255.0 U 0 0 0 wlan0
169.254.0.0 0.0.0.0 255.255.0.0 U 303 0 0 wlan0
172.29.58.96 0.0.0.0 255.255.255.252 U 306 0 0 wlan1
$
One can easily modify the compiler itself. It sounds hard first but thinking about it, it seams obvious. So modifying the compiler sources directly expose a library and make it a shared library should not take that much of afford (depending on the actual implementation).
Just replace every file access with a solution of a memory mapped file.
It is something I am about to do with compiling something transparently in the background to op codes and execute those from within Java.
-
But thinking about your original question it seams you want to speed up compilation and your edit and run cycle. First of all get a SSD-Disk you get almost memory speed (use a PCI version) and lets say its C we are talking about. C does this linking step resulting in very complex operations that are likely to take more time than reading and writing from / to disk. So just put everything on SSD and live with the lag.
Finally the answer to OP question is yes!
I found memrun repo from guitmz, that demoed running (x86_64) ELF from memory, with golang and assembler. I forked that, and provided C version of memrun, that runs ELF binaries (verified on x86_64 and armv7l), either from standard input, or via first argument process substitution. The repo contains demos and documentation (memrun.c is 47 lines of code only):
https://github.com/Hermann-SW/memrun/tree/master/C#memrun
Here is simplest example, with "-o /dev/fd/1" gcc compiled ELF gets sent to stdout, and piped to memrun, which executes it:
pi#raspberrypi400:~/memrun/C $ gcc info.c -o /dev/fd/1 | ./memrun
My process ID : 20043
argv[0] : ./memrun
no argv[1]
evecve --> /usr/bin/ls -l /proc/20043/fd
total 0
lr-x------ 1 pi pi 64 Sep 18 22:27 0 -> 'pipe:[1601148]'
lrwx------ 1 pi pi 64 Sep 18 22:27 1 -> /dev/pts/4
lrwx------ 1 pi pi 64 Sep 18 22:27 2 -> /dev/pts/4
lr-x------ 1 pi pi 64 Sep 18 22:27 3 -> /proc/20043/fd
pi#raspberrypi400:~/memrun/C $
The reason I was interested in this topic was usage in "C script"s. run_from_memory_stdin.c demonstrates all together:
pi#raspberrypi400:~/memrun/C $ wc memrun.c | ./run_from_memory_stdin.c
foo
bar 42
47 141 1005 memrun.c
pi#raspberrypi400:~/memrun/C $
The C script producing shown output is so small ...
#!/bin/bash
echo "foo"
./memrun <(gcc -o /dev/fd/1 -x c <(sed -n "/^\/\*\*$/,\$p" $0)) 42
exit
/**
*/
#include <stdio.h>
int main(int argc, char *argv[])
{
printf("bar %s\n", argc>1 ? argv[1] : "(undef)");
for(int c=getchar(); EOF!=c; c=getchar()) { putchar(c); }
return 0;
}
P.S:
I added tcc's "-run" option to gcc and g++, for details see:
https://github.com/Hermann-SW/memrun/tree/master/C#adding-tcc--run-option-to-gcc-and-g
Just nice, and nothing gets stored in filesystem:
pi#raspberrypi400:~/memrun/C $ uname -a | g++ -O3 -Wall -run demo.cpp 42
bar 42
Linux raspberrypi400 5.10.60-v7l+ #1449 SMP Wed Aug 25 15:00:44 BST 2021 armv7l GNU/Linux
pi#raspberrypi400:~/memrun/C $
I have a Dreambox 500 which on Wikipedia says has a PCP processor which is PowerPC:
$ cat /proc/cpuinfo
processor: 0
cpu: STBx25xx
clock: 252MHz
Review: 9.80 (pvr 5151 0950)
bogomips: 250.36
Machine: Dream Multimedia Dreambox TV
plb bus clock: 63MHz
I would normally install GCC but it has low storage on it and I need to compile a program for it.
I've heard GCC can compile powerpc but I had no luck doing so.
Example this code
#include <stdio.h>
int main()
{
printf("Hello World!\n");
return 0;
}
And I use this to compile
gcc example.c -mtune=powerpc
But it give this error
example.c:1:0 error: bad value (powerpc) for -mtune- switch
#include <stdio.h>
^
Thank you!
You should use cross-compiler, because your target architecture differs from host one. Host is the architecture of your system (usually amd64 (x86_64) or i386 (x86_32)). And target arch is the arch on which your compiled program will run (powerpc in your case).
Many GNU/Linux distors provide crosscompilers as a separate packages. For example, for Ubuntu these packages are available:
sudo apt-get install gcc-4.8-powerpc-linux-gnu g++-4.8-powerpc-linux-gnu binutils-4.8-powerpc-linux-gnu
Packages above are for trusty. In later releases different GCC versions are available.
Then you can compile your program using powerpc-linux-gnu-gcc-4.8. Or you can set your environment variables CC and CXX to powerpc-linux-gnu-gcc-4.8 and powerpc-linux-gnu-g++-4.8 accordingly.
upd:
I found crosscompiler toolchain for Dreambox 500 here, but it contains relatively old GCC (3.4).
In order to use it extract downloaded file to /opt/cross/dm500, add /opt/cross/dm500/cdk/bin to path via export PATH=$PATH:/opt/cross/dm500/cdk/bin and use gcc from here with appropriate prefix.
After being on a programming forum for a while, found a guy with the same problem, and after a while he found a way to fix it and I tried it and it works.
The thing I have to do is
powerpc-gcc someprog.c -static
I have no idea what the -static does but it increases the executable file size and at the end it works!
Not so long ago I've installed Debian and configured it with my friend's help.
Yesterday I have downloaded GCC 4.4 and I created a simple program to test it out.
This is the code:
#include <stdio.h>
int main () {
int result;
printf ("Hello Wor... Linux! This is my %dst program compiled in Debian.\nHow many is 2+2?\n", 1);
scanf ("%d", &result);
while (result!=4) {
printf ("Oh no! You're not going anywhere until you type the correct result! 2+2 is?\n");
scanf ("%d", &result);
}
printf ("Congrats!\n");
return 0;
}
I've compiled it by typing gcc-4.4 myfile.c in bash. Then I've tried to run the resulting binary file and it worked just as I wanted it to. Then I've sent the binary file to my friend to test it on his PC also. When he tried to run it, he received a segmentation fault message and the program didn't work.
He also uses Debian and his kernel's version is very similar to mine (2.6.32-5-686). The only difference is that his kernel is an amd-64 one (he owns a 64-bit processor, while mine is 32-bit).
Why is this happening? Does it mean that 64-bit Linux users will be unable to run my 32-bit programs? If so, can I compile it in a way which will let them to run it?
Please note that I'm not really experienced with Linux.
he may need a chroot for it.
apt-get install ia32-libs
should work for most cases.
see "Using an IA32 chroot to run 32bit applications" http://alioth.debian.org/docman/view.php/30192/21/debian-amd64-howto.html#id292205
Alternatively, set up your compiler to target 64-bit binaries by following the instructions at the OSDev wiki: In brief:
Set up the new repos in /etc/apt/sources.list
deb http://www.tucs.org.au/~jscott4/debian/ stable main #Primary Mirror. Hosted by University of Tasmania.
Add the signing key:
gpg --recv-keys 0x2F90DE4A
gpg -a --export 0x2F90DE4A | sudo apt-key add -
Update your repo indices and get the appropriate cross-compilation package:
apt-get update
apt-get install osdev-crosscompiler-x86-64-elf
Then use the x86_64-elf variant of gcc to target x64. For instance
x86_64-elf-gcc --pedantic -Wall -o foo foo.c
(In fact all the GCC tools and Binutils will have an x86_64-elf- variant now.)
EDIT -- Vastly improved instructions by pulling from a reference instead of from memory.
EDIT -- removed stale mirror
chroot is one option. But remember it requires a lot of disk space as it installs 32-bit libraries.
Alternatively you can compile your file for a 64-bit environment by using the -m64 compiler flag of gcc which sets int to 32 bits and long and pointer to 64 bits and generates code for AMD's x86-64 architecture.
I can't seem to get zlib to do anything on mingw under windows.
I downloaded zlib # http://sourceforge.net/projects/mingw/files_beta/MinGW/zlib/zlib-1.2.3-1-mingw32/ and put the header and lib files in the right place.
Simple code like:
#include <stdlib.h>
#include <stdio.h>
#include "zlib.h"
int main(int argc, char *argv[])
{
long a;
char buffer[1024];
a = 1024;
compress(buffer,&a,"testing",7);
return 0;
}
compiled:
gcc test.c -lzlib -Wall -o test.exe
Compiles fine.
However the exe crashes at the compress function.
Any ideas?
I recommend using MSYS2 for this kind of thing. These instructions assume you want to compile a 64-bit program, but they can easily be modified for 32-bit.
After installing MSYS2, run the "MinGW-w64 Win64 Shell" shortcut in your Start Menu. Install the 64-bit toolchain by running:
pacman -S mingw-w64-x86_64-toolchain mingw-w64-x86_64-zlib
Then compile your code by running something like this:
gcc test.c -lz -o test
I did not check your code carefully, but I was able to run your code without any crashing, so your code might be OK. Your code also gives no output so it's hard to tell if it really worked.
Looking at the zlib manual it says:
ZEXTERN int ZEXPORT compress OF((Bytef *dest, uLongf *destLen,
const Bytef *source, uLong sourceLen));
Compresses the source buffer into the
destination buffer. sourceLen is the
byte length of the source buffer. Upon
entry, destLen is the total size of
the destination buffer, which must be
at least the value returned by
compressBound(sourceLen). Upon exit,
destLen is the actual size of the
compressed buffer.
Maybe a=1024 isn't big enough? I think you need to call compressBound to get a suitable value.
I tried to use the zlib from MSYS (accessible with mingw-get) and got the same problem as described below.
The solution is to do a static link instead of using the shared library.
Just remove or rename the import library libz.dll.a to avoid the linker to do a link with the msys-z.dll.
Recompile and it will be working.
Another way is to install zlib yourself from the zlib.net website.
Remove the one from mingw-get.
Using zlib in your code is extremely simple, something that the documentation ( or the various answers on stackoverflow I found ) don't make obvious.
The following technique works for any compiler and IDE. I tested it in windows mingw using code:blocks, which is why I am posting it as an answer to this question.
Download the zlib source code from http://www.zlib.net/
Copy all the .c and .h files from the root folder of the zlib source to a folder in your compiler search path.
Add the zlib source files to the IDE project.
Add #include "zlib.h" to your source code
Call compress or uncompress
That's it. It could hardly be simpler.
All you have to be careful about is memory management, since this is c code.
To make things even simpler for myself, I have put together a c++ wrapper which you are welcome to use, like this:
/** ZLIB C++ wrapper
Usage:
<pre>
#include "cZLIB.h"
{
// compress data in bigbuffer
raven::set::cZLIB ZLIB;
ZLIB.Compress( bigbuffer, sizebigbuffer );
// use compressed buffer, before ZLIB goes out of scope
use( ZLIB.Buffer(), ZLIB.Length() );
}
...
{
// decompress data in smallbuffer
raven::set::cZLIB ZLIB;
ZLIB.Inflate( smallbuffer, sizesmallbuffer )
// use decompressed data, before ZLIB goes out of scope
use( ZLIB.Buffer(), ZLIB.Length() );
}
</pre>
Build:
Download this code ( cZLIB.h and cZLIB.cpp ) from
https://github.com/JamesBremner/raven-set
and install somewhere in your compiler search path.
Let's assume you install it in folder .../src.
Download the zlib source code from http://www.zlib.net/
Copy all the .c and .h files from the root folder of the zlib source
to a new folder .../src/zlib
Add the files cZLIB.h, cZLIB.cpp and all the files in .../src/zlib
to the IDE project.
Build.
*/
class cZLIB
...