I have an application, written in C, which runs on multiple platforms: Windows and various flavors of Unix. The two most important are Linux, then Windows.
One of the core algorithms could benefit from CUDA acceleration. However, everything must run normally if the system isn't CUDA-capable, (or the user doesn't specifically ask for GPGPU acceleration).
So, I need to make sure the application tries to load the CUDA libraries only under the right circumstances.
On Windows, the delay-load mechanism makes this fairly easy.
Is there a similarly easy mechanism to do this on Linux? Or do I have to go through the contortions involved with dlopen()?
You have two choices, both perfectly valid:
As you note in your question, use dlopen to load dynamic libraries at runtime
Statically link the CUDA toolkit libraries into your application. NVIDIA have been shipping static versions of the toolkit libraries for linux for several major release cycles, and static linking is fully supported. The downside of this approach is application size.
Related
I've built a programme in C with Visual Studio using standard C libraries + a couple of windows libraries.
The code just acquires user input (with scanf, so through cmd window) does some calculations based on the input and outputs some text files, that's about it.
I'm wondering what would I then need to do to run my exe on another standard Windows computer without it needing to install any additional files e.g. the whole Windows SDK.?
Is it just a case of being able to build the release version (as opposed to the debug)?
Many thanks
If you pick the right CPU target (Project Configuration Properties -> C/C++: Enable Enhanced Instruction Set) such that the binary code doesn't include instructions understood by only a very narrow subset of CPUs, and the default for Visual Studio is to use instructions that are supported by the widest set of x86 or x64 CPUs, then your program will run on almost any Windows computer. You need to distribute any DLLs that aren't part of the base Windows installation though, which includes any additional dynamically linked language runtimes such as the Visual C++ runtime.
A good way to derive the list of DLLs that you have to package with your executable is to create a virtual machine with a fresh Windows installation without any development tools in it and then try to run your code there. If you get an error for a missing DLL, add it and repeat. When it comes to the Visual C++ runtime, Microsoft provides installable packages for the different versions that you are allowed to distribute as part of your installation (Visual C++ Whatever Redistributable Package).
Also, mind that programs compiled for 32-bit Windows will mostly run on 64-bit versions, but the opposite is not true.
Generally speaking, C programs are not "portable" meaning they can't be copied over to other machines and expected to run.
There are a few exceptional cases where C executables can safely be ran on other machines:
The CPUs support the same instruction set, with the same set of compatible side-effects and possibly the same bugs.
The Operating systems support the same system api points, with the same set of compatible side-effects and possibly the same bugs.
The installed non-operating system libraries support the same api points, with the same set of compatible side-effects and possibly the same bugs.
The API calling conventions are the same between the source (platform you built the code on) and destination (platform you will run the executable on).
Of course, not all of the CPU and OS need to be 100% compatible, just the parts your compiled program uses (which is not always easy to see as it is compiled, and not a 100% identical representation of the source code)
For these conditions to hold, typically you are using the same release of the operating system, or a compatibility interface designed by the operating system packagers that supports the current version of the operating system and older versions too.
The details on how this is most easily done differ between operating systems, and generally speaking, even if a compatibility layer is present, you need to do adequate testing as the side-effects and bugs tend to differ despite promises of multi-operating system compatibility.
Finally, there are some environments that can make non-same CPU executables run on an operating system (like QEmu) by simulating the foreign CPU instruction set at runtime, interperting those instructions into ones that are compatible with the current CPU. Such systems are not common across non-Linux operating systems; and, they may stumble if the loading of dynamic libraries can't locate and load foreign instruction set libraries.
With all of these caveats, you can see why most people decide to write portable source code and recompile it on each of the target platforms; or, write their code for an interpreter that already exists on multiple platforms. With the interpreter approach, the CPU is conceptually a virtual one, which is implemented to be identical across hardware, letting you write one set of source code to be interpreted across all the hardware platforms.
I've built a programme in C with Visual Studio using standard C libraries + a couple of windows libraries.
I'm wondering what would I then need to do to run my exe on another standard Windows computer without it needing to install any additional files e.g. the whole Windows SDK.?
You don't explain what your program is really doing. Does it have a graphical user interface? Is it a web server? Do you have some time to improve it or enhance it? Can it run on the command line?
Why cannot you share the C source code (e.g. using github)?
If you want some graphical interface, consider using GTK. It has been ported to Windows.
If a web interface is enough, consider using libonion in your program, or find some HTTP server library in C for your particular platform.
But what you need understand is mostly related to linking, not only to the C programming language (read the n1570 specification). Hence read about Linkers and loaders.
You should prefer static libraries. Their availability is platform specific. Read more about Operating Systems. Also, sharing some Microsoft Windows system WinAPI libraries could be illegal (even when technically possible), so consult your lawyer after showing him the EULA that you are bound to.
My opinion is that Linux distributions are very friendly when learning to program in C (e.g. using GCC or Clang as your C compiler). Both Ubuntu and Debian are freely downloadable, but practically require about a hundred gigabytes of consecutive free disk space. Be sure to backup your important data before installing one of them.
Did you consider porting, adapting and compiling your C code to WebAssembly? If you did that, your code would run inside most recent Web browsers. Look also into Bellard's JSLinux.
Related answer here. Notice that C can be transpiled to JavaScript and then run locally inside recent web browsers.
I would like to build an OS some time in the future, and now thinking of some light sketches on how it would be. I have pretty much been coding in C compiled for the Windows environment (and some little Java). I would have to recompile any of my C programs should I want to run it under Linux. So the binaries, the product of compilation, must be different for each operating system. If I design a totally new OS from scratch, for both hobby and academic purpose, without using the Linux kernel or any known base code of an OS, what I understand as to happen is that I cannot compile my C programs with GCC since my OS will not be among its target systems. Here my question written on the title emerges. Thanks in advance for any hints.
It depends. You could easily choose to re-use an existing compiler, such as the immemorial example GCC, and thus you would reap the benefits of an existing compiler. But there are some big provisos that must be cleared up.
Regards of whether or not you choose to build a new compiler, the challenge will remain in porting a C library. You technically can use C without a standard library (which is what the Linux kernel, or any self-hosted example for that matter, has to do, for example) but this is a ridiculous proposition for programs intended to run under an operating system, as most systems impose memory restrictions, etc, meaning that you cannot just have carte blanche in terms of using memory. Thusly, a C library call such as malloc is required.
Since any programs under your kernel (99% of your OS in all likelihood) will need a set of functions to link against, porting a C library is your biggest task. The C library is a huge monolith, and writing your own would be rather silly, especially with many implementations already available, the most well known being GCC's. So, the question you really should be asking is, do you want to write my own version of libc? (The answer is almost always no, and most alternative implementations are for niche use cases.) Plus, if you want to make your OS POSIX-compliant, then you'll have to implement more functions, adding to the hassle.
Whether you write your own compiler for your OS is a minor detail compared to which C library will be included with it. You can always use your own compiler with an already-written implementation of the C library.
My advice to your rather opinion-based question: no. Port an existing compiler such as GCC or clang, and then use that. Plus, that has several advantages:
Compatibility with existing tools and toolchains
A familiar program (no need for your users to learn how to use a new compiler)
They're open source - and in spite of that, you'd be insane to go at it alone. Heck, even Apple integrated two already existing compilers - GCC and clang - into their toolchains rather than do it themselves, and they're a billion-dollar company.
Take a look at this page. It demonstrates how to port GCC to your OS using Newlib as your C library.
No, you can just port an existing compiler. You can even choose an existing executable format, such as ELF, and use your standard GCC + GNU Binutils toolchain. You will need to port the standard library and C runtime, and you will need to write an ELF loader into your operating system.
I suspect the majority of the work will be in porting the C library.
A search turned up this page: Porting GCC to your OS
(1) No, you usually don't have to write your own compiler. Writing a good optimizing compiler can be actually big task which I would better avoid.
But in order to enable writing applications for your OS in some higher level language you will either need to provide
some (2.1) API emulation layer (so that code written and compiled for other OS can be run on your OS)
or you'll have to (2.2) port some existing compiler to your OS
or at least make your OS a new available (2.3) target platform in an existing compiler
or some other option I don't know about
The choices are multiple each with its own pros/cons.
Some examples (other then the obvious GCC already mentioned by #dietrich-epp, #sevenbits) to help you decide which way you want to follow:
(3.1) Free Pascal (see http://www.freepascal.org) compiler can be extended with another target platform
Free Pascal is a 32,64 and 16 bit professional Pascal compiler. It can target multiple processor architectures: Intel x86, AMD64/x86-64, PowerPC, PowerPC64, SPARC, and ARM. Supported operating systems include Linux, FreeBSD, Haiku, Mac OS X/iOS/Darwin, DOS, Win32, Win64, WinCE, OS/2, MorphOS, Nintendo GBA, Nintendo DS, and Nintendo Wii. Additionally, JVM, MIPS (big and little endian variants), i8086 and Motorola 68k architecture targets are available in the development versions
...
Source: http://www.freepascal.org
(3.2) Inferno Operating System (see http://www.vitanuova.com/inferno) has its own application language (see Limbo) with OS specific words, own compiler etc. Applications run in virtual machine (see Dis)
Inferno® is a compact operating system designed for building distributed and networked systems on a wide variety of devices and platforms. With many advanced and unique features, Inferno puts an unrivalled set of tools into your hands...Inferno can run as a user application on top of an existing operating system or as a stand alone operating system...
Source: http://www.vitanuova.com/inferno
(3.3) Squeak (see http://en.wikipedia.org/wiki/Squeak) is a self contained OS with graphics and everything. It uses Smalltalk-80 as the language. Compiler included, applications run in virtual machine (see Cog VM). The VM could be emitted as portable C code and then ported to a bare-bone hardware.
Squeak is a modern, open source, full-featured implementation of the powerful Smalltalk programming language and environment. Squeak is highly-portable, running on almost any platform you could name and you can really truly write once run anywhere. Squeak is the vehicle for a wide range of projects from multimedia applications and educational platforms to commercial web application development...
Source: http://www.squeak.org
(3.4) MenuetOS (see http://www.menuetos.net/) is 64bit OS written in assembly language. Flat Assembler (see FASM) compiler which can emit native binaries was ported to the OS including OS API and is included in basic installation. Later on C library was also ported
MenuetOS is an Operating System in development for the PC written entirely in 32/64 bit assembly language...supports 32/64 bit x86 assembly programming for smaller, faster and less resource hungry applications...Menuet isn't based on other operating system nor has it roots within UNIX or the POSIX standards. The design goal, since the first release in year 2000, has been to remove the extra layers between different parts of an OS, which normally complicate programming and create bugs...
Source: http://www.menuetos.net
(3.5) Google's Android OS (see Wikipedia: Android (operating system)) ported Java Virtual Machine (see Dalvik later replaced by Android Runtime) and provided OS APIs for the Java programming language, reusing existing compilers and IDEs just consuming the produced binaries
Android Runtime (ART) is an application runtime environment used by the Android mobile operating system. ART replaces Dalvik, which is the process virtual machine originally used by Android, and performs transformation of the application's bytecode into native instructions that are later executed by the device's runtime environment...
Source: http://en.wikipedia.org/wiki/Android_Runtime
There are many more useful examples available. Whether you have to or don't have to basically depends on the programming paradigm your new OS will introduce. Why you want to build it and how will it differ from the existing ones.
Examples for no are: (3.1), (3.4), (3.5)
Examples for yes are: (3.2), (3.3)
If I compile a self-contained project with gcc and statically link the C run-time library, will this work on any Linux distro? I don't care about any static link disadvantages (such as big file size) so long as it works. It is a closed-source project and this is beyond my control. The architecture of the PCs is the same.
If you can guarantee that all of the code it will execute is self-contained in the binary you ship, then theoretically (emphasis on theoretically) it should work on any linux distribution. In this process there are an enormous number of pitfalls. My personal opinion is that it is pretty much impossible to make this work due to changing interfaces across versions. Interfacing with other libraries is a fragile nightmare.
Most companies that I'm familiar with (including my own) produce builds made for different distributions. There are some complications that result from having to make a build for SLES, Redhat, etc., but I'm confident that providing a few different builds ultimately is simpler and causes less problems than attempting to statically link everything.
Is it possible to write code in C, then statically build it and make a binary out of it like an ELF/PE then remove its header and all unnecessary meta-data so to create a raw binary and at last be able to put this raw binary in any other kind of OS specific like (ELF > PE) or (PE > ELF)?!
have you done this before?
is it possible?
what are issues and concerns?
how this would be possible?!
and if not, just tell me why not?!!?!
what are my pitfalls in understanding the static build?
doesn't it mean that it removes any need for 3rd party and standard as well as os libs and headers?!
Why cant we remove the meta of for example ELF and put meta and other specs needed for PE?
Mention:
I said, Cross OS not Cross Hardware
[Read after reading below!]
As you see the best answer, till now (!) just keep going and learn cross platform development issues!!! How crazy is this?! thanks to philosophy!!!
I would say that it's possible, but this process must be crippled by many, many details.
ABI compatibility
The first thing to think of is Application Binary Interface compatibility. Unless you're able to call your functions the same way, the code is broken. So I guess (though I can't check at the moment) that compiling code with gcc on Linux/OS X and MinGW gcc on Windows should give the same binary code as far as no external functions are called. The problem here is that executable metadata may rely on some ABI assumptions.
Standard libraries
That seems to be the largest hurdle. Partly because of C preprocessor that can inline some procedures on some platforms, leaving them to run-time on others. Also, cross-platform dynamic interoperation with standard libraries is close to impossible, though theoretically one can imagine a code that uses a limited subset of the C standard library that is exposed through the same ABI on different platforms.
Static build mostly eliminates problems of interaction with other user-space code, but still there is a huge issue of interfacing with kernel: it's int $0x80 calls on x86 Linux and a platform-specifc set of syscall numbers that does not map to Windows in any direct way.
OS-specific register use
As far as I know, Windows uses register %fs for storing some OS-wide exception-handling stuff, so a binary compiled on Linux should avoid cluttering it. There might be other similar issues. Also, C++ exceptions on Windows are mostly done with OS exceptions.
Virtual addresses
Again, AFAIK Windows DLLs have some predefined address they're must be loaded into in virtual address space of a process, whereas Linux uses position-independent code for shared libraries. So there might be issues with overlapping areas of an executable and ported code, unless the ported position-dependent code is recompiled to be position-independent.
So, while theoretically possible, such transformation must be very fragile in real situations and it's impossible to re-plant the whole static build code - some parts may be transferred intact, but must be relinked to system-specific code interfacing with other kernel properly.
P.S. I think Wine is a good example of running binary code on a quite different system. It tricks a Windows program to think it's running in Windows environment and uses the same machine code - most of the time that works well (if a program does not use private system low-level routines or unavailable libraries).
What things should be kept most in mind when writing cross-platform applications in C? Targeted platforms: 32-bit Intel based PC, Mac, and Linux. I'm especially looking for the type of versatility that Jungle Disk has in their USB desktop edition ( http://www.jungledisk.com/desktop/download.aspx )
What are tips and "gotchas" for this type of development?
I maintained for a number of years an ANSI C networking library that was ported to close to 30 different OS's and compilers. The library didn't have any GUI components, which made it easier. We ended up abstracting out into dedicated source files any routine that was not consistent across platforms, and used #defines where appropriate in those source files. This kept the code that was adjusted per platform isolated away from the main business logic of the library. We also made extensive use of typedefs and our own dedicated types so that we could easily change them per platform if needed. This made the port to 64-bit platforms fairly easy.
If you are looking to have GUI components, I would suggest looking at GUI toolkits such as WxWindows or Qt (which are both C++ libraries).
Try to avoid platform-dependent #ifdefs, as they tend to grow exponentially when you add new platforms. Instead, try to organize your source files as a tree with platform-independent code at the root, and platform-dependent code on the "leaves". There is a nice book on the subject, Multi-Platform Code Management. Sample code in it may look obsolete, but ideas described in the book are still brilliantly vital.
Further to Kyle's answer, I would strongly recommend against trying to use the Posix subsystem in Windows. It's implemented to an absolute bare minimum level such that Microsoft can claim "Posix support" on a feature sheet tick box. Perhaps somebody out there actually uses it, but I've never encountered it in real life.
One can certainly write cross-platform C code, you just have to be aware of the differences between platforms, and test, test, test. Unit tests and a CI (continuous integration) solution will go a long way toward making sure your program works across all your target platforms.
A good approach is to isolate the system-dependent stuff in one or a few modules at most. Provide a system-independent interface from that module. Then build everything else on top of that module, so it doesn't depend on the system you're compiling for.
XVT have a cross platform GUI C API which is mature 15+ years and sits on top of the native windowing toollkits. See WWW.XVT.COM.
They support at least LINUX, Windows, and MAC.
Try to write as much as you can with POSIX. Mac and Linux support POSIX natively and Windows has a system that can run it (as far as I know - I've never actually used it). If your app is graphical, both Mac and Linux support X11 libraries (Linux natively, Mac through X11.app) and there are numerous ways of getting X11 apps to run on Windows.
However, if you're looking for true multi-platform deployment, you should probably switch to a language like Java or Python that's capable of running the same program on multiple systems with little or no change.
Edit: I just downloaded the application and looked at the files. It does appear to have binaries for all 3 platforms in one directory. If your concern is in how to write apps that can be moved from machine to machine without losing settings, you should probably write all your configuration to a file in the same directory as the executable and not touch the Windows registry or create any dot directories in the home folder of the user that's running the program on Linux or Mac. And as far as creating a cross-distribution Linux binary, 32-bit POSIX/X11 would probably be the safest bet. I'm not sure what JungleDisk uses as I'm currently on a Mac.
There do exist quite few portable libraries just examples I've worked within the past
1) glib and gtk+
2) libcurl
3) libapr
Those cover nearly every platform and so they are extremly useful tool.
Posix is fine on Unices but well I doubt it's that great on windows, besides we do not have any stuff for portable GUIs there.
I also second the recommendation to separate code for different platforms into different modules/trees instead of ifdefs.
Also I recommend to check beforehand what are the differences in you platforms and how you could abstract them. E.g. this is some OS related stuff (e.g. the annoying CR,CRLF,LF in text files), or hardware stuff. E.g. the previous mentioned posix compability doesnt stop you from
int c;
fread(&c, sizeof(int), 1, file);
But on different hardware platforms the internal memory layout can be complete different (endianess), forcing you to use conversion functions on some of the target platforms.
You can use NAppGUI for both console and desktop apps. The SDK uses ANSI-C and your code will work on Windows/macOS/Linux.
https://www.nappgui.com
It's free and OpenSource.