Is self-modifying code possible in a portable manner in C?
The reason I ask is that, in a way, OOP relies on self-modifying code (because the code that executes at run-time is actually generated as data, e.g. in a v-table), and yet, it seems that, if this is taken too far, it would prevent most optimizations in a compiler.
For example:
void add(char *restrict p, char *restrict pAddend, int len)
{
for (int i = 0; i < len; i++)
p[i] += *pAddend;
}
An optimizing compiler could hoist the *pAddend out of the loop, because it wouldn't interfere with p. However, this is no longer a valid optimization in self-modifying code.
In this way, it seems that C doesn't allow for self-modifying code, but at the same time, wouldn't that imply that you can't do some things like OOP in C? Does C really support self-modifying code?
Self-modifying code is not possible in C for many reasons, the most important of which are:
The code generated by the compiler is completely up to the compiler, and might not look anything like what the programmer trying to write code that modifies itself expects. This is a fundamental problem with doing SMC at all, not just a portability problem.
Function and data pointers are completely separate in C; the language provides no way to convert back and forth between them. This issue is not fundamental, since some implementations or higher-level standards (POSIX) guarantee that code and data pointers share a representation.
Aside from that, self-modifying code is just a really really bad idea. 20 years ago it might have had some uses, but nowadays it will result in nothing but bugs, atrocious performance, and portability failures. Note that on some ISAs, whether the instruction cache even sees changes that were made to cached code might be unspecified/unpredictable!
Finally, vtables have nothing to do with self-modifying code. It's purely a matter of modifying function pointers, which are data, not code.
Strictly speaking, self-modifying code cannot be implemented in a portable manner in C or C++ if I understood the standard correctly.
Self modifying code in C/C++ would mean something like this:
uint8_t code_buffer[FUNCTION_SIZE];
void call_function(void)
{
... modify code_buffer here to the machine code we'd like to run.
((void (*)(void))code_buffer)();
}
This is not legal and will crash on most modern architectures. This is impossible to implement on Harvard architectures as executable code is strictly read-only, so it cannot be part of any standard.
Most modern OSes do have a facility to be able to do this hackery, which is used by dynamic recompilers for one. mprotect() in Unix for example.
Related
So, im programming in C89, and its going well so far except one issue, Im doing multithreaded applications and I need to use atomic.
I dont want to switch to C11 because I want my code to be compatable on every compiler and system and for my code to last a very long time.
Iv'e searched through stackoverflow for an existing question on this topic but didn't find any questions.
Does anyone know how to use the Atomic in C89.
Say I have two threads using a bool
#include <stdatomic.h>
_Atomic bool theBool = false;
void funFromThirstThread()
{
theBool = true;
}
void funFromSecondThread() /*gets called repeatedly*/
{
if(theBool)
{
/*Do something*/
}
}
The above code is what I would do in C11, using the atomic in that, but how would I do this in C89? Can this be done? Preferably without volatile and locks thanks.
It can't be done.
Prior to C11, to get atomic operations, you had to use inline assembler or compiler-specific intrinsics to access the appropriate instructions. And since the language had no formal memory model, you had to rely on knowledge of compiler-specific internals (often undocumented) to know what optimizations it would or wouldn't perform in what contexts. Or else, throw around a lot of volatiles and cross your fingers. Sometimes both. Nothing was portable in any way, and subtle bugs were common.
If there had been a reliable and portable way to use atomics prior to C11, then C11 probably wouldn't have bothered to include them. There is a very good reason why they did.
Per the comments, you say you are using the plibsys library for threads, and UnholySheep points out that it also has support for atomics. So you should probably just use those. Still, though, keep in mind that a generic C89 compiler doesn't make any promises to avoid optimizations that would break the required memory ordering. Usually they were not smart enough to do such optimizations in the first place, but everything is much more at your own risk.
I dont want to switch to C11 because I want my code to be compatible on every compiler and system and for my code to last a very long time.
That goal is basically unattainable for any program more complex than "Hello World". But my feeling is that using C11 gets you closer to it, not further away.
I have read this, saying that
For example, it is common for programs that are written primarily in C to contain portions that are in an assembly language for optimization of processor efficiency.
I have never seen a program written primarily in C that contains assembly code too, at least not directly as source code. Only, their example with the Linux kernel.
Is this statement true and if so, how could it possibly optimize processor efficiency?
Aren't C code just translated into assembly code by the compiler?
No, it's not true. I'd estimate that less than 1% of C programmers even know how to program in assembly, and the need to use it is very rare. It's generally only needed for very special applications, such as some parts of an OS kernel or programming embedded systems, because they need to perform machine operations that don't have corresponding C code (such as directly manipulating CPU registers). In earlier days some programmers would use it for performance-critical sections of code, but compiler optimizations have improved significantly, and CPUs have gotten faster, so this is rarely needed now. It might still be used in the built-in libraries, so that functions like strcpy() will be as fast as possible. But application programmers almost never have to resort to assembly.
Aren't C code just translated into assembly code by the compiler?
Yes, but...
There are situations where you may want to access a specific register or other platform-specific location, and Standard C doesn't provide good ways to do that. If you want to look at a status word or load/read a data register directly, then you often need to drop down to the assembler level.
Also, even in this age of very smart optimizing compilers, it's still possible for a human assembly programmer to write assembly code that will out-perform code generated by the compiler. If you need to wring every possible cycle out of your code, you may need to "go manual" for a couple of routines.
Imagine a situation where you can't or don't want to use any of the libraries provided by the compiler as "standard", nor any external library. You can't use even the compiler extensions (such as gcc extensions).
What is the remaining part you get if you strip C language of all the things a lot of people use as a matter of course?
In such a way, probably a list of every callable function supported by any big C compiler (not only ANSI C) out-of-box would be satisfying as as answer as it'd at least approximately show the use-case of the language.
First I thought about sizeof() and printf() (those were already clarified in the comments - operator + stdio), so... what remains? In-line assembly seem like an extension too, so that pretty much strips even the option to use assembly with C if I'm right.
Probably in the matter of code it'd be easier to understand. Imagine a code compiled with only e.g. gcc main.c (output flag permitted) that has no #include, nor extern.
int main() {
// replace_me
return 0;
}
What can I call to actually do something else than "boring" type math and casting from type to type?
Note that switch, goto, if, loops and other constructs that do nothing and only allow repeating a piece of code aren't the thing I'm looking for (if it isn't obvious).
(Hopefully the edit clarified wtf I'm actually asking, but Matteo's answer pretty much did it.)
If you remove all libraries essentially you have something similar to a freestanding implementation of C (which still has to provide some libraries - say, string.h, but that's nothing you couldn't easily implement yourself in portable C), and that's what normally you start with when programming microcontrollers and other computers that don't have a ready-made operating system - and what operating system writers in general use when they compile their operating systems.
There you typically have two ways of doing stuff besides "raw" computation:
assembly blocks (where you can do literally anything the underlying machine can do);
memory mapped IO (you set a volatile pointer to some hardware dependent location and read/write from it; that affects hardware stuff).
That's really all you need to build anything - and after all, it all boils down to that stuff anyway, the C library of a regular hosted implementation is normally written in C itself, with some assembly used either for speed or to communicate with the operating system1 (typically the syscalls are invoked through some kind of interrupt).
Again, it's nothing you couldn't implement yourself. But the point of having a standard library is both to avoid to continuously reinvent the wheel, and to have a set of portable functions that spare you to have to rewrite everything knowing the details of each target platform.
And mainstream operating systems, in turn, are generally written in a mix or C and assembly as well.
C has no "built-in" functions as such. A compiler implementation may include "intrinsic" functions that are implemented directly by the compiler without provision of an external library, although a prototype declaration is still required for intrinsics, so you would still normally include a header file for such declarations.
C is a systems-level language with a minimal run-time and start-up requirement. Because it can directly access memory and memory mapped I/O there is very little that it cannot do (and what it cannot do is what you use assembly, in-line assembly or intrinsics for). For example, much of the library code you are wondering what you can do without is written in C. When running in an OS environment however (using C as an application-level rather then system-level language), you cannot practically use C in that manner - the OS has control over such things as I/O and memory-management and in modern systems will normally prevent unmediated access to such resources. Of course that OS itself is likely to largely written in C (and/or C++).
In a standalone of bare-metal environment with no OS, C is often used very early in the bootstrap process initialising hardware and establishing an application execution environment. In fact on ARM Cortex-M processors it is possible to boot directly into C code from reset, since the hardware loads an initial stack-pointer and start address from the vector table on start-up; this being enough to run C code that does not rely on library or static data initialisation - such initialisation can however be written in C before calling main().
Note that sizeof is not a function, it is an operator.
I don't think you really understand the situation.
You don't need a header to call a function in C. You can call with unchecked parameters - a bad idea and an obsolete feature, but still supported. And if a compiler links a library by default instead of only when you explicitly tell it to, that's only a little switch within the compiler to "link libc". Notoriously Unix compilers need to be told to link the math library, it wasn't linked by default because some very early programs didn't use floating point.
To be fair, some standard library functions like memcpy tend to be special-cased these days as they lend themselves to inlining and optimisation.
The standard library is documented and is usually available, though in effect deprecated by Microsoft for security reasons. You can write pretty much any function quite easily with only stdlib functions, what you can't do is fancy IO.
What's the purpose of using assembly language inside a C program? Compilers are able to generate assembly language already. In what cases would it be better to write assembly than C? Is performance a consideration?
In addition to what everyone said: not all CPU features are exposed to C. Sometimes, especially in driver and operating system programming, one needs to explicitly work with special registers and/or commands that are not otherwise available.
Also vector extensions.
That was especially true before the advent of compiler intrinsics. Those alleviate the need for inline assembly somewhat.
One more use case for inline assembly has to do with interfacing C with reflected languages. Specifically, assembly is all but necessary if you need to call a function when its prototype is not known at compile time. In other words, when the quantity and datatypes of that function's arguments are but runtime variables. C variadic functions and the stdarg machinery won't help you in this case - they would help you parse a stack frame, but not build one. In assembly, on the other hand, it's quite doable.
This is not an OS/driver scenario. There are at least two technologies out there - Java's JNI and COM Automation - where this is a must. In case of Automation, I'm talking about the way the COM runtime is marshaling dual interfaces using their type libraries.
I can think of a very crude C alternative to assembly for that, but it'd be ugly as sin. Slightly less ugly in C++ with templates.
Yet another use case: crash/run-time error reporting. For postmortem debugging, you'd want to capture as much of program state at the point of crash as possible (i. e. all the CPU registers), and assembly is a much better vehicle for that than C. Postmortem debugging of crashing native code usually involves staring at the assembly anyway.
Yet another use case - code that is intended for execution in another process without that process' co-operation or knowledge. This is often referred to as "shellcode", but it doesn't have to be shell related. Code like that needs to be very carefully written, and it can't rely on the conveniences of a high level language (like the run time library, or having a data section) that are normally taken for granted. When one is after injecting a significant piece of functionality into a target process, they usually end up loading a dynamic library, but the initial trampoline code that loads the library and passes control to it tends to be in assembly.
I've been only covering cases where assembly is necessary. Hand-optimizing for performance is covered in other answers.
There are a few, although not many, cases where hand-optimized assembly language can be made to run more efficiently than assembly language generated by C compilers from C source code. Also, for developers used to assembly language, some things can just seem easier to write in assembler.
For these cases, many C compilers allow inline assembly.
However, this is becoming increasingly rare as C compilers get better and better and producing efficient code, and most platforms put restrictions on some of the low-level type of software that is often the type of software that benefits most from being written in assembler.
In general, it is performance but performance of a very specific kind. For example, the SIMD parallel instructions of a processor might not generated by the compiler. By utilizing processor specific data formats and then issuing processor specific parallel instructions (e.g. ARM NEON or Intel SSE), very fast performance on graphics or signal processing problems can occur. Even then, some compilers allow these to be expressed in C using intrinsic functions.
While it used to be common to use assembly language inserts to hand-optimize critical functions, those days are largely done. Modern compilers are very good and modern processors have very complicated timing requirements so hand optimized code is often less optimal than expected.
There were various reasons to write inline assemblies in C. We can simply categorize the reasons into necessary and unnecessary.
For the reasons of unnecessary, possibly be:
platform compatibility
performance concerning
code optimization
etc.
I consider above as unnecessary because sometime they can be discard or implemented through pure C. For example of platform compatibility, you can totally implement particular version for each platform, however, use inline assemblies might reduce the effort. Here we are not going to talk too much about the unnecessary reasons.
For necessary reasons, they possibly be:
something with standard libraries was insufficient to do
some instruction set was not supported by compilers
object code generated incorrectly
writing stack-sensitive code
etc.
These reasons considered necessary, because of they are almost not possibly done with pure C language. For example, in old DOSes, software interrupt INT21 was not reentrantable. If you want to write a Virtual Dirve fully use INT21 supported by the compiler, it was impossible to do. In this situation, you would need to hook the original INT21, and make it reentrantable. However, the compiled code wraps your every call with prolog/epilog. Thus, you can never break something restricted, or you just crashed the code. You can try any of trick by using the pure language of C with libraries; but even you can successfully find a trick, that would mean you found a particular order that the compiler generates the machine code; this is implying: you tried to let the compiler compiles your code to exactly machine code. So, why not just write inline assemblies directly?
This example explained all above of necessary reasons except instruction set not supported, but I think that was easy to think about.
In fact, there're more reasons to write inline assemblies, but now you have some ideas of them, and so on.
Just as a curiosity, I'm adding here a concrete example of something not-so-low-level you can only do in assembly. I read this in an assembly book from my university time where it was used to show an inherent limitation of C/C++, and how to overcome it with assembly.
The problem is how do I invoke a function when the exact number of parameters is only known at runtime? In fact, in C/C++ you can easily define a function that takes a variable number of arguments like printf. But when it comes to calling that function, the compiler wants to know exactly how many parameters must be passed. You may pass more paremters than required, that won't do any harm. But what if the number grows unexpectedly to 100 or 1000 parameters, and must be picked out of an array?
The solution of course is using assembly, where you can dynamically create a stack frame of the proper size, copy the parameters on the stack, invoke the function, and finally reset the stack.
In practice, this would hardly ever be a limitation (except if the library you're using is really really bad designed). People who use assembly in C have much better reasons to do so like others have pointed out in their answers. Still, I think may be an interesting fact to know.
I would rather think of that as a way to write a very specific code for a specific platform, optimization, though still common, is used less nowadays. Knowledge and usage of assembly in C is also practiced by all-color hats.
I wonder why other languages do not support this feature. What I can understand that C / C++ code is platform dependent so to make it work (compile and execute) across various platform, is achieved by using preprocessor directives. And there are many other uses of this apart from this. Like you can put all your debug printf's inside #if DEBUG ... #endif. So while making the release build these lines of code do not get compiled in the binary. But in other languages, achieving this thing (later part) is difficult (or may be impossible, I'm not sure). All code will get compiled in the binary increasing its size. So my question is "why do Java, or other modern compiled languages no support this kind of feature?" which allows you to include or exclude some piece of code from the binary in a much handy way.
The major languages that don't have a preprocessor usually have a different, often cleaner, way to achieve the same effects.
Having a text-preprocessor like cpp is a mixed blessing. Since cpp doesn't actually know C, all it does is transform text into other text. This causes many maintenance problems. Take C++ for example, where many uses of the preprocessor have been explicitly deprecated in favor of better features like:
For constants, const instead of #define
For small functions, inline instead of #define macros
The C++ FAQ calls macros evil and gives multiple reasons to avoid using them.
The portability benefits of the preprocessor are far outweighed by the possibilities for abuse. Here are some examples from real codes I have seen in industry:
A function body becomes so tangled with #ifdef that it is very hard to read the function and figure out what is going on. Remember that the preprocessor works with text not syntax, so you can do things that are wildly ungrammatical
Code can become duplicated in different branches of an #ifdef, making it hard to maintain a single point of truth about what's going on.
When an application is intended for multiple platforms, it becomes very hard to compile all the code as opposed to whatever code happens to be selected for the developer's platform. You may need to have multiple machines set up. (It is expensive, say, on a BSD system to set up a cross-compilation environment that accurately simulates GNU headers.) In the days when most varieties of Unix were proprietary and vendors had to support them all, this problem was very serious. Today when so many versions of Unix are free, it's less of a problem, although it's still quite challenging to duplicate native Windows headers in a Unix environment.
It Some code is protected by so many #ifdefs that you can't figure out what combination of -D options is needed to select the code. The problem is NP-hard, so the best known solutions require trying exponentially many different combinations of definitions. This is of course impractical, so the real consequence is that gradually your system fills with code that hasn't been compiled. This problem kills refactoring, and of course such code is completely immune to your unit tests and your regression tests—unless you set up a huge, multiplatform testing farm, and maybe not even then.
In the field, I have seen this problem lead to situations where a refactored application is carefully tested and shipped, only to receive immediate bug reports that the application won't even compile on other platforms. If code is hidden by #ifdef and we can't select it, we have no guarantee that it typechecks—or even that it is syntactically correct.
The flip side of the coin is that more advanced languages and programming techniques have reduced the need for conditional compilation in the preprocessor:
For some languages, like Java, all the platform-dependent code is in the implementation of the JVM and in the associated libraries. People have gone to huge lengths to make JVMs and libraries that are platform-independent.
In many languages, such as Haskell, Lua, Python, Ruby, and many more, the designers have gone to some trouble to reduce the amount of platform-dependent code compared to C.
In a modern language, you can put platform-dependent code in a separate compilation unit behind a compiled interface. Many modern compilers have good facilities for inlining functions across interface boundaries, so that you don't pay much (or any) penalty for this kind of abstraction. This wasn't the case for C because (a) there are no separately compiled interfaces; the separate-compilation model assumes #include and the preprocessor; and (b) C compilers came of age on machines with 64K of code space and 64K of data space; a compiler sophisticated enough to inline across module boundaries was almost unthinkable. Today such compilers are routine. Some advanced compilers inline and specialize methods dynamically.
Summary: by using linguistic mechanisms, rather than textual replacement, to isolate platform-dependent code, you expose all your code to the compiler, everything gets type-checked at least, and you have a chance of doing things like static analysis to ensure suitable test coverage. You also rule out a whole bunch of coding practices that lead to unreadable code.
Because modern compilers are smart enough to remove dead code in most any case, making manually feeding the compiler this way no longer necessary. I.e. instead of :
#include <iostream>
#define DEBUG
int main()
{
#ifdef DEBUG
std::cout << "Debugging...";
#else
std::cout << "Not debugging.";
#endif
}
you can do:
#include <iostream>
const bool debugging = true;
int main()
{
if (debugging)
{
std::cout << "Debugging...";
}
else
{
std::cout << "Not debugging.";
}
}
and you'll probably get the same, or at least similar, code output.
Edit/Note: In C and C++, I'd absolutely never do this -- I'd use the preprocessor, if nothing else that it makes it instantly clear to the reader of my code that a chunk of it isn't supposed to be complied under certain conditions. I am saying, however, that this is why many languages eschew the preprocessor.
A better question to ask is why did C resort to using a pre-processor to implement these sorts of meta-programming tasks? It isn't a feature as much as it is a compromise to the technology of the time.
The pre-processor directives in C were developed at a time when machine resources (CPU speed, RAM) were scarce (and expensive). The pre-processor provided a way to implement these features on slow machines with limited memory. For example, the first machine I ever owned had 56KB of RAM and a 2Mhz CPU. It still had a full K&R C compiler available, which pushed the system's resources to the limit, but was workable.
More modern languages take advantage of today's more powerful machines to provide better ways of handling the sorts of meta-programming tasks that the pre-processor used to deal with.
Other languages do support this feature, by using a generic preprocessor such as m4.
Do we really want every language to have its own text-substitution-before-execution implementation?
The C pre-processor can be run on any text file, it need not be C.
Of course, if run on another language, it might tokenize in weird ways, but for simple block structures like #ifdef DEBUG, you can put that in any language, run the C pre-processor on it, then run your language specific compiler on it, and it will work.
Note that macros/preprocessing/conditionals/etc are usually considered a compiler/interpreter feature, as opposed to a language feature, because they are usually completely independent of the formal language definition, and might vary from compiler to compiler implementation for the same language.
A situation in many languages where conditional compilation directives can be better than if-then-else runtime code is when compile-time statements (such as variable declarations) need to be conditional. For example
$if debug
array x
$endif
...
$if debug
dump x
$endif
only declares/allocates/compiles x when needing x, whereas
array x
boolean debug
...
if debug then dump x
probably has to declare x regardless of whether debug is true.
Many modern languages actually have syntactic metaprogramming capabilities that go way beyond CPP. Pretty much all modern Lisps (Arc, Clojure, Common Lisp, Scheme, newLISP, Qi, PLOT, MISC, ...) for example have extremely powerful (Turing-complete, actually) macro systems, so why should they limit themselves to the crappy CPP style macros which aren't even real macros, just text snippets?
Other languages with powerful syntactic metaprogramming include Io, Ioke, Perl 6, OMeta, Converge.
Because decreasing the size of the binary:
Can be done in other ways (compare the average size of a C++ executable to a C# executable, for example).
Is not that important, when it you weigh it against being able to write programs that actually work.
Other languages also have better dynamic binding. For example, we have some code that we cannot ship to some customers for export reasons. Our "C" libraries use #ifdef statements and elaborate Makefile tricks (which is pretty much the same).
The Java code uses plugins (ala Eclipse), so that we just don't ship that code.
You can do the same thing in C through the use of shared libraries... but the preprocessor is a lot simpler.
A other point nobody else mentioned is platform support.
Most modern languages can not run on the same platforms as C or C++ can and are not intended to run on this platforms. For example, Java, Python and also native compiled languages like C# need a heap, they are designed to run on a OS with memory management, libraries and large amount of space, they do not run in a freestanding environment. There you can use other ways to archive the same. C can be used to program controllers with 2KiB ROM, there you need a preprocessor for most applications.