Can anyone point me to the definition of strlen() in GCC? I've been grepping release 4.4.2 for about a half hour now (while Googling like crazy) and I can't seem to find where strlen() is actually implemented.
You should be looking in glibc, not GCC -- it seems to be defined in strlen.c -- here's a link to strlen.c for glibc version 2.7... And here is a link to the glibc SVN repository online for strlen.c.
The reason you should be looking at glibc and not gcc is:
The GNU C library is used as the C library in the GNU system and most systems with the Linux kernel.
Here's the bsd implementation
size_t
strlen(const char *str)
{
const char *s;
for (s = str; *s; ++s)
;
return (s - str);
}
I realize this question is 4yrs old, but gcc will often include its own copy of strlen if you do not #include <string.h> and none of the answers (including the accepted answer) account for that. If you forget, you will get a warning:
file_name:line_number: warning: incompatible implicit declaration of built-in function 'strlen'
and gcc will inline its copy which on x86 is the repnz scasb asm variant unless you pass -Werror or -fno-builtin. The files related to this are in gcc/config/<platform>/<platform>.{c,md}
It is also controlled by gcc/builtins.c. In case you wondered if and how a strlen() was optimized to a constant, see the function defined as tree c_strlen(tree src, int only_value) in this file. It also controls how strlen (amongst others) is expanded and folded (based on the previously mentioned config/platform)
defined in glibc/string/strlen.c
#include <string.h>
#include <stdlib.h>
#undef strlen
#ifndef STRLEN
# define STRLEN strlen
#endif
/* Return the length of the null-terminated string STR. Scan for
the null terminator quickly by testing four bytes at a time. */
size_t
STRLEN (const char *str)
{
const char *char_ptr;
const unsigned long int *longword_ptr;
unsigned long int longword, himagic, lomagic;
/* Handle the first few characters by reading one character at a time.
Do this until CHAR_PTR is aligned on a longword boundary. */
for (char_ptr = str; ((unsigned long int) char_ptr
& (sizeof (longword) - 1)) != 0;
++char_ptr)
if (*char_ptr == '\0')
return char_ptr - str;
/* All these elucidatory comments refer to 4-byte longwords,
but the theory applies equally well to 8-byte longwords. */
longword_ptr = (unsigned long int *) char_ptr;
/* Bits 31, 24, 16, and 8 of this number are zero. Call these bits
the "holes." Note that there is a hole just to the left of
each byte, with an extra at the end:
bits: 01111110 11111110 11111110 11111111
bytes: AAAAAAAA BBBBBBBB CCCCCCCC DDDDDDDD
The 1-bits make sure that carries propagate to the next 0-bit.
The 0-bits provide holes for carries to fall into. */
himagic = 0x80808080L;
lomagic = 0x01010101L;
if (sizeof (longword) > 4)
{
/* 64-bit version of the magic. */
/* Do the shift in two steps to avoid a warning if long has 32 bits. */
himagic = ((himagic << 16) << 16) | himagic;
lomagic = ((lomagic << 16) << 16) | lomagic;
}
if (sizeof (longword) > 8)
abort ();
/* Instead of the traditional loop which tests each character,
we will test a longword at a time. The tricky part is testing
if *any of the four* bytes in the longword in question are zero. */
for (;;)
{
longword = *longword_ptr++;
if (((longword - lomagic) & ~longword & himagic) != 0)
{
/* Which of the bytes was the zero? If none of them were, it was
a misfire; continue the search. */
const char *cp = (const char *) (longword_ptr - 1);
if (cp[0] == 0)
return cp - str;
if (cp[1] == 0)
return cp - str + 1;
if (cp[2] == 0)
return cp - str + 2;
if (cp[3] == 0)
return cp - str + 3;
if (sizeof (longword) > 4)
{
if (cp[4] == 0)
return cp - str + 4;
if (cp[5] == 0)
return cp - str + 5;
if (cp[6] == 0)
return cp - str + 6;
if (cp[7] == 0)
return cp - str + 7;
}
}
}
}
libc_hidden_builtin_def (strlen)
glibc 2.26 has several hand optimized assembly implementations of strlen
As of glibc-2.26, a quick:
git ls-files | grep strlen.S
in the glibc tree shows a dozen of assembly hand-optimized implementations for all major archs and variations.
In particular, x86_64 alone has 3 variations:
sysdeps/x86_64/multiarch/strlen-avx2.S
sysdeps/x86_64/multiarch/strlen-sse2.S
sysdeps/x86_64/strlen.S
A quick and dirty way to determine which one is used, is to step debug a test program:
#include <assert.h>
#include <stdlib.h>
#include <string.h>
#include <stdio.h>
int main(void) {
size_t size = 0x80000000, i, result;
char *s = malloc(size);
for (i = 0; i < size; ++i)
s[i] = 'a';
s[size - 1] = '\0';
result = strlen(s);
assert(result == size - 1);
return EXIT_SUCCESS;
}
compiled with:
gcc -ggdb3 -std=c99 -O0 a.c
Off the bat:
disass main
contains:
callq 0x555555554590 <strlen#plt>
so the libc version is being called.
After a few si instruction level steps into that, GDB reaches:
__strlen_avx2 () at ../sysdeps/x86_64/multiarch/strlen-avx2.S:52
52 ../sysdeps/x86_64/multiarch/strlen-avx2.S: No such file or directory.
which tells me that strlen-avx2.S was used.
Then, I further confirm with:
disass __strlen_avx2
and compare the disassembly with the glibc source.
It is not surprising that the AVX2 version was used, since I have an i7-7820HQ CPU with launch date Q1 2017 and AVX2 support, and AVX2 is the most advanced of the assembly implementations, with launch date Q2 2013, while SSE2 is much more ancient from 2004.
This is where a great part of the hardcoreness of glibc comes from: it has a lot of arch optimized hand written assembly code.
Tested in Ubuntu 17.10, gcc 7.2.0, glibc 2.26.
-O3
TODO: with -O3, gcc does not use glibc's strlen, it just generates inline assembly, which is mentioned at: https://stackoverflow.com/a/19885891/895245
Is it because it can optimize even better? But its output does not contain AVX2 instructions, so I feel that this is not the case.
https://www.gnu.org/software/gcc/projects/optimize.html mentions:
Deficiencies of GCC's optimizer
glibc has inline assembler versions of various string functions; GCC has some, but not necessarily the same ones on the same architectures. Additional optab entries, like the ones for ffs and strlen, could be provided for several more functions including memset, strchr, strcpy and strrchr.
My simple tests show that the -O3 version is actually faster, so GCC made the right choice.
Asked at: https://www.quora.com/unanswered/How-does-GCC-know-that-its-builtin-implementation-of-strlen-is-faster-than-glibcs-when-using-optimization-level-O3
Although the original poster may not have known this or been looking for this, gcc internally inlines a number of so-called "builtin" c functions that it defines on its own, including some of the mem*() functions and (depending on the gcc version) strlen. In such cases, the library version is essentially never used, and pointing the person at the version in glibc is not strictly speaking correct. (It does this for performance reasons -- in addition to the improvement that inlining itself produces, gcc "knows" certain things about the functions when it provides them, such as, for example, that strlen is a pure function and that it can thus optimize away multiple calls, or in the case of the mem*() functions that no aliasing is taking place.)
For more information on this, see http://gcc.gnu.org/onlinedocs/gcc/Other-Builtins.html
Is this what you are looking for? strlen() source. See the git repository for more information. The glibc resources page has links to the git repositories if you want to grab them rather than looking at the web view.
Google Code Search is a good starting point for questions like that. They usually point to various different sources and implementations of a function.
In your particular case: GoogleCodeSearch(strlen)
Google Code Search was completely shut down on March 2013
I realize that this is old question, you can find the linux kernel sources at github here, and the 32 bit implementation for strlen() could be found in strlen_32.c on github. The mentioned file has this implementation.
#include <linux/types.h>
#include <linux/string.h>
#include <linux/module.h>
size_t strlen(const char *s)
{
/* Get an aligned pointer. */
const uintptr_t s_int = (uintptr_t) s;
const uint32_t *p = (const uint32_t *)(s_int & -4);
/* Read the first word, but force bytes before the string to be nonzero.
* This expression works because we know shift counts are taken mod 32.
*/
uint32_t v = *p | ((1 << (s_int << 3)) - 1);
uint32_t bits;
while ((bits = __insn_seqb(v, 0)) == 0)
v = *++p;
return ((const char *)p) + (__insn_ctz(bits) >> 3) - s;
}
EXPORT_SYMBOL(strlen);
You can use this code, the simpler the better !
size_t Strlen ( const char * _str )
{
size_t i = 0;
while(_str[i++]);
return i;
}
Related
This question already has answers here:
Find size of a function in C
(9 answers)
Closed 4 years ago.
I am writing code for an embedded system where it is more efficient to copy code from ROM to the SOC's internal memory then execute it. Is there any way to programmatically get the size in bytes of the function so I can use a function like memcpy to copy the function's instructions to internal memory.
Is there a better way to do this?
Is there a better way to do this?
There likely is, and I truly hope someone else responds with an answer that provides a simpler approach, but for now I'll shed some details on your suggested method.
If the program is compiled in ELF format (I don't know for the other formats), all your functions will be included in the .text section of the ELF file. You can use the symbol table to find this function in the text section. To get the size of this function, you might be able to use the st_size member of the Elf64_Sym or Elf32_Sym struct, but I'm not entirely certain that will give the correct size. What you could do (a little hacky, admittedly) is iterate through the other symbols, and find the one immediately after it, and subtract to get the size. Of course you'd have to keep in mind alignment rules, but that's not too much of an issue- if you copy extra bytes, they won't be executed anyways.
Also keep in mind that some code get compiled with certain assumptions about its offset in memory. You'll might need to manually patch the GOT and/or PLT if you copy the function directly into memory. Know you should probably compile the function you want to include with -PIC and -fPIC for position independent code, at least in GCC.
If you need more details on how to access the symbol table, or the text section of your ELF, I could add more details.
With some compilers you can get a function size by computing the difference between the function's address and the address of another function that immediatly follows the first one.
But it really depends of the compiler. With Visual C++ for example, both functions has to be static functions. With GCC, it does not work anymore if optimization O2 or better is activated.
And even if you manage to copy your function elsewhere in memory, you may not be able to use it, especially if it refers other functions, or if it refers global/static variables, or if the code is not position independant, etc.
So this is a simple solution, it may work in your case, but it can't be considered as a general solution.
Below there's an example that works with gcc and visual C++, tested on windows 10 and WSL (do not activate optimizations with gcc).
#include <stdio.h>
#include <string.h>
#ifdef _WIN32
#include <windows.h>
#endif
#ifdef __linux
#include <sys/mman.h>
#endif
// The function to copy
static int fib(int m)
{
int v1 = 0, v2 = 1, n;
if (m == 0) return 0;
for (n = 1; n < m; n++)
{
int v = v1 + v2;
v1 = v2;
v2 = v;
}
return v2;
}
static void endFib(void)
{
// This function follow immediatly the fib function
// and it exists only to get its address and compute the size of fib function
}
int main(int argc, char *argv)
{
long sizeFib;
int (*copyFib)(int);
printf("&fib=%p\n", (char *)fib);
sizeFib = (char *)endFib - (char *)fib;
printf("size of fib : %ld\n", sizeFib);
printf("fib(8) : %d\n", fib(8));
// For the example the allocated copy must be in an executable part of the memory.
#ifdef _WIN32
copyFib = VirtualAlloc(NULL, sizeFib, MEM_COMMIT, PAGE_EXECUTE_READWRITE);
#endif
#ifdef __linux
copyFib = mmap(NULL, sizeFib, PROT_EXEC | PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, 0, 0);
#endif
memcpy(copyFib, fib, sizeFib);
printf("©Fib=%p\n", copyFib);
printf("copyFib(8) : %d\n", copyFib(8));
return 0;
}
I am calling two functions on my char* s = "pratik" as:
User code:
#include <zlib.h>
int main()
{
char *s = "pratik";
printf("%x\n",crc32(0x80000000, s, strlen(s)));
return 0;
}
Output:
66fa3c99
Kernel code:
#include <linux/crc32.h>
int main()
{
char *s = "pratik";
u32 checksum = crc32(0x80000000, s, strlen(s));
printk("\nChecksum --> %x", checksum);
return checksum;
}
Output:
Checksum --> d7389d3a
Why are the values of the checksums on the same strings different?
It appears that someone was disturbed by the fact that the standard Ethernet (PKZIP, ITU V.42 etc. etc.) CRC-32 does a pre- and post-exclusive-or with 0xffffffff. So the version in the Linux kernel leaves that out, and expects the application to do that. Go figure.
Anyway, you can get the same result as the (correct) zlib crc32(), using the (non-standard) Linux crc32() instead, thusly:
crc_final = crc32(crc_initial ^ 0xffffffff, buf, len) ^ 0xffffffff;
In fact, that exact same code would allow you to duplicate the Linux crc32() using the zlib crc32() as well.
I am new to use XeonPhi Intel co-processor. I want to write code for a simple Vector sum using AVX 512 bit instructions. I use k1om-mpss-linux-gcc as a compiler and want to write inline assembly. Here it is my code:
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/time.h>
#include <assert.h>
#include <stdint.h>
void* aligned_malloc(size_t size, size_t alignment) {
uintptr_t r = (uintptr_t)malloc(size + --alignment + sizeof(uintptr_t));
uintptr_t t = r + sizeof(uintptr_t);
uintptr_t o =(t + alignment) & ~(uintptr_t)alignment;
if (!r) return NULL;
((uintptr_t*)o)[-1] = r;
return (void*)o;
}
int main(int argc, char* argv[])
{
printf("Starting calculation...\n");
int i;
const int length = 65536;
unsigned *A = (unsigned*) aligned_malloc(length * sizeof(unsigned), 64);
unsigned *B = (unsigned*) aligned_malloc(length * sizeof(unsigned), 64);
unsigned *C = (unsigned*) aligned_malloc(length * sizeof(unsigned), 64);
for(i=0; i<length; i++){
A[i] = 1;
B[i] = 2;
}
const int AVXLength = length / 16;
unsigned char * pA = (unsigned char *) A;
unsigned char * pB = (unsigned char *) B;
unsigned char * pC = (unsigned char *) C;
for(i=0; i<AVXLength; i++ ){
__asm__("vmovdqa32 %1,%%zmm0\n"
"vmovdqa32 %2,%%zmm1\n"
"vpaddd %0,%%zmm0,%%zmm1;"
: "=m" (pC) : "m" (pA), "m" (pB));
pA += 64;
pB += 64;
pC += 64;
}
// To prove that the program actually worked
for (i=0; i <5 ; i++)
{
printf("C[%d] = %f\n", i, C[i]);
}
}
However when I run the program, I've got segmentation fault from the asm part. Can somebody help me with that???
Thanks
Xeon Phi Knights Corner doesn't support AVX. It only supports a special set of vector extensions, called Intel Initial Many Core Instructions (Intel IMCI) with a vector size of 512b. So trying to put any sort of AVX specific assembly into a KNC code will lead to crashes.
Just wait for Knights Landing. It will support AVX-512 vector extensions.
Although Knights Corner (KNC) does not have AVX512 it has something very similar. Many of the mnemonics are the same. In fact, in the OP's case the mnemoics vmovdqa32 and vpaddd are the same for AVX512 and KNC.
The opcodes likely differ but the compiler/assembler takes care of this. In the OPs case he/she is using a special version of GCC, k1om-mpss-linux-gcc which is part of the many core software stack KNC which presumably generates the correct opcodes. One can compile on the host using k1om-mpss-linux-gcc and then scp the binary to the KNC card. I learned about this from a comment in this question.
As to why the OPs code is failing I can only make guess since I don't have a KNC card to test with.
In my limited experience with GCC inline assembly I have learned that it's good to look at the generated assembly in the object file to make sure the compiler did what you expect.
When I compile your code with a normal version of GCC I see that the line "vpaddd %0,%%zmm0,%%zmm1;" produces assembly with the semicolon. I don't think the semicolon should be there. That could be one problem.
But since the OPs mnemonics are the same as AVX512 we can using AVX512 intrinsics to figure out the correct assembly
#include <x86intrin.h>
void foo(int *A, int *B, int *C) {
__m512i a16 = _mm512_load_epi32(A);
__m512i b16 = _mm512_load_epi32(B);
__m512i s16 = _mm512_add_epi32(a16,b16);
_mm512_store_epi32(C, s16);
}
and gcc -mavx512f -O3 -S knc.c procudes
vmovdqa64 (%rsi), %zmm0
vpaddd (%rdi), %zmm0, %zmm0
vmovdqa64 %zmm0, (%rdx)
GCC chose vmovdqa64 instead of vmovdqa32 even though the Intel documentaion says it should be vmovdqa32. I am not sure why. I don't know what the difference is. I could have used the intrinsic _mm512_load_si512 which does exist and according to Intel should map vmovdqa32 but GCC maps it to vmovdqa64 as well. I am not sure why there are also _mm512_load_epi32 and _mm512_load_epi64 now. SSE and AVX don't have these corresponding intrinsics.
Based on GCC's code here is the inline assembly I would use
__asm__ ("vmovdqa64 (%1), %%zmm0\n"
"vpaddd (%2), %%zmm0, %%zmm0\n"
"vmovdqa64 %%zmm0, (%0)"
:
: "r" (pC), "r" (pA), "r" (pB)
: "memory"
);
Maybe vmovdqa32 should be used instead of vmovdqa64 but I expect it does not matter.
I used the register modifier r instead of the memory modifier m because from past experience m the memory modifier did not produce the assembly I expected.
Another possibility to consider is to use a version of GCC that supports AVX512 intrinsics to generate the assembly and then use the special KNC version of GCC to convert the assembly to binary. For example
gcc-5.1 -O3 -S foo.c
k1om-mpss-linux-gcc foo.s
This may be asking for trouble since k1om-mpss-linux-gcc is likely an older version of GCC. I have never done something like this before but it may work.
As explained here the reason the AVX512 intrinsics
_mm512_load/store(u)_epi32
_mm512_load/store(u)_epi64
_mm512_load/store(u)_si512
is that the parameters have been converted to void*. For example with SSE you have to cast
int *x;
__m128i v;
__mm_store_si128((__m128*)x,v)
whereas with SSE you no longer need to
int *x;
__m512i;
__mm512_store_epi32(x,v);
//__mm512_store_si512(x,v); //this is also fine
It's still not clear to me why there is vmovdqa32 and vmovdqa64 (GCC only seems to use vmovdqa64 currently) but it's probably similar to movaps and movapd in SSE which have not real difference and exists only in case they may make a difference in the future.
The purpose of vmovdqa32 and vmovdqa64 is for masking which can be doing with these intrsics
_mm512_mask_load/store_epi32
_mm512_mask_load/store_epi64
Without masks the instructions are equivalent.
I am trying to make the .text section writable for a C program. I looked through the options provided in this SO question and zeroed on modifying the linker script to achieve this.
For this I created a writable memory region using
MEMORY { rwx (wx) : ORIGIN = 0x400000, LENGTH = 256K}
and at the section .text added:
.text :
{
*(.text.unlikely .text.*_unlikely)
*(.text.exit .text.exit.*)
*(.text.startup .text.startup.*)
*(.text.hot .text.hot.*)
*(.text .stub .text.* .gnu.linkonce.t.*)
/* .gnu.warning sections are handled specially by elf32.em. */
*(.gnu.warning)
} >rwx
On compiling the code with gcc flag -T and giving my linker file as an argument I am getting an error:
error: no memory region specified for loadable section '.interp'
I am only trying to change the memory permissions for the .text region. Working on Ubuntu x86_64 architecture.
Is there a better way to do this?
Any help is highly appreciated.
Thanks
The Linker Script
Linker Script on pastie.org
In Linux, you can use mprotect() to enable/disable text section write protection from the runtime code; see the Notes section in man 2 mprotect.
Here is a real-world example. First, however, a caveat:
I consider this just a proof of concept implementation, and not something I'd ever use in a real world application. It may look enticing for use in a high-performance library of some sort, but in my experience, changing the API (or the paradigm/approach) of the library usually yields much better results -- and fewer hard-to-debug bugs.
Consider the following six files:
foo1.c:
int foo1(const int a, const int b) { return a*a - 2*a*b + b*b; }
foo2.c:
int foo2(const int a, const int b) { return a*a + b*b; }
foo.h.header:
#ifndef FOO_H
#define FOO_H
extern int foo1(const int a, const int b);
extern int foo2(const int a, const int b);
foo.h.footer:
#endif /* FOO_H */
main.c:
#include <unistd.h>
#include <sys/mman.h>
#include <errno.h>
#include <string.h>
#include <stdio.h>
#include "foo.h"
int text_copy(const void *const target,
const void *const source,
const size_t length)
{
const long page = sysconf(_SC_PAGESIZE);
void *start = (char *)target - ((long)target % page);
size_t bytes = length + (size_t)((long)target % page);
/* Verify sane page size. */
if (page < 1L)
return errno = ENOTSUP;
/* Although length should not need to be a multiple of page size,
* adjust it up if need be. */
if (bytes % (size_t)page)
bytes = bytes + (size_t)page - (bytes % (size_t)page);
/* Disable write protect on target pages. */
if (mprotect(start, bytes, PROT_READ | PROT_WRITE | PROT_EXEC))
return errno;
/* Copy code.
* Note: if the target code is being executed, we're in trouble;
* this offers no atomicity guarantees, so other threads may
* end up executing some combination of old/new code.
*/
memcpy((void *)target, (const void *)source, length);
/* Re-enable write protect on target pages. */
if (mprotect(start, bytes, PROT_READ | PROT_EXEC))
return errno;
/* Success. */
return 0;
}
int main(void)
{
printf("foo1(): %d bytes at %p\n", foo1_SIZE, foo1_ADDR);
printf("foo2(): %d bytes at %p\n", foo2_SIZE, foo2_ADDR);
printf("foo1(3, 5): %d\n", foo1(3, 5));
printf("foo2(3, 5): %d\n", foo2(3, 5));
if (foo2_SIZE < foo1_SIZE) {
printf("Replacing foo1() with foo2(): ");
if (text_copy(foo1_ADDR, foo2_ADDR, foo2_SIZE)) {
printf("%s.\n", strerror(errno));
return 1;
}
printf("Done.\n");
} else {
printf("Replacing foo2() with foo1(): ");
if (text_copy(foo2_ADDR, foo1_ADDR, foo1_SIZE)) {
printf("%s.\n", strerror(errno));
return 1;
}
printf("Done.\n");
}
printf("foo1(3, 5): %d\n", foo1(3, 5));
printf("foo2(3, 5): %d\n", foo2(3, 5));
return 0;
}
function-info.bash:
#!/bin/bash
addr_prefix=""
addr_suffix="_ADDR"
size_prefix=""
size_suffix="_SIZE"
export LANG=C
export LC_ALL=C
nm -S "$#" | while read addr size kind name dummy ; do
[ -n "$addr" ] || continue
[ -n "$size" ] || continue
[ -z "$dummy" ] || continue
[ "$kind" = "T" ] || continue
[ "$name" != "${name#[A-Za-z]}" ] || continue
printf '#define %s ((void *)0x%sL)\n' "$addr_prefix$name$addr_suffix" "$addr"
printf '#define %s %d\n' "$size_prefix$name$size_suffix" "0x$size"
done || exit $?
Remember to make it executable using chmod u+x ./function-info.bash
First, compile the sources using valid sizes but invalid addresses:
gcc -W -Wall -O3 -c foo1.c
gcc -W -Wall -O3 -c foo2.c
( cat foo.h.header ; ./function-info.bash foo1.o foo2.o ; cat foo.h.footer) > foo.h
gcc -W -Wall -O3 -c main.c
The sizes are correct but the addresses are not, because the code is yet to be linked. Relative to the final binary, the object file contents are usually relocated at link time. So, link the sources to get example executable, example:
gcc -W -Wall -O3 main.o foo1.o foo2.o -o example
Extract the correct (sizes and) addresses:
( cat foo.h.header ; ./function-info.bash example ; cat foo.h.footer) > foo.h
Recompile and link,
gcc -W -Wall -O3 -c main.c
gcc -W -Wall -O3 foo1.o foo2.o main.o -o example
and verify that the constants now do match:
mv -f foo.h foo.h.used
( cat foo.h.header ; ./function-info.bash example ; cat foo.h.footer) > foo.h
cmp -s foo.h foo.h.used && echo "Done." || echo "Recompile and relink."
Due to high optimization (-O3) the code that utilizes the constants may change size, requiring a yet another recompile-relink. If the last line outputs "Recompile and relink", just repeat the last two steps, i.e. five lines.
(Note that since foo1.c and foo2.c do not use the constants in foo.h, they obviously do not need to be recompiled.)
On x86_64 (GCC-4.6.3-1ubuntu5), running ./example outputs
foo1(): 21 bytes at 0x400820
foo2(): 10 bytes at 0x400840
foo1(3, 5): 4
foo2(3, 5): 34
Replacing foo1() with foo2(): Done.
foo1(3, 5): 34
foo2(3, 5): 34
which shows that the foo1() function indeed was replaced. Note that the longer function is always replaced with the shorter one, because we must not overwrite any code outside the two functions.
You can modify the two functions to verify this; just remember to repeat the entire procedure (so that you use the correct _SIZE and _ADDR constants in main()).
Just for giggles, here is the generated foo.h for the above:
#ifndef FOO_H
#define FOO_H
extern int foo1(const int a, const int b);
extern int foo2(const int a, const int b);
#define foo1_ADDR ((void *)0x0000000000400820L)
#define foo1_SIZE 21
#define foo2_ADDR ((void *)0x0000000000400840L)
#define foo2_SIZE 10
#define main_ADDR ((void *)0x0000000000400610L)
#define main_SIZE 291
#define text_copy_ADDR ((void *)0x0000000000400850L)
#define text_copy_SIZE 226
#endif /* FOO_H */
You might wish to use a smarter scriptlet, say an awk one that uses nm -S to obtain all function names, addresses, and sizes, and in the header file replaces only the values of existing definitions, to generate your header file. I'd use a Makefile and some helper scripts.
Further notes:
The function code is copied as-is, no relocation etc. is done. (This means that if the machine code of the replacement function contains absolute jumps, the execution continues in the original code. These example functions were chosen, because they're unlikely to have absolute jumps in them. Run objdump -d foo1.o foo2.o to verify from the assembly.)
That is irrelevant if you use the example just to investigate how to modify executable code within the running process. However, if you build runtime-function-replacing schemes on top of this example, you may need to use position independent code for the replaced code (see the GCC manual for relevant options for your architecture) or do your own relocation.
If another thread or signal handler executes the code being modified, you're in serious trouble. You get undefined results. Unfortunately, some libraries start extra threads, which may not block all possible signals, so be extra careful when modifying code that might be run by a signal handler.
Do not assume the compiler compiles the code in a specific way or uses a specific organization. My example uses separate compilation units, to avoid the cases where the compiler might share code between similar functions.
Also, it examines the final executable binary directly, to obtain the sizes and addresses to be modified to modify an entire function implementation. All verifications should be done on the object files or final executable, and disassembly, instead of just looking at the C code.
Putting any code that relies on the address and size constants into a separate compilation unit makes it easier and faster to recompile and relink the binary. (You only need to recompile the code that uses the constants directly, and you can even use less optimization for that code, to eliminate extra recompile-relink cycles, without impacting the overall code quality.)
In my main.c, both the address and length supplied to mprotect() are page-aligned (based on the user parameters). The documents say only the address has to be. Since protections are page-granular, making sure the length is a multiple of the page size does not hurt.
You can read and parse /proc/self/maps (which is a kernel-generated pseudofile; see man 5 proc, /proc/[pid]/maps section, for further info) to obtain the existing mappings and their protections for the current process.
In any case, if you have any questions, I'd be happy to try and clarify the above.
Addendum:
It turns out that using the GNU extension dl_iterate_phdr() you can enable/disable write protection on all text sections trivially:
#define _GNU_SOURCE
#include <unistd.h>
#include <dlfcn.h>
#include <sys/mman.h>
#include <link.h>
static int do_write_protect_text(struct dl_phdr_info *info, size_t size, void *data)
{
const int protect = (data) ? PROT_READ | PROT_EXEC : PROT_READ | PROT_WRITE | PROT_EXEC;
size_t page;
size_t i;
page = sysconf(_SC_PAGESIZE);
if (size < sizeof (struct dl_phdr_info))
return ENOTSUP;
/* Ignore libraries. */
if (info->dlpi_name && info->dlpi_name[0] != '\0')
return 0;
/* Loop over each header. */
for (i = 0; i < (size_t)info->dlpi_phnum; i++)
if ((info->dlpi_phdr[i].p_flags & PF_X)) {
size_t ptr = (size_t)info->dlpi_phdr[i].p_vaddr;
size_t len = (size_t)info->dlpi_phdr[i].p_memsz;
/* Start at the beginning of the relevant page, */
if (ptr % page) {
len += ptr % page;
ptr -= ptr % page;
}
/* and use full pages. */
if (len % page)
len += page - (len % page);
/* Change protections. Ignore unmapped sections. */
if (mprotect((void *)ptr, len, protect))
if (errno != ENOMEM)
return errno;
}
return 0;
}
int write_protect_text(int protect)
{
int result;
result = dl_iterate_phdr(do_write_protect_text, (void *)(long)protect);
if (result)
errno = result;
return result;
}
Here is an example program you can use to test the above write_protect_text() function:
#define _POSIX_C_SOURCE 200809L
int dump_smaps(void)
{
FILE *in;
char *line = NULL;
size_t size = 0;
in = fopen("/proc/self/smaps", "r");
if (!in)
return errno;
while (getline(&line, &size, in) > (ssize_t)0)
if ((line[0] >= '0' && line[0] <= '9') ||
(line[0] >= 'a' && line[0] <= 'f'))
fputs(line, stdout);
free(line);
if (!feof(in) || ferror(in)) {
fclose(in);
return errno = EIO;
}
if (fclose(in))
return errno = EIO;
return 0;
}
int main(void)
{
printf("Initial mappings:\n");
dump_smaps();
if (write_protect_text(0)) {
fprintf(stderr, "Cannot disable write protection on text sections: %s.\n", strerror(errno));
return EXIT_FAILURE;
}
printf("\nMappings with write protect disabled:\n");
dump_smaps();
if (write_protect_text(1)) {
fprintf(stderr, "Cannot enable write protection on text sections: %s.\n", strerror(errno));
return EXIT_FAILURE;
}
printf("\nMappings with write protect enabled:\n");
dump_smaps();
return EXIT_SUCCESS;
}
The example program dumps /proc/self/smaps before and after changing the text section write protection, showing that it indeed does enable/disable write protectio on all text sections (program code). It does not try to alter write protect on dynamically-loaded libraries. This was tested to work on x86-64 using Ubuntu 3.8.0-35-generic kernel.
If you just want to have one executable with a writable .text, you can just link with -N
At least for me, binutils 2.22 , ld -N objectfile.o
will produce a binary that i can happily write around in.
Reading gcc pages, you can pass the linker option from gcc by : gcc -XN source
I am testing an assembler I am writing which generates X86 instructions. I would like to do something like this to test whether the instructions work or not.
#include<stdio.h>
unsigned char code[2] = {0xc9, 0xc3};
int main() {
void (*foo)();
foo = &code;
foo();
return 0;
}
However it seems that OS X is preventing this due to DEP. Is there a way to either (a) disable DEP for this program or (b) enter the bytes in another format such that I can jump to them.
If you just need to test, try this instead, it's magic...
const unsigned char code[2] = {0xc9, 0xc3};
^^^^^
The const keyword causes the compiler to place it in the const section (warning! this is an implementation detail!), which is in the same segment as the text section. The entire segment should be executable. It is probably more portable to do it this way:
__attribute__((section("text"))
const unsigned char code[2] = {0xc9, 0xc3};
And you can always do it in an assembly file,
.text
.globl code
code:
.byte 0xc9
.byte 0xc3
However: If you want to change the code at runtime, you need to use mprotect. By default, there are no mappings in memory with both write and execute permissions.
Here is an example:
#include <stdlib.h>
#include <sys/mman.h>
#include <err.h>
#include <stdint.h>
int main(int argc, char *argv[])
{
unsigned char *p = malloc(4);
int r;
// This is x86_64 code
p[0] = 0x8d;
p[1] = 0x47;
p[2] = 0x01;
p[3] = 0xc3;
// This is hackish, and in production you should do better.
// Casting 4095 to uintptr_t is actually necessary on 64-bit.
r = mprotect((void *) ((uintptr_t) p & ~(uintptr_t) 4095), 4096,
PROT_READ | PROT_WRITE | PROT_EXEC);
if (r)
err(1, "mprotect");
// f(x) = x + 1
int (*f)(int) = (int (*)(int)) p;
return f(1);
}
The mprotect specification states that its behavior is undefined if the memory was not originally mapped with mmap, but you're testing, not shipping, so just know that it works just fine on OS X because the OS X malloc uses mmap behind the scenes (exclusively, I think).
Don't know about your DEP on OSX, but another thing you could do would be to malloc() the memory you write the code to and then jump into this malloc'ed area. At least on Linux this memory would not be exec-protected (and in fact that's how a JIT usually does the trick).