in context of a protokoll I get messages in AMF Format.
The AMF Object Type "Number" is defined as
number-type = number-marker DOUBLE
The data following a Number type marker is always an 8 byte IEEE-754 double [...] in network byte order.
The following Examples are captured using Wireshark:
Hex: 40 00 00 00 00 00 00 00
Number: 2
Hex: 40 08 00 00 00 00 00 00
Number: 3
Hex: 3f f0 00 00 00 00 00 00
Number: 1
I tried to treat these as doube, long long and int64_t but none of these Types seems to use the correct order/format.
The implementation needs to be in C so I cant use any Librarys (The are none as it seems)
What would be the correct approach?
Likely your platform supports 8-byte IEEE-754 doubles but requires them to be in little-endian format. Your examples are in big-endian format. If you store them in an aligned array of unsigned characters from last to first and cast the pointer to a double *, you should get the right value.
Why stripped binary shows _cxa_finalize instead of libc_start_main?
I am trying to locate and disassemble main() in a very simple C program on Linux (Ubuntu). The binary is stripped. Below you can see disassembly (not stripped) vs disassembly (stripped) of the same instructions.
Question: what is _cxa_finalize in the stripped version? Why is libc_start_main is replaced by _cxa_finalize?
Not stripped:
106d: 48 8d 3d c1 00 00 00 lea rdi,[rip+0xc1] # 1135 <main>
1074: ff 15 66 2f 00 00 call QWORD PTR [rip+0x2f66] # 3fe0 <__libc_start_main#GLIBC_2.2.5>
Stripped:
106d: 48 8d 3d c1 00 00 00 lea rdi,[rip+0xc1] # 1135 <__cxa_finalize#plt+0xf5>
1074: ff 15 66 2f 00 00 call QWORD PTR [rip+0x2f66] # 3fe0 <__cxa_finalize#plt+0x2fa0>
It's not __cxa_finalize. It's __cxa_finalize#plt+0xf5 and __cxa_finalize#plt+0x2fa0 (notice the significant offsets). The disassembler has no information about the symbol main or __libc_start_main because you removed the symbol table, but for technical reasons it is still aware of the symbols assocated with PLT thunks (because they're needed for binding at dynamic linking time, and the disassembler probably falls back to using that information when it lacks s symbol table). In general, the disassembler works backward from an address until it finds an address named by a symbol, and assumes (wrongly, here) that the address being disassembled is part of that function.
Through the IDA disassembler I've reached this address:
0010FD74 00 00 00 00 00 00 03 00 00 00 00 00 82 03 80 02
Now I need, given the address to get particular bytes; for example the 7th position where there is "03".
I've tried using C language to do this:
char *dummycharacter;
*dummycharacter = *(char*)0x10FD74;
Now if I try to access 7th value with this:
dummycharacter[6]
I don't get 0x03…where am I going wrong?
You're trying to assign the value dummycharacter points to (which is pretty much nowhere, since it's not initialized). Try dummycharacter = (char*)0x10FD74;.
I just had an exam in my class today --- reading C code and input, and the required answer was what will appear on the screen if the program actually runs. One of the questions declared a[4][4] as a global variable and at a point of that program, it tries to access a[27][27], so I answered something like "Accessing an array outside its bounds is an undefined behavior" but the teacher said that a[27][27] will have a value of 0.
Afterwards, I tried some code to check whether "all uninitialized golbal variable is set to 0" is true or not. Well, it seems to be true.
So now my question:
Seems like some extra memory had been cleared and reserved for the code to run. How much memory is reserved? Why does a compiler reserve more memory than it should, and what is it for?
Will a[27][27] be 0 for all environment?
Edit :
In that code, a[4][4] is the only global variable declared and there are some more local ones in main().
I tried that code again in DevC++. All of them is 0. But that is not true in VSE, in which most value are 0 but some have a random value as Vyktor has pointed out.
You were right: it is undefined behavior and you cannot count it always producing 0.
As for why you are seeing zero in this case: modern operating systems allocate memory to processes in relatively coarse-grained chunks called pages that are much larger than individual variables (at least 4KB on x86). When you have a single global variable, it will be located somewhere on a page. Assuming a is of type int[][] and ints are four bytes on your system, a[27][27] will be located about 500 bytes from the beginning of a. So as long as a is near the beginning of the page, accessing a[27][27] will be backed by actual memory and reading it won't cause a page fault / access violation.
Of course, you cannot count on this. If, for example, a is preceded by nearly 4KB of other global variables then a[27][27] will not be backed by memory and your process will crash when you try to read it.
Even if the process does not crash, you cannot count on getting the value 0. If you have a very simple program on a modern multi-user operating system that does nothing but allocate this variable and print that value, you probably will see 0. Operating systems set memory contents to some benign value (usually all zeros) when handing over memory to a process so that sensitive data from one process or user cannot leak to another.
However, there is no general guarantee that arbitrary memory you read will be zero. You could run your program on a platform where memory isn't initialized on allocation, and you would see whatever value happened to be there from its last use.
Also, if a is followed by enough other global variables that are initialized to non-zero values then accessing a[27][27] would show you whatever value happens to be there.
Accessing an array out of bounds is undefined behavior, which means the results are unpredictable so this result of a[27][27] being 0 is not reliable at all.
clang tell you this very clearly if we use -fsanitize=undefined:
runtime error: index 27 out of bounds for type 'int [4][4]'
Once you have undefined behavior the compiler can really do anything at all, we have even seen examples where gcc has turned a finite loop into an infinite loop based on optimizations around undefined behavior. Both clang and gcc in some circumstances can generate and undefined instruction opcode if it detects undefined behavior.
Why is it undefined behavior, Why is out-of-bounds pointer arithmetic undefined behaviour? provides a good summary of reasons. For example, the resulting pointer may not be a valid address, the pointer could now point outside the assigned memory pages, you could be working with memory mapped hardware instead of RAM etc...
Most likely the segment where static variables are being stored is much larger then the array you are allocating or the segment that you are stomping though just happens to be zeroed out and so you are just lucky in this case but again completely unreliable behavior. Most likely your page size is 4k and access of a[27][27] is within that bound which is probably why you are not seeing a segmentation fault.
What the standard says
The draft C99 standard tell us this is undefined behavior in section 6.5.6 Additive operators which covers pointer arithmetic which is what an array access comes down to. It says:
When an expression that has integer type is added to or subtracted
from a pointer, the result has the type of the pointer operand. If the
pointer operand points to an element of an array object, and the array
is large enough, the result points to an element offset from the
original element such that the difference of the subscripts of the
resulting and original array elements equals the integer expression.
[...]
If both the pointer operand and the result point to elements of the
same array object, or one past the last element of the array object,
the evaluation shall not produce an overflow; otherwise, the behavior
is undefined. If the result points one past the last element of the
array object, it shall not be used as the operand of a unary *
operator that is evaluated.
and the standards definition of undefined behavior tells us that the standard imposes no requirements on the behavior and notes possible behavior is unpredictable:
behavior, upon use of a nonportable or erroneous program construct or
of erroneous data, for which this International Standard imposes no
requirements
NOTE Possible undefined behavior ranges from ignoring the situation
completely with unpredictable results, [...]
Here is the quote from the standard, that specifies what is undefined behavior.
J.2 Undefined behavor
An array subscript is out of range, even if an object is apparently accessible with the
given subscript (as in the lvalue expression a[1][7] given the declaration int
a[4][5]) (6.5.6).
Addition or subtraction of a pointer into, or just beyond, an array object and an
integer type produces a result that points just beyond the array object and is used as
the operand of a unary * operator that is evaluated (6.5.6).
In your case you the array subscript is completely outside of the array. Depending that the value will be zero is completely unreliable.
Furthermore the behavior of entire program is in question.
If just run your code from visual studio 2012 and got result like this (different at each run):
Address of a: 00FB8130
Address of a[4][4]: 00FB8180
Address of a[27][27]: 00FB834C
Value of a[27][27]: 0
Address of a[1000][1000]: 00FBCF50
Value of a[1000][1000]: <<< Unhandled exception at 0x00FB3D8F in GlobalArray.exe:
0xC0000005: Access violation reading location 0x00FBCF50.
When you look at Modules window you see that your application module memory range is 00FA0000-00FBC000. And unless you have CRT Checks turned on nothing will control what do you do inside your memory (as long as you don't violate memory protection).
So you got 0 at a[27][27] purely by chance. When you open memory view from position 00FB8130 (a) you will probably see something like this:
0x00FB8130 08 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0x00FB8140 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0x00FB8150 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0x00FB8160 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0x00FB8170 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0x00FB8180 01 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 ................
0x00FB8190 c0 90 45 00 b0 e9 45 00 00 00 00 00 00 00 00 00 À.E.°éE.........
0x00FB81A0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0x00FB81B0 00 00 00 00 80 5c af 0f 00 00 00 00 00 00 00 00 ....€\¯.........
0x00FB81C0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
..........
0x00FB8330 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0x00FB8340 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ <<<<
0x00FB8350 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
.......... ^^ ^^ ^^ ^^
It's possible that with your compiler you will always get 0 for that code because of how it uses memory, but just few bytes away you can find another variable.
For example with memory shown above a[6][0] points to address 0x00FB8190 which contains integer value of 4559040.
Then get your teacher to explain this one.
I don't know if this will work on your system but playing about with blatting memory AFTER the array a with non-zero'd bytes gives a different result for a[27][27].
On my system, when I printed contents of a[27][27] it was 0xFFFFFFFF. ie -1 converted to unsigned is all bits set in twos complement.
#include <stdio.h>
#include <string.h>
#define printer(expr) { printf(#expr" = %u\n", expr); }
unsigned int d[8096];
int a[4][4]; /* assuming an int is 4 bytes, next 4 x 4 x 4 bytes will be initialised to zero */
unsigned int b[8096];
unsigned int c[8096];
int main() {
/* make sure next bytes do not contain zero'd bytes */
memset(b, -1, 8096*4);
memset(c, -1, 8096*4);
memset(d, -1, 8096*4);
/* lets check normal access */
printer(a[0][0]);
printer(a[3][3]);
/* Now we disrepect the machine - undefined behaviour shall result */
printer(a[27][27]);
return 0;
}
This is my output:
a[0][0] = 0
a[3][3] = 0
a[27][27] = 4294967295
I saw in comments about viewing memory in Visual Studio. Easiest way is to add a break-point somewhere in your code (to halt execution) then go into Debug... windows... Memory menu, select eg Memory 1. You then find the memory address of your array a. In my case address was 0x0130EFC0. so you enter 0x0130EFC0 in the address fiend and press Enter. This shows the memory at that location.
Eg in my case.
0x0130EFC0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ..................................
0x0130EFE2 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ff ff ff ff ..............................ÿÿÿÿ
0x0130F004 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ
0x0130F026 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ
0x0130F048 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ
The zeros are of the course the array a, which has a byte size of 4 x 4 x sizeof an int (4 in my case) = 64 bytes. The bytes from address 0x0130EFC0 are 0xFF each (from b,c, or d contents).
Note that:
0x130EFC0 + 64 = 0x130EFC0 + 0x40 = 130F000
which is that the start of all those ff bytes you see. Probably array b.
For common compilers, accessing an array beyond its bounds can give predictable results only in very special cases, and you should not rely on that. Example :
int a[4][4];
int b[4][4];
Provided there are no alignment problem, and you ask neither aggressive optimisation nor sanitization checks, a[6][1] should in reality be b[2][1]. But please never do that in production code !
On a particular system, your teacher may be correct -- that may be how your particular compiler and operating system would behave.
On a generic system (i.e. without "insider" knowledge) then your answer is correct: this is UB.
First of all C language have not boundary check. In effect it have no check at all on almost everything. This is the joy and the doom of C.
Now going back to the issue, if you overflow the memory doesn't mean that you trigger a segfault.
Lets have a closer look to how it works.
When you start a program, or enter a subroutine the processor saves on the stack the address to which return when function ends.
The stack has been initialized from OS during process memory allocation, and got a range of legal memory where you can read or write as you like, not only store return addresses.
The common practice used by compilers to create local (automatic) variables is to reserve some space on the stack, and use that space for variables. Look following well known 32 bits assembler sequence, named prologue, that you'll find on any function enter:
push ebp ;save register on the stack
mov ebp,esp ;get actual stack address
sub esp,4 ;displace the stack of 4 bytes that will be used to store a 4 chars array
considering that stack grows in the reverse direction of data, the layout of memory is:
0x0.....1C [Parameters (if any)] ;former function
0x0.....18 [Return Address]
0x0.....14 EBP
0x0.....10 0x0......x ;Local DWORD parameter
0x0.....0C [Parameters (if any)] ;our function
0x0.....08 [Return Address]
0x0.....04 EBP
0x0.....00 0, 'c', 'b', 'a' ;our string of 3 chars plus final nul
This is known as stack frame.
Now consider the string of four bytes starting at 0x0....0 and ending at 0x....3. If we write more than 3 chars in the array we will go replacing sequentially: the saved copy of EBP, the return address, parameters, local variables of previous function then its EBP, return address, etc.
The most scenographic effect we get is that, on function return, the CPU try to jump back to a wrong address generating a segfault. Same behaviour can be achieved if one of local variables are pointers, in this case we will try to read, or write, to wrong locations triggering again the segfault.
When segfault could not happen:
when the bloated variable is not on the stack, or you have so many local variables that you overwrite them without touching the return address (and they are not pointers).
Another case is that the processor reserves a guard space between local variables and return address, in this case the buffer overflow doesn't reach the address.
Another possibility is accessing array elements randomly, in this case an oversized array can exceed stack space and overflow on other data, but luckily we mdon't touch those elements that are mapped where is saved the return address (everythibng can happen...).
When we can have segfault bloating variables that are not on stack?
When overflowing array bound or pointers.
I hope these are useful info...
I have a C header file like this:
#define NAME_LEN 8
#define DEV_MAX 4
typedef struct __device
{
int iDevID;
int iDevSN;
}DEVICE;
typedef struct __person
{
int iID;
char acName[NAME_LEN];
DEVICE aDevices[DEV_MAX];
}PERSON;
and a binary data file maybe like this:
0000000 01 00 08 00 4a 61 63 6b 00 00 00 00 0a 00 00 00
0000020 11 11 11 11 0b 00 00 00 22 22 22 22 0c 00 00 00
0000040 33 33 33 33 0d 00 00 00 44 44 44 44
All that what I need is to visulized data representation with field names using the C header file above....
It'll be better like this...
m--iID : 0x80001
m--acName : Jack
m--aDevices[]
|--aDevices[0]
|--|--iDevID : 0xa
|--|--iDevSN : 0x11111111
|--aDevices[1]
|--|--iDevID : 0xb
|--|--iDevSN : 0x22222222
|--aDevices[2]
|--|--iDevID : 0xc
|--|--iDevSN : 0x33333333
|--aDevices[3]
|--|--iDevID : 0xd
|--|--iDevSN : 0x44444444
or other structured data .. xml / python pickle / json strings / whatever
Of course, the header file which I faced is far more complicated, there will be a msgtype and a msglenth field in the data, so I can find out which is the correct structure and how long is it.
How badly do you need that?
A possible solution might be to make a GCC plugin or a MELT extension (MELT is a domain specific language to extend GCC), but to do that you'll need to understand in some details the internal representation of GCC (notably Tree, and perhaps Gimple), and that will take you some time (days, not hours).
If your declarations are simpler, perhaps consider using SWIG (or maybe the RPCXDR parser), but that supposes that you are able to change or simplify them.
If the binary format were identical to the memory layout of your structure, you could just cast it, no parsing required (with some caveats). However, that evidently isn't what you mean, since your hex dump and sample output don't match that interpretation.
You'll need to actually explain your format though: as described below, it isn't obvious.
You seem to have fixed-length 4-octet integers in little-endian order, OK.
If I assume variable length strings with a nul-terminator, 4a 61 63 6b 00 = acName:"Jack" and 0a 00 00 00 = iDevID:0x0a looks ok, but there is a 3-octet sequence between them I don't know the meaning of.
Or is Jack not nul-terminated, in which case it's fixed at 4 characters long and not the 8 you defined for NAME_LEN? That would make 00 6f 70 65 another 4-byte integer, but I still don't know what it means.
...