I'm attempting to build a control-flow graph of the assembly results that are returned via a call to objdump -d . Currently the best method I've come up with is to put each line of the result into a linked list, and separate out the memory address, opcode, and operands for each line. I'm separating them out by relying on the regular nature of objdump results (the memory address is from character 2 to character 7 in the string that represents each line) .
Once this is done I start the actual CFG instruction. Each node in the CFG holds a starting and ending memory address, a pointer to the previous basic block, and pointers to any child basic blocks. I'm then going through the objdump results and comparing the opcode against an array of all control-flow opcodes in x86_64. If the opcode is a control-flow one, I record the address as the end of the basic block, and depending on the opcode either add two child pointers (conditional opcode) or one (call or return ) .
I'm in the process of implementing this in C, and it seems like it will work but feels very tenuous. Does anyone have any suggestions, or anything that I'm not taking into account?
Thanks for taking the time to read this!
edit:
The idea is to use it to compare stack traces of system calls generated by DynamoRIO against the expected CFG for a target binary, I'm hoping that building it like this will facilitate that. I haven't re-used what's available because A) I hadn't really though about it and B) I need to get the graph into a usable data structure so I can do path comparisons. I'm going to take a look at some of the utilities on the page you lined to, thanks for pointing me in the right direction. Thanks for your comments, I really appreciate it!
You should use an IL that was designed for program analysis. There are a few.
The DynInst project (dyninst.org) has a lifter that can translate from ELF binaries into CFGs for functions/programs (or it did the last time I looked). DynInst is written in C++.
BinNavi uses the ouput from IDA (the Interactive Disassembler) to build an IL out of control flow graphs that IDA identifies. I would also recommend a copy of IDA, it will let you spot check CFGs visually. Once you have a program in BinNavi you can get its IL representation of a function/CFG.
Function pointers are just the start of your troubles for statically identifying the control flow graph. Jump tables (the kinds generated for switch case statements in certain cases, by hand in others) throw a wrench in as well. Every code analysis framework I know of deals with those in a very heuristics-heavy approach. Then you have exceptions and exception handling, and also self-modifying code.
Good luck! You're getting a lot of information out of the DynamoRIO trace already, I suggest you utilize as much information as you can from that trace...
I found your question since I was interested in looking for the same thing.
I found nothing and wrote a simple python script for this and threw it on github:
https://github.com/zestrada/playground/blob/master/objdump_cfg/objdump_to_cfg.py
Note that I have some heuristics to deal with functions that never return, the gcc stack protector on 32bit x86, etc... You may or may not want such things.
I treat indirect calls similar to how you do (basically have a node in the graph that is a source when returning from an indirect).
Hopefully this is helpful for anyone looking to do similar analysis with similar restrictions.
I was also facing a similar issue in the past and wrote asm2cfg tool for this purpose: https://github.com/Kazhuu/asm2cfg. Tool has support for GDB disassembly and objdump inputs and spits out CFG as a dot or pdf.
Hopefully someone finds this helpful!
Related
I am trying to run some MC9S12DP256 example files, but I want to see the code to understand it. Are there any ways to convert a .s19 or .abs file to a C code?
An ".s19" or an ".abs" file contains mainly the machine code of the application. The source code of it is not included, independent of the language used to write it. Even if it were written in assembly language, all symbolic informations and comments are excluded.
However, you can try to de-compile the machine code. This is not a trivial or quick task, you need to know the target really well. I did this with software for other processors, it is feasible for code up to some KB.
These are the steps I recommend:
Get a disassembler and an assembler for the target processor, optimally from the vendor.
Let it disassemble the machine code into assembly source code. You might need to convert the ".s19" file into a binary file, one possible tool for this is "srecord".
Assemble the resulting source code again into ".s19" or ".abs", and make sure that it generates the same contents as your original.
Insert labels for the reset and interrupt entry points. Start at the reset entry point with your analysis.
Read the source code, think about what it does.
You will quickly "dive" into subroutines that execute small functions, like reading ADC or sending data. Place a label and replace the numerical value at the call sites with the label.
Expect sections of (constant) data mixed with executable code.
Repeat often from point 3. If you have a difference, undo your last step and redo it in another way until you produce the same contents.
If you want C source, it is commonly much more difficult. You need a lot of experience how C is compiled into machine code. Be aware that variables or even functions are commonly placed in another sequence than they are declared. If you want to go that route, you usually also have to use the exact version of the compiler used to generate the original machine code.
Be aware that the original might be produced with any other language.
is it possible to get the the path where the program was called from?
I call the program on z/Os like this
call 'MCOE.XXXXXXXX.C.LOAD(args)' 'hi there'
My intention is to get the MCOE.XXXXXXXX.C.LOAD dataset in called program without specifying this path as a parameter.
Thanks!
PetrS
This is a non-trivial exercise, but it can generally be done.
You'd start by using CSVINFO to get some information about your program, and then the trick is to emulate the search z/OS would have done to find your module...private/task library, STEPLIB/JOBLIB, (M)LPA search, LNKLST, etc - once you know the load module name definitely (your "args" program name might be an alias, or something the caller setup with an IDENTIFY macro), you can get a lot of this with BLDL, assuming you know which DCB to use.
Once you figure out the DDNAME and concatenation number (after all, there might be 10 libraries in your STEPLIB!), you'd scan the allocation data structures to get the actual dataset name. Typically, this is done by traversing the data structures in memory (PSATOLD->TCBTIO, then indexing into the TIOT till you find the entry you want...the matching TIOT entry will have a pointer to the JFCB - or a SWA manager token - that you can use to get the JFCB, and the JFCB has the dataset name and all the other details you want).
In the case of a fetch from LNKLST, you have extra work to figure out exactly which dataset in the LNKLST concatenation you were fetched from. Again, possible but it requires a bit of finesse.
If your program happens to be in (M)LPA, I'm not sure you can reliably retrieve the original dataset name it was fetched from - this might be the worst case, although there are no doubt a variety of other potential challenges, such as dealing with UNIX Services executable pathnames.
Good luck if you decide to give it a try!
A debugger gets a line number of an expression and translates it into an program address, what does the implementation look like? I want to implement this in a program I'm writing and the most promising library I've found to accomplish this is libbfd. All I would need is the address of the expression, and I can wait for it with ptrace(2). I can imagine that the debugger looks for the function name from the C file within the executable, but after that I'm lost.
Does anyone know? I don't need a code example, just enough info so that I can get an idea.
And I don't mind architecture-specific answers, the only ones I really care about are Arm and x86-64.
You should take a look at the DWARF2 format to try to understand how the mapping is done. Do consider how DWARF2 is vast and complex. It's not for everyone, but reading about it might satisfy your curiosity faster and more easily than reading the source for GCC/GDB.
It's difficult to tell what is being asked here. This question is ambiguous, vague, incomplete, overly broad, or rhetorical and cannot be reasonably answered in its current form. For help clarifying this question so that it can be reopened, visit the help center.
Closed 10 years ago.
I had my code working earlier in the day on the unix machine, but when compiled under windows it gave me completely strange and incorrect output.
Since our code is going to be marked based on compilation on unix I thought hey that's good enough. But now I just finished refactoring my code (basically just adding comments, getting rid of variables which were never used in the program and getting rid of some functions which I wrote to test the program) and now suddenly my code seems to be giving me the proper output on windows and wrong output on unix.
Note that I have done nothing to modify the functionality of the code.
After spending so many hours working on this banging my head against Seg Fault errors through the week, this last minute bug is going to put it all to waste. What am I supposed to do when the bug is seemingly appearing at random?
Edit: The program is supposed to read a file similar to an html file and print out the tables. I'm loading the data of each individual cell onto a node in a Linked List and then printing out the info based on an algorithm. The output is working fine on windows now but not on unix. I don't even know what part of the code I need to look since I have no idea what's causing this.
Based on the amount of information you supplied (next to none), the best guess is to look for uninitialized variables. That will produce different output on different platforms, and is a common beginner mistake in C.
I suggest you use gdb to debug your code and check where the segmentation fault is arising. That will give you a good hint of were to start looking, even though you don't remember to have done any modification.
There is plenty documentation on the web.
These are the basics:
shell> gdb myprogram
gdb> backtrace #lists the steps until the segmentation fault arises
gdb> select 2 #You can select any step you want (e.g. 2)
gdb> print number #print variables to hack around
There are a lot of features for gdb. I think this will give you a hint quickly.
Don't forget to use a version control system the next time. It's a safe and nice way of having your code organized and clean, and off course!, to avoid these terrible accidents.
(SVN or GIT are cool enough)
Step 1, make a copy of everything.
Copy the entire project somewhere. Make a note of what state the project was in when you made that copy and the date:time. DO NOT edit that copy. You may even make the files unwritable if you want. You need to be able to see what you have changed as well as go back to it. Even though the program does not currently work on Unix, it does work under Windows, so you know that it does have some merit and is close to being useful to turn in. When I get upset at a program I am writing or at the compiler for not understanding it (this happens a lot less now then it did 10 years ago) I tend to lose track of what all I am changing, so changing it back becomes difficult. Using some type of version control (even just keeping extra copies around) will help you to keep track of what you have changes so when you make a mistake you can unmake that mistake pretty easily. Differencing tools, like diff are very helpful when you know how to use them. For right now you might want to try:
diff --minimal --side-by-side --ignore-all-space old_file.c new_file.c | less
Hopefully you are using a diff that supports those options because I think that they may be the most helpful for you in the short time that you have. If you find that you need to fit more on the screen and your terminal window is large you can also add in the --width= command and give it a number of characters in a line on your terminal.
Anyway, make and keep lots of copies of your code until you know that you won't need them anymore (and maybe even then).
If you have graphical access see if kdiff3 is available. It may be easier for you to use quickly. The 3 in its name refers to the ability to compare 3 versions of a file at one time (a common starting point and two edited versions of that file) and is useful, but you can learn about that later. It is perfectly able to compare just two versions of a file and produce decent output.
Step 2 Don't ignore warnings
I suggest that you compile it with the highest warning level possible with your compiler and DO NOT ignore any warnings. If you already have warnings without telling the compiler to issue more warnings then examine those first. Warnings are there for a reason, and only occasionally should you ever encounter code the produces warnings that should just be ignored (and even then I usually add a comment about the expected type of warning and why it is not an error). With gcc you can add the -Wall option to the compile command to issue all warnings.
gcc -Wall my_program.c -o my_program
Some may not make sense to you, but you can at least look at the code and see what might be unclear about it in the vicinity of the warning line.
step 3 Use simple lines of code
Something that will make warnings easier to understand is using very simple to understand lines of code. Trying to fit too much functionality into one line of code makes it so that any warning or error message about that line of code is much more difficult to understand.
step 4 Use temporary variables
Temporary variables don't necessarily mean "my program uses more memory" but they do often mean the compiler gives more meaningful warnings because the data-types of variables in expressions are much clearer.
step 5 Use functions
This is just a continuation of the philosophy from 3 and 4. Functions make things easier to understand. They also make it so that often when you find an error and fix it you don't have to worry about having copies of the erroneous code elsewhere in the program that also needs to be fixed (though you should still search for similar code just to be sure).
step 6 assert
There is a macro (like a function, but not quite) called assert that lives in #include <assert.h> and can help you find all kinds of errors by making your program fail earlier than it otherwise would. This sounds bad, but very often (especially with memory related problems like segmentation faults (SIGSEGV) ) programs are in a fatal state well before they die. Using assert helps you to move their death to an earlier place so that you can see what their fatal mistake was, rather than just seeing the result of it.
assert takes as its parameter a boolean expression -- any comparison, integer, floating point number, or pointer will do. Anything that you could use as a condition in an if or while will do. If this expression is false (0 or NULL) then your program will die right there and on many systems it will give you a helpful error message about where the assertion that killed the program was located and maybe even what the assertion was. There is another helpful thing that this causes which I'll talk about in a little bit, but for now, to use assert you just do:
assert(x < y);
and if x is not less than y the program will abort (actually call the abort function).
This is helpful for things like:
int clear_buffer(char * buffer, unsigned len)
{ /* len should be size_t but I don't want to explain that right now */
assert(buffer);
memset(buffer, 0, len);
}
Step 7, Use a debugger
If you have gdb on your Unix system then GREAT. If not, you probably have some other debugger than you can learn how to use. Many Unix C compilers take the -g option to mean "include debugging symbols", so add that to the other options you are passing to the compiler and recompile your program, and then do:
gdb ./myprogram
Which will print some stuff and then prompt you with:
(gdb)
Then you can set break points and all kinds of good stuff, but since you are in a hurry and getting crashes just do:
(gdb) r
Include any arguments after the r that you would be passing to your program when you normally ran it. gdb will then run your program until something odd happens. The something odd, in this case, should be a SIGSEGV (what UNIXes do to your program when it tries to access memory addresses that it shouldn't). gdb will then prompt you with (gdb) again. You can then do:
(gdb) bt
bt stands for back trace and gdb will print out the call stack, meaning all functions that were called to get to the current function. You should see main near the bottom. Look for the first function near the top that is a function you wrote. This is where you need to start trying to find errors. If the top function on the list is not one of yours then try issuing:
(gdb) up
Which will make it examine the previous function on the call stack. Once in one of your functions say:
(gdb) list
And it will show you some code around the area where things are wrong.
To exit gdb you do:
(gdb) quit
And answer Y if it ask you if you really want to quit.
If you were to use assert and that killed your program then you would not end up with quite as much library stuff on top of the call stack to confuse you.
Sadly 3, 4, and 5 mess up the ability to get good info from diff so I suggest trying to limit your adding of this programming style into places where you are having errors or warnings already (at least for now).
I hope that this helps
First of all, we will need your code to see what's going on. But if what you described is true then it is most likely that your code contains what's called undefined behavior. Undefined behavior can occur due to too many reasons, such as crossing array boundaries, incorrectly deleting pointers etc.etc. So, without code nothing can be said
Run it through valgrind.
I can guarantee you will find your error with valgrind.
If you've got access to a unix or linux machine, you should never release code that you haven't run through valgrind, even if the code works.
With the data you've provided, here is my solution.
Take a break and zone out of the problem domain for a while.
Use a debugger, step through the program, identify where it is segfaulting.
Print data at the point of the segfault and validate it.
That should solve the problem.
Compile your code with all warnings on.
Don't hide warnings with bogus casts, but take them seriously and resolve the real problems.
Use different compilers. On linux clang is a good alternative and gives way more indications than gcc.
I was writing a function to figure out if a given system of linear inequalities has a solution, when all of a sudden it started giving the wrong answers after a seemingly innocuous change.
I undid some changes, re-did them, and then proceeded to fiddle for the next two hours, until I had reduced it to absurdity.
The following, inserted anywhere into the function body, but nowhere else in the program, fixes it:
if(0) {
__asm__("nop\n");
__asm__("nop\n");
__asm__("nop\n");
__asm__("nop\n");
}
It's for a school assignment, so I probably shouldn't post the function on the web, but this is so ridiculous that I don't think any context is going to help you. And all the function does is a bunch of math and looping. It doesn't even touch memory that isn't allocated on the stack.
Please help me make sense of the world! I'm loathe to chalk it up to the GCC, since the first rule of debugging is not to blame the compiler. But heck, I'm about to. I'm running Mac OS 10.5 on a G5 tower, and the compiler in question identifies itself as 'powerpc-apple-darwin9-gcc-4.0.1' but I'm thinking it could be an impostor...
UPDATE: Curiouser and curiouser... I diffed the .s files with nops and without. Not only are there too many differences to check, but with no nops the .s file is 196,620 bytes, and with it's 156,719 bytes. (!)
UPDATE 2: Wow, should have posted the code! I came back to the code today, with fresh eyes, and immediately saw the error. See my sheepish self-answer below.
Most times when you modify the code inconsequentially and it fixes your problem, it's a memory corruption problem of some sort. We may need to see the actual code to do proper analysis, but that would be my first guess, based on the available information.
It's faulty pointer arithmetic, either directly (through a pointer) or indirectly (by going past the end of an array). Check all your arrays. Don't forget that if your array is
int a[4];
then a[4] doesn't exist.
What you're doing is overwriting something on the stack accidentally. The stack contains both locals, parameters, and the return address from your function. You might be damaging the return address in a way that the extra noops cures.
For example, if you have some code that is adding something to the return address, inserting those extra 16 bytes of noops would cure the problem, because instead of returning past the next line of code, you return into the middle of some noops.
One way you might be adding something to the return address is by going past the end of a local array or a parameter, for example
int a[4];
a[4]++;
I came back to this after a few days busy with other things, and figured it out right away. Sorry I didn't post the code sooner, but it was hard coming up with minimal example that displayed the problem.
The root problem was that I left out the return statements in the recursive function. I had:
bool function() {
/* lots of code */
function()
}
When it should have been:
bool function() {
/* lots of code */
return function()
}
This worked because, through the magic of optimization, the right value happened to be in the right register at the right time, and made it to the right place.
The bug was originally introduced when I broke the first call into its own special-cased function. And, at that point, the extra nops were the difference between this first case being inlined directly into the general recursive function.
Then, for reasons that I don't fully understand, inlining this first case led to the right value not being in the right place at the right time, and the function returning junk.
Does it happen in debug and release mode build (with symbols and without)? Does it behave the same way using a debugger? Is the code moultithreaded? Are you compiling with optimizations? Can you try another machine?
Can you confirm that you are indeed getting different executables when you add the if(0) {nops}? I don't see nops on my system.
$ gcc --version
powerpc-apple-darwin9-gcc-4.0.1 (GCC) 4.0.1 (Apple Inc. build 5490)
$ cat nop.c
void foo()
{
if (0) {
__asm__("nop");
__asm__("nop");
__asm__("nop");
__asm__("nop");
}
}
$ gcc nop.c -S -O0 -o -
.
.
_foo:
stmw r30,-8(r1)
stwu r1,-48(r1)
mr r30,r1
lwz r1,0(r1)
lmw r30,-8(r1)
blr
$ gcc nop.c -S -O3 -o -
.
.
_foo:
blr
My guess is stack corruption -- though gcc should optimize anything inside an if(0) out, I would have thought.
You could try sticking a big array on the stack in your function and see if that also fixes it -- that would also implicate stack corruption.
Are you sure you're running what you think you're running? (dumb question, but it happens.)
Looks like you will need to put in some hard work and elbow grease
Your problem sounds similar to something I have debugged in the past where my app was running regular ... when out of nowhere it jumped to a different part of the app and the callstack got completely messed up ( however this was embedded programming )!
It sounds like you are spending your time "thinking" about "what should be happening" ... when you should be "looking" at "what is actually happening". A lot of the times the hardest bugs are things that you would never think "should happen".
I would approach the problem like so:
Break out your favorite debugger
Start stepping through your code and watch the call stack and local variables and look for suspicious activity
Make the system fail
Focus in to where the system is failing
Focus on iterating your code changes:
making code changes that will "make the system fail"
running/debugging and watching
If it runs fine you are looking/trying the wrong thing and you need to try something else. If you make it fail then you have made progress towards finding the bug.
If you don't know where or how the system fails you will not be able to solve the problem.
This will be a good opportunity to build your debugging skills. For more help on building your debugging skills read check out the book "9 rules for debugging".
Here is a poster from the book:
(source: google.com)
Concrete suggestions:
If you think it is the compiler, then run a different platform/OS/compiler.
Once you have ruled out the platform/OS/compiler, then try restructuring the code. Look for the "clever" code parts and see if they are actually doing what the code meant to do... maybe the clever solution wasn't actually clever and is doing something else.
I am the author of "Debugging" so kindly referenced above by Trevor Boyd Smith. He has it right -- the key rules here are #2 Make It Fail (which you seem to be doing okay), and #3 Quit Thinking and Look. The conjectures above are very good (demonstrating mastery of rule #1 -- Understand the System -- in this case the way code size can change a bug). But actually watching it fail with a debugger will show you what's actually happening without guesswork.
Break out that one function into a separate .c file (or .cpp or whatever). Compile just that one file with the nops and without them, to .s files and compare them.
Try an old version of gcc. Go back 5 or 10 years and see if things get stranger.