I am new to binutils development. I am trying to add a new custom instruction that takes two operands (size, base virtual address) . I am using binutils 2.25.The opcode is 3 byte long and I am running it on x86 machine.Here is what I did :
Used i386-opc.tbl to add an entry as follows.
enclsecreate, 2, 0x0f01cf, None, 3, Cpu386, No_bSuf|No_wSuf|No_lSuf|No_sSuf|No_qSuf|No_ldSuf, {Reg64, Reg64}
My understanding,The second operand states the number of operands, followed by the opcode, followed by None, followed by number of bytes in the opcode.
The use i386-gen :
./i386-gen --srcdir=
which creates i386-tbl.h
i386-gen was not built, i built using make i386-gen and then ran the above step.
To enable the use of disassembler, we need to update i386-dis.c.We need to add an entry to an table.I am lost at this point as to which table I need to add as there are so many of them and I dont understand the format of them.
It would be great if someone could guide me through further steps I need to take or point me to some documentations that contains the necessary information.Looking forward to your kind help.
Thanks
You could have showed the actual encoding for this instruction as it is not yet in the official intel instruction set reference (january 2015 version).
I find it strange that you say it has 2 operands, because I don't see a place for encoding them (maybe they are implicit). So I'll just assume no operands for the following.
The comment at the top of i386-dis.c says:
/* The main tables describing the instructions is essentially a copy
of the "Opcode Map" chapter (Appendix A) of the Intel 80386
Programmers Manual. Usually, there is a capital letter, followed
by a small letter. The capital letter tell the addressing mode,
and the small letter tells about the operand size. Refer to the
Intel manual for details. */
The applicable table of the manual for your opcode is Table A-6. Opcode Extensions for One- and Two-byte Opcodes by Group Number. If you have an updated copy, you should find your instruction in the 0F 01 11B row, column 001 with low 3 bits of (111). This is of course the breakdown of the CF.
The first thing for binutils is the 0F 01 group and the 001 column. This means you have to edit the table RM_0F01_REG_1. That lists the 8 possible instructions in order of their low bits. My copy currently has monitor and mwait there, for 000 and 001 respectively. You might have others too. Since the low 3 bits of our new instruction are 111 which is 7 in decimal, it has to go in the last slot in that table. Pad the table with Bad_Opcode as necessary, then insert the new entry.
If you need special decoding of the operands, you can add a new function to handle it. Use an existing one (e.g. OP_Monitor) as template. By the way, it's also an easy way to locate the required table: just look for an existing instruction that is in the same encoding group as your new instruction.
Disassembly of section .text:
0000000000000000 <.text>:
0: 0f 01 cf enclsecreate
Yay, success!
Related
I want to make a disassembly of the m68k compiled binary myself and make an emulator.
I've disassembled the binary file into a text file with a thirdparty tool m68k-linux-gnu-objdump -b binary -m m68k:68000 ... to have a better vision of what is going on in the binary
Here I have an instruction:
0604 0320 addib #32,%d4
From this table I see that addi function has the next binary scheme:
0000 011 0 <S> <M> <Xn>
and my instruction has representation of:
0000 011 0 00 000 100
Which means I have addi operation with size (8 bits), addressing mode "data register" and the register is encoded to be D4.
Ok, addib with destination %d4 but what does this data column on the right side mean?
I see that the second word (2 bytes of data) in the disassembly is 0x0320 where the last 4 bits 0x20 actually my #32 literal in decimal. But what is this 0x03 in the middle? I've seen some other addi instructions in the disassembly and everywhere there was a 4 bits of something in the middle and the last 4 bits were my number in hex.
I'm probably not taking the last column of the table into account "data" but I failed to understand how to interpret it.
For the example above the table says, data type - "any type" + immediate mode but what is this "any type".
The size of addi instruction said to be any b/w/l in the second (green) column of the table. Are these three things like blue data's first sub-column(B,W,/), green size column (B/W/L), and pink sector of the scheme (00 - B, 01 - W, 10 - L) related? I'm completely confused
And the problem I don't understand the boundaries of the instructions. I've seen some instructions that were maximum 16 bits long (as shown in general schema for each operation) but there are "brief extension words" and "full extension words", what the book says about them I can't get completely. The only thing I probably understood is that the first 16 bits of the opcode is "Single Effective Address Operation Word" and that is.
This is my first approach in trying to understand such a low level of programming
Do what the CPU does with the first byte of the immediate data word of a byte size instruction: Ignore it.
By encoding the two size bits as "00", you told the CPU that you want to add an 8-bit immediate value to the byte-size part of d4 - That means, the upper byte of the immediate data word is not used, but still the 68000 will only read instructions word-wise. Thus, the upper part of this data word is simply "don't care" - You can put anything in there without changing the effect of the instruction, because the CPU won't use it. (Thus, the actual value "3" you see there in your case is irrelevant and probably just some random value left over from the assembler)
If you encode the instruction as ".w" (that is, you want to do a 16-bit add), the upper byte of the data word becomes relevant. If you encode the very same instruction as .l (32-bit add), the assembler will add yet another word to the instruction and put the 32-bit immediate in those 2 words.
For the sake of specifics, let's consider GCC compiler, the latest version.
Consider the instruction int i = 7;.
In assembly it will be something like
MOV 7, R1
This will insert the value seven to register R1. The exact instruction may not be important here.
In my understanding, now the compiler will convert the MOV instruction to processor specific OPCODE. Then it will allocate a (possibly virtual) register. Then the constant value 7 needs to go in the register.
My question:
How does the 7 is actually converted to binary?
Does the compiler actually repeatedly divide by 2 to get the binary representation? (May be afterwards it will convert to HEX, but let's remain on the binary step).
Or, considering that the 7 is written as a character in a text file, is there a clever look up table based technique to convert any string (representing a number) to a binary value?
If the current GCC compiler uses built in function to convert a string 7 to a binary 0111, then how did the first compiler convert a text based string to a binary value?
Thank you.
How does the 7 is actually converted to binary?
First of all, there's a distinction between the binary base 2 number format and what professional programmers call "a binary executable", meaning generated machine code and most often expressed in hex for convenience. Addressing the latter meaning:
Disassemble with binaries (for example at https://godbolt.org/) and see for yourself
int main (void)
{
int i = 7;
return i;
}
Does indeed get translated to something like
mov eax,0x7
ret
Translated to binary op codes:
B8 07 00 00 00
C3
Where B8 = mov eax, B9 = mov ecx and so on. The 7 is translated into 07 00 00 00 since mov expects 4 bytes and this is a little endian CPU.
And this is the point where the compiler/linker stops caring. The code was generated according to the CPU's ABI (Application Binary Interface) and how to deal with this machine code from here on is up to the CPU.
As for how this makes it into the hardware in the actual form of base 2 binary... it's already in that form. Everything we see in a PC is a translated convenience for the human users, who have an easier time reading decimal or hex than raw binary.
If the current GCC compiler uses built in function to convert a string 7 to a binary 0111, then how did the first compiler convert a text based string to a binary value? This is egg chicken problem but to simply put these compilers are created step by step and at some point the compiler is written in its language such that c compiler is written by c etc.
Before to answer your question we should define what we mean by "compilation" or what compiler does. to simply put this compilation is a pipeline. Takes your high level code does some operations and generates an assembly code(specific to machine) and machine defined assembler takes your assembly code and converts it into a binary object file.
At the compiler level all they do is to create corresponding assembly format in a text file.
and assembler is another program that takes this text file and converts it into "binary" format. Assembler can be also written by c language here we also need a mapping i.e movl->(0000110101110...) but this one is binary not ascii. and we need to write this binary into a file as-is.
Converting numbers into binary format is also redundant because numbers are already in binary form when they are loaded into memory.
the question is how they are converted/placed in to memory is a problem of the loader program of the operating system which exceeds my knowledge.
May I get an explanation about what opcode sequences are and how to find them in PE32 files?
I am trying to extract them from PE32 files.
what opcode sequences are
A CPU instruction is composed from 1 to multiple bytes, each of of those bytes have different meanings.
An opcode (operation code) is the part of an instruction that defines the behavior of the instruction itself (as in, this instruction is an 'ADD', or an 'XOR', a NOP, etc.).
For x86 / x64 CPUs (IA-32; IA-32e in Intel linguo) an instruction is composed of at least an opcode (1 to 3 bytes), but can comes with multiple other bytes (various prefixes, ModR/M, SIB, Disp. and Imm.) depending on its encoding:
Opcode is often synonym with "instruction" (since the opcode defines the behavior of the instruction); therefore when you have multiple instructions you then have an opcode sequence (which is a bit of a misnomer since it's really a instruction sequence, unless all instructions in the sequence are only composed of opcodes).
how to find them in PE32 files?
As instructions can be multiple bytes long, you can't just start at a random location in the .text section (which, for a PE file, contains the executable code of the program). There's a specific location in the PE file - called the "entry point" - which defines the start of the program.
The entry point for a PE File is given by the AddressOfEntryPoint member of the IMAGE_OPTIONAL_HEADER structure (parts of the PE header structures). Note that this member is an RVA, not a "full" VA.
From there you know you are at the start of an instruction. You can start disassembling / counting instructions from this point, following the encoding rules for instructions (these rules are explained to great length in the Intel and AMD manuals).
Most instruction are "fall-through", which means that once an instruction has executed, the next to execute is the following one (this seems obvious, but!). The trick is when there's a non-fall-through instruction, you must know what this instruction does to continue your disassembling (e.g. it might jump somewhere, go to a specific handler, etc.)
Use the radare 2 library, it can extract opcode sequences very quickly.
I'm working on the specification of ARMV6M and the instruction's T2 encoding is as below.
0 1 0 0 0 1 | 0 0 | DN | Rm | Rdn
DN is always used as a prefix for Rdn register file position and I couldn't understand why it's not just put in the Rdn.
Many of the thumb instructions, esp ones that are ALU operations have the three bit register specifications encoded in the lower 6 bits, three bits per indicating r0-r7. this specific add instruction allows for operations on low and high registers r0-r15, so the other two bits need a home they happened to put one in bit 6 that goes with 5:3 so the other went above that.
So perhaps they were thinking of saving a few gates or readability or some other reason that cant be answered here at SO. so instead of using 7:4 and 3:0 like they would in a full sized arm instruction for this one special one off instruction, they put the upper bits in 7:6 an even better question is why didn't they put the left right the same as the other two why isn't it [7,5,4,3] [6,2,1,0] instead of [6,5,4,3] and [7,2,1,0]? IMO that would have helped readability esp if you started off on the arm thumb docs that were only in print (paper) originally where H1/H2 seemed swapped.
In the pseudo code they show (DN:Rdn) and talk about a four bit number and then Rm being a 4 bit number, so that indicates what the older docs did in a different way.
I suspect they used the n at the end as lower bits and the capital N as larger bits and yes it would have read better as RdN instead of DN. or Rd3 would have been even better than that.
Instruction sets can to some extent do whatever they want, ARM is no different here the designers way back when thumb started chose what they chose. Arbitrary decision, you will be lucky to find anyone here that was in the room at the time, I have seen these things be mistakes too, in the meeting one thing is decided, the implementation by someone is backward, but by the time that gets around to the room to much investment has been made in testing, etc, to just leave it. Possible that/those engineers are already retired, or got golden parachutes a while ago, maybe you will get lucky.
Also understand that it is not uncommon that the documentation folks are often a separate department, so this could have been a game time decision (or typo) by an individual technical writer, and later determined not to change the docs down the road.
Don't read anything magical into this or some industry nomenclature, that is a bad habit to have anyway, what matters is that you understand what the bits do not how they are labelled.
I want self-education purpose implement a simple virtual machine for a dynamic language, prefer in C. Something like the Lua VM, or Parrot, or Python VM, but simpler. Are there any good resources/tutorials on achieving this, apart from looking at code and design documentations of the existing VMs?
Edit: why close vote? I don't understand - is this not programming. Please comment if there is specific problem with my question.
I assume you want a virtual machine rather than a mere interpreter. I think they are two points on a continuum. An interpreter works on something close to the original representation of the program. A VM works on more primitive (and self-contained) instructions. This means you need a compilation stage to translate the one to the other. I don't know if you want to work on that first or if you even have an input syntax in mind yet.
For a dynamic language, you want somewhere that stores data (as key/value pairs) and some operations that act on it. The VM maintains the store. The program running on it is a sequence of instructions (including control flow). You need to define the set of instructions. I'd suggest a simple set to start with, like:
basic arithmetic operations, including arithmetic comparisons, accessing the store
basic control flow
built-in print
You may want to use a stack-based computation approach to arithmetic, as many VMs do. There isn't yet much dynamic in the above. To get to that we want two things: the ability to compute the names of variables at runtime (this just means string operations), and some treatment of code as data. This might be as simple as allowing function references.
Input to the VM would ideally be in bytecode. If you haven't got a compiler yet this could be generated from a basic assembler (which could be part of the VM).
The VM itself consists of the loop:
1. Look at the bytecode instruction pointed to by the instruction pointer.
2. Execute the instruction:
* If it's an arithmetic instruction, update the store accordingly.
* If it's control flow, perform the test (if there is one) and set the instruction pointer.
* If it's print, print a value from the store.
3. Advance the instruction pointer to the next instruction.
4. Repeat from 1.
Dealing with computed variable names might be tricky: an instruction needs to specify which variables the computed names are in. This could be done by allowing instructions to refer to a pool of string constants provided in the input.
An example program (in assembly and bytecode):
offset bytecode (hex) source
0 01 05 0E // LOAD 5, .x
3 01 03 10 // .l1: LOAD 3, .y
6 02 0E 10 0E // ADD .x, .y, .x
10 03 0E // PRINT .x
12 04 03 // GOTO .l1
14 78 00 // .x: "x"
16 79 00 // .y: "y"
The instruction codes implied are:
"LOAD x, k" (01 x k) Load single byte x as an integer into variable named by string constant at offset k.
"ADD k1, k2, k3" (02 v1 v2 v3) Add two variables named by string constants k1 and k2 and put the sum in variable named by string constant k3.
"PRINT k" (03 k) Print variable named by string constant k.
"GOTO a" (04 a) Go to offset given by byte a.
You need variants for when variables are named by other variables, etc. (and the levels of indirection get tricky to reason about). The assembler looks at the arguments like "ADD .x, .y, .x" and generates the correct bytecode for adding from string constants (and not computed variables).
Well, it's not about implementing a VM in C, but since it was the last tab I had open before I saw this question, I feel like I need point out an article about implementing a QBASIC bytecode compiler and virtual machine in JavaScript using the <canvas> tag for display. It includes all of the source code to get enough of QBASIC implemented to run the "nibbles" game, and is the first in a series of articles on the compiler and bytecode interpreter; this one describes the VM, and he's promising future articles describing the compiler as well.
By the way, I didn't vote to close your question, but the close vote you got was as a duplicate of a question from last year on how to learn about implementing a virtual machine. I think this question (about a tutorial or something relatively simple) is different enough from that one that it should remain open, but you might want to refer to that one for some more advice.
Another resource to look at is the implementation of the Lua language. It is a register-based VM that has a good reputation for performance. The source code is in ANSI C89, and is generally very readable.
As with most high performance scripting languages, the end user sees a readable, high level dynamic language (with features like closures, tail calls, immutable strings, numbers and hash tables as the primary data types, functions as first class values, and more). Source text is compiled to the VM's bytecode for execution by a VM implementation whose outline is pretty much as described by Edmund's answer.
A great deal of effort has gone into keeping the implementation of the VM itself both portable and efficient. If even more performance is needed, a just in time compiler from VM byte code to native instructions exists for 32-bit x86, and is in beta release for 64-bit.
For starting (even if not C, but C++) you could give a look to muParser.
It's a math expression parser that use a simple virtual machine to execute operations. I think that even you need time to understand everything; anyway this code is more simple than a complete VM able to run a real complete program. (By the way, I'm designing a similar lib in C# - it is its early stages but next versions will allow compilation to .NET/VM IL or maybe a new simple VM like muParser).
An other interesting thing is NekoVM (it executes .n bytecode files). It's an open source project written in C and it's main language (.neko) is thought to be generated by source-to-source compiler technology. In the spirit of last topic see Haxe from same author (open source too).
Like you I have also been studying virtual machines and compilers and one good book I can recommend is Compiler Design: Virtual Machines. It describes virtual machines for imperative, functional, logic, and object-oriented languages by giving the instruction set for each VM along with a tutorial on how to compile a higher-level language to that VM. I've only implemented the VM for the imperative language and already it has been a very useful exercise.
If you're just starting out then another resource I can recommend is PL101. It is an interactive set of lessons in JavaScript that guides you through the process of implementing parsers and interpreters for various languages.
I am late for the party, but I would.recommend Game Scripting Mastery, which takes your hand to write a working script language and its VM from zero. And with very little prerequisite.