I have the array defined as below
INT32 LUT_OffsetValues[6][12] = {
0,180,360,540,720,900,1080,1260,1440,1620,1800,1980,
2160,2340,2520,2700,2880,3060,3240,3420,3600,3780,3960,4140,
4320,4500,4680,4860,5040,5220,5400,5580,5760,5940,6120,6300,
6480,6660,6840,7020,7200,7380,7560,7740,7920,8100,8280,8460,
8640,8820,9000,9180,9360,9540,9720,9900,10080,10260,10440,10620,
10800,10980,11160,11340,11520,11700,11880,12060,12240,12420,12600,12780
};
int main(int argc,char *argv[])
{
int var_row_index = 4 ;
int var_column_index = 5 ;
int computed_val = 0 ;
FILE *fp = NULL ;
fp = fopen("./LUT_Offset.bin","wb");
if(NULL != fp)
{
fwrite(LUT_OffsetValues,sizeof(INT32),72,fp);
fclose(fp);
}
printf("Size of Array:%d\n",sizeof(LUT_OffsetValues));
//computed_val = LUT_OffsetValues[var_row_index][var_column_index];
return 0;
}
Above is the code snippet with which I have generated the .bin file. Is that the right way of doing it?
No, it is not the right way if you plan to transfer the file to a different machine and read it as you haven't considered the Endianness. Let's say the file is:
Written in little endian machine but read in big endian machine
Written in big endian machine but read in little endian machine
It won't work for none of the cases above.
Out of the order of the bytes signaled by askinoor, that way is not generic because the reader have to now it is an INT32[6][12] when it read it
Why the useless variables var_row_index etc in your program ?
As already mentioned, when serializing data in and out of the CPU it is preferable to force network byte order. This can be done easily using functions like htonl(), which should be available on most platforms (and compile down to nothing on big endian machines).
Here's the doc from Linux:
https://linux.die.net/man/3/htonl
Also, it's not good practice to explicitly code sizes and types into your program.
Use sizeof(array[0][0]) to get the size of the element type of array, then iterate over it and use htonl() to write each element to the file.
Related
I am about to do some data processing in C, and the processing part is working logically, but I am having a strange file problem. I conveniently have 32-bits of numbers to consider, so I need a file of 32-bits of 0s, and then I will change the 0 to 1 if something exists in a finite field.
My question is: What is the best way to make a file with all "0s" in C?
What I am currently doing, seems to make sense but is not working. I currently am doing the following, and it doesn't stop at the 2.4GiB mark. I have no idea what's wrong or if there's a better way.
#include <stdlib.h>
#include <stdio.h>
typedef uint8_t u8;
typedef uint32_t u32;
int main (int argc, char **argv) {
u32 l_counter32 = 0;
u8 l_ubyte = 0;
FILE *f_data;
f_data = fopen("file.data", "wb+");
if (f_data == NULL) {
printf("file error\n");
return(0);
}
for (l_counter32 = 0; l_counter32 <= 0xfffffffe; l_counter32++) {
fwrite(&l_ubyte, sizeof(l_ubyte), 1, f_data);
}
fwrite(&l_ubyte, sizeof(l_ubyte), 1, f_data); //final byte at 0xffffffff
fclose(f_data);
}
I increment my counter in the loop to be 0xFFFFFFFe, so that it doesn't wrap around and run forever.. I haven't waited for it to stop actually, I just keep checking on the disk via ls -alF and when it's larger than 2.4GiB, I stop it. I checked sizeof(l_ubyte), and it is indeed 8-bits.
I feel that I must be missing some mundane detail.
You are counting up to 0xffffffff, which is equal to 4,294,967,295. You want to count up to 0x80000000 for exactly 2 GB of data.
The faster way to create initalize a file with zeroes (alias \0 null bytes) is using truncate()/ftruncate(). See man page here
I have a legacy code and some unformatted data files that it reads, and it worked with gnu-4.1.2. I don't have access to the method that originally generated these data files. When I compile this code with a newer gnu compiler (gnu-4.7.2) and attempt to load the old data files on a different computer, it is having difficulty reading them. I start by opening the file and reading in the first record which consists of three 32-bit integers:
open(unit, file='data.bin', form='unformatted', status='old')
read(unit) x,y,z
I am expecting these three integers here to describe x,y,z spans so that next it can load a 3D matrix of float values with those same dimensions. However, instead it's loading a 0 for the first value, then the next two are offset.
Expecting:
x=26, y=127, z=97 (1A, 7F, 61 in hex)
Loaded:
x=0, y=26, z=127 (0, 1A, 7F in hex)
When I checked the data file in a hex editor, I think I figured out what was happening.
The first record marker in this case has a value of 12 (0C in hex) since it's reading three integers at 4 bytes each. This marker is stored both before and after the record. However, I notice that the 32bits immediately after each record marker is 00000000. So either the record markers are treated as 64bit integers (little-Endian) or there is a 32-bit zero padding after each record marker. Either way, the code generated with the new compiler is reading the record markers as 32-bit integers and not expecting any padding. This effectively intrudes/corrupts the data being read in.
Is there an easy way to fix this non-portable issue? The old and new hardware are 64 bit architecture and so is the executable I compiled. If I try to use the older compiler version again will it solve the problem, or is it hardware dependent? I'd prefer to use the newer compilers because they are more efficient, and I really don't want to edit the source code to open all the files as access='stream' and manually read in a trailing 0 integer after each record marker, both before and after each record.
P.S. I could probably write a C++ code to alter the data files and remove these zero paddings if there is no easier alternative.
See the -frecord-marker= option in the gfortran manual. With -frecord-marker=8 you can read the old style unformatted sequential files produced by older versions of gfortran.
Seeing as how Fortran doesn't have a standardization on this, I opted to convert the data files to a new format that uses 32-bit wide record lengths instead of 64-bit wide. In case anyone needs to do this in the future I've included some Visual C++ code here that worked for me and should be easily modifiable to C or another language. I have also uploaded a Windows executable (fortrec.zip) here.
CFile OldFortFile, OutFile;
const int BUFLEN = 1024*20;
char pbuf[BUFLEN];
int i, iIn, iRecLen, iRecLen2, iLen, iRead, iError = 0;
CString strInDir = "C:\folder\";
CString strIn = "file.dat";
CString strOutDir = "C:\folder\fortnew\"
system("mkdir \"" + strOutDir + "\""); //create a subdir to hold the output files
strIn = strInDir + strIn;
strOut = strOutDir + strIn;
if(OldFortFile.Open(strIn,CFile::modeRead|CFile::typeBinary)) {
if(OutFile.Open(strOut,CFile::modeCreate|CFile::modeWrite|CFile::typeBinary)) {
while(true) {
iRead = OldFortFile.Read(&iRecLen, sizeof(iRecLen)); //Read the record's raw data
if (iRead < sizeof(iRecLen)) //end of file reached
break;
OutFile.Write(&iRecLen, sizeof(iRecLen));//Write the record's raw data
OldFortFile.Read(&iIn, sizeof(iIn));
if (iIn != 0) {//this is the padding we need to ignore, ensure it's always zero
//Padding not found
iError++;
break;
}
i = iRecLen;
while (i > 0) {
iLen = (i > BUFLEN) ? BUFLEN : i;
OldFortFile.Read(&pbuf[0], iLen);
OutFile.Write(&pbuf[0], iLen);
i -= iLen;
}
if (i != 0) { //Buffer length mismatch
iError++;
break;
}
OldFortFile.Read(&iRecLen2, sizeof(iRecLen2));
if (iRecLen != iRecLen2) {//ensure we have reached the end of the record proeprly
//Record length mismatch
iError++;
break;
}
OutFile.Write(&iRecLen2, sizeof(iRecLen));
OldFortFile.Read(&iIn, sizeof(iIn));
if (iIn != 0) {//this is the padding we need to ignore, ensure it's always zero
//Padding not found
break;
}
}
OutFile.Close();
OldFortFile.Close();
}
else { //Could not create the ouput file.
OldFortFile.Close();
return;
}
}
else { //Could not open the input file
}
if (iError == 0)
//File successfully converted
else
//Encountered error
I have to find out the size of a instruction which I have in memory (actually, I have a small code segment in memory and want to get the size of the first instruction).
It took me some time to find libopcodes and libbfd. I red the headers and tried to come up with a simple solution but it seems like I missunderstood something since the program always crashes:
int main(int argc, char **argv) {
disassemble_info *dis = malloc(sizeof(*dis));
assert(dis != NULL);
dis->arch = bfd_arch_i386;
dis->read_memory_func = buffer_read_memory;
dis->buffer_length = 64;
dis->buffer = malloc(dis->buffer_length);
memset(dis->buffer, 0x90, dis->buffer_length);
disassemble_init_for_target(dis);
int instr_size = print_insn_i386(0, dis);
printf("instruction size is %d\n", instr_size);
return 0;
}
The expected result would be an instruction size of 1 (nop).
EDIT:
sorry guys, I'm a stupid person.
memset(dis, 0, sizeof(*dis));
There is some code in the Linux kernel you can steal. It should work well if copied into a user mode program.
Take a look at arch/x86/lib and arch/x86/tools
There's an opcode map file there, and an awk script that reads it to produce a table in a file named innat.c. There are some other files there that use the table to implement a decoder.
It is sufficient to determine instruction sizes.
This assumes you are ok with GPL, of course.
It looks like the disassemble_info data structure requires more initialization than you have provided. From examples I have been studying, the correct way to initialize is to call init_disassemble_info().
See if that helps. Failing that, compile your program with debug info ('-g') and run gdb to diagnose where the crash occurs.
I am trying to make a simple kernel using C. Everything loads and works fine, and I can access the video memory and display characters, but when i try to implement a simple puts function for some reason it doesn't work. I've tried my own code and other's. Also, when I try to use a variable which is declared outside a function it doesn't seem to work. This is my own code:
#define PUTCH(C, X) pos = putc(C, X, pos)
#define PUTSTR(C, X) pos = puts(C, X, pos)
int putc(char c, char color, int spos) {
volatile char *vidmem = (volatile char*)(0xB8000);
if (c == '\n') {
spos += (160-(spos % 160));
} else {
vidmem[spos] = c;
vidmem[spos+1] = color;
spos += 2;
}
return spos;
}
int puts(char* str, char color, int spos) {
while (*str != '\0') {
spos = putc(*str, color, spos);
str++;
}
return spos;
}
int kmain(void) {
int pos = 0;
PUTSTR("Hello, world!", 6);
return 0;
}
The spos (starting position) stuff is because I can't make a global position variable. putc works fine, but puts doesn't. I also tried this:
unsigned int k_printf(char *message, unsigned int line) // the message and then the line #
{
char *vidmem = (char *) 0xb8000;
unsigned int i=0;
i=(line*80*2);
while(*message!=0)
{
if(*message=='\n') // check for a new line
{
line++;
i=(line*80*2);
*message++;
} else {
vidmem[i]=*message;
*message++;
i++;
vidmem[i]=7;
i++;
};
};
return(1);
};
int kmain(void) {
k_printf("Hello, world!", 0);
return 0;
}
Why doesn't this work? I tried using my puts implementation with my native GCC (without the color and spos data and using printf("%c")) and it worked fine.
Since you're having an issue with global variables in general, the problem most likely has to-do with where the linker is placing your "Hello World" string literal in memory. This is due to the fact that string literals are typically stored in a read-only portion of global memory by the linker ... You have not detailed exactly how you are compiling and linking your kernel, so I would attempt something like the following and see if that works:
int kmain(void)
{
char array[] = "Hello World\n";
int pos = 0;
puts(array, 0, pos);
return 0;
}
This will allocate the character array on the stack rather than global memory, and avoid any issues with where the linker decides to place global variables.
In general, when creating a simple kernel, you want to compile and link it as a flat binary with no dependencies on external OS libraries. If you're working with a multiboot compliant boot-loader like GRUB, you may want to look at the bare-bones sample code from the multiboot specification pages.
Since this got references outside of SO, I'll add a universal answer
There are several kernel examples around the internet, and many are in various states of degradation - the Multiboot sample code for instance lacks compilation instructions. If you're looking for a working start, a known good example can be found at http://wiki.osdev.org/Bare_Bones
In the end there are three things that should be properly dealt with:
The bootloader will need to properly load the kernel, and as such they must agree on a certain format. GRUB defines the fairly common standard that is Multiboot, but you can roll your own. It boils down that you need to choose a file format and locations where all the parts of your kernel and related metadata end up in memory before the kernel code will ever get executed. One would typically use the ELF format with multiboot which contains that information in its headers
The compiler must be able to create binary code that is relevant to the platform. A typical PC boots in 16-bit mode after which the BIOS or bootloader might often decide to change it. Typically, if you use GRUB legacy, the Multiboot standard puts you in 32-bit mode by its contract. If you used the default compiler settings on a 64-bit linux, you end up with code for the wrong architecture (which happens to be sufficiently similar that you might get something that looks like the result you want). Compilers also like to rename sections or include platform-specific mechanisms and security features such as stack probing or canaries. Especially compilers on windows tend to inject host-specific code that of course breaks when run without the presence of Windows. The example provided deliberately uses a separate compiler to prevent all sorts of problems in this category.these
The linker must be able to combine the code in ways needed to create output that adheres to the bootloader's contract. A linker has a default way of generating a binary, and typically it's not at all what you want. In pretty much all cases, choosing gnu ld for this task means that you're required to write a linker script that puts all the sections in the places where you want. Omitted sections will result in data going missing, sections at the wrong location might make an image unbootable. Assuming you have gnu ld, you can also use the bundled nm and objdump tools besides your hex editor of choice to tell you where things have appeared in your output binary, and with it, check if you've been following the contract that has been set for you.
Problems of this fundamental type are eventually tracked back to not following one or more of the steps above. Use the reference at the top of this answer and go find the differences.
I want to write a piece of code that changes itself continuously, even if the change is insignificant.
For example maybe something like
for i in 1 to 100, do
begin
x := 200
for j in 200 downto 1, do
begin
do something
end
end
Suppose I want that my code should after first iteration change the line x := 200 to some other line x := 199 and then after next iteration change it to x := 198 and so on.
Is writing such a code possible ? Would I need to use inline assembly for that ?
EDIT :
Here is why I want to do it in C:
This program will be run on an experimental operating system and I can't / don't know how to use programs compiled from other languages. The real reason I need such a code is because this code is being run on a guest operating system on a virtual machine. The hypervisor is a binary translator that is translating chunks of code. The translator does some optimizations. It only translates the chunks of code once. The next time the same chunk is used in the guest, the translator will use the previously translated result. Now, if the code gets modified on the fly, then the translator notices that, and marks its previous translation as stale. Thus forcing a re-translation of the same code. This is what I want to achieve, to force the translator to do many translations. Typically these chunks are instructions between to branch instructions (such as jump instructions). I just think that self modifying code would be fantastic way to achieve this.
You might want to consider writing a virtual machine in C, where you can build your own self-modifying code.
If you wish to write self-modifying executables, much depends on the operating system you are targeting. You might approach your desired solution by modifying the in-memory program image. To do so, you would obtain the in-memory address of your program's code bytes. Then, you might manipulate the operating system protection on this memory range, allowing you to modify the bytes without encountering an Access Violation or '''SIG_SEGV'''. Finally, you would use pointers (perhaps '''unsigned char *''' pointers, possibly '''unsigned long *''' as on RISC machines) to modify the opcodes of the compiled program.
A key point is that you will be modifying machine code of the target architecture. There is no canonical format for C code while it is running -- C is a specification of a textual input file to a compiler.
Sorry, I am answering a bit late, but I think I found exactly what you are looking for : https://shanetully.com/2013/12/writing-a-self-mutating-x86_64-c-program/
In this article, they change the value of a constant by injecting assembly in the stack. Then they execute a shellcode by modifying the memory of a function on the stack.
Below is the first code :
#include <stdio.h>
#include <unistd.h>
#include <errno.h>
#include <string.h>
#include <sys/mman.h>
void foo(void);
int change_page_permissions_of_address(void *addr);
int main(void) {
void *foo_addr = (void*)foo;
// Change the permissions of the page that contains foo() to read, write, and execute
// This assumes that foo() is fully contained by a single page
if(change_page_permissions_of_address(foo_addr) == -1) {
fprintf(stderr, "Error while changing page permissions of foo(): %s\n", strerror(errno));
return 1;
}
// Call the unmodified foo()
puts("Calling foo...");
foo();
// Change the immediate value in the addl instruction in foo() to 42
unsigned char *instruction = (unsigned char*)foo_addr + 18;
*instruction = 0x2A;
// Call the modified foo()
puts("Calling foo...");
foo();
return 0;
}
void foo(void) {
int i=0;
i++;
printf("i: %d\n", i);
}
int change_page_permissions_of_address(void *addr) {
// Move the pointer to the page boundary
int page_size = getpagesize();
addr -= (unsigned long)addr % page_size;
if(mprotect(addr, page_size, PROT_READ | PROT_WRITE | PROT_EXEC) == -1) {
return -1;
}
return 0;
}
It is possible, but it's most probably not portably possible and you may have to contend with read-only memory segments for the running code and other obstacles put in place by your OS.
This would be a good start. Essentially Lisp functionality in C:
http://nakkaya.com/2010/08/24/a-micro-manual-for-lisp-implemented-in-c/
Depending on how much freedom you need, you may be able to accomplish what you want by using function pointers. Using your pseudocode as a jumping-off point, consider the case where we want to modify that variable x in different ways as the loop index i changes. We could do something like this:
#include <stdio.h>
void multiply_x (int * x, int multiplier)
{
*x *= multiplier;
}
void add_to_x (int * x, int increment)
{
*x += increment;
}
int main (void)
{
int x = 0;
int i;
void (*fp)(int *, int);
for (i = 1; i < 6; ++i) {
fp = (i % 2) ? add_to_x : multiply_x;
fp(&x, i);
printf("%d\n", x);
}
return 0;
}
The output, when we compile and run the program, is:
1
2
5
20
25
Obviously, this will only work if you have finite number of things you want to do with x on each run through. In order to make the changes persistent (which is part of what you want from "self-modification"), you would want to make the function-pointer variable either global or static. I'm not sure I really can recommend this approach, because there are often simpler and clearer ways of accomplishing this sort of thing.
A self-interpreting language (not hard-compiled and linked like C) might be better for that. Perl, javascript, PHP have the evil eval() function that might be suited to your purpose. By it, you could have a string of code that you constantly modify and then execute via eval().
The suggestion about implementing LISP in C and then using that is solid, due to portability concerns. But if you really wanted to, this could also be implemented in the other direction on many systems, by loading your program's bytecode into memory and then returning to it.
There's a couple of ways you could attempt to do that. One way is via a buffer overflow exploit. Another would be to use mprotect() to make the code section writable, and then modify compiler-created functions.
Techniques like this are fun for programming challenges and obfuscated competitions, but given how unreadable your code would be combined with the fact you're exploiting what C considers undefined behavior, they're best avoided in production environments.
In standard C11 (read n1570), you cannot write self modifying code (at least without undefined behavior). Conceptually at least, the code segment is read-only.
You might consider extending the code of your program with plugins using your dynamic linker. This require operating system specific functions. On POSIX, use dlopen (and probably dlsym to get newly loaded function pointers). You could then overwrite function pointers with the address of new ones.
Perhaps you could use some JIT-compiling library (like libgccjit or asmjit) to achieve your goals. You'll get fresh function addresses and put them in your function pointers.
Remember that a C compiler can generate code of various size for a given function call or jump, so even overwriting that in a machine specific way is brittle.
My friend and I encountered this problem while working on a game that self-modifies its code. We allow the user to rewrite code snippets in x86 assembly.
This just requires leveraging two libraries -- an assembler, and a disassembler:
FASM assembler: https://github.com/ZenLulz/Fasm.NET
Udis86 disassembler: https://github.com/vmt/udis86
We read instructions using the disassembler, let the user edit them, convert the new instructions to bytes with the assembler, and write them back to memory. The write-back requires using VirtualProtect on windows to change page permissions to allow editing the code. On Unix you have to use mprotect instead.
I posted an article on how we did it, as well as the sample code.
These examples are on Windows using C++, but it should be very easy to make cross-platform and C only.
This is how to do it on windows with c++. You'll have to VirtualAlloc a byte array with read/write protections, copy your code there, and VirtualProtect it with read/execute protections. Here's how you dynamically create a function that does nothing and returns.
#include <cstdio>
#include <Memoryapi.h>
#include <windows.h>
using namespace std;
typedef unsigned char byte;
int main(int argc, char** argv){
byte bytes [] = { 0x48, 0x31, 0xC0, 0x48, 0x83, 0xC0, 0x0F, 0xC3 }; //put code here
//xor %rax, %rax
//add %rax, 15
//ret
int size = sizeof(bytes);
DWORD protect = PAGE_READWRITE;
void* meth = VirtualAlloc(NULL, size, MEM_COMMIT, protect);
byte* write = (byte*) meth;
for(int i = 0; i < size; i++){
write[i] = bytes[i];
}
if(VirtualProtect(meth, size, PAGE_EXECUTE_READ, &protect)){
typedef int (*fptr)();
fptr my_fptr = reinterpret_cast<fptr>(reinterpret_cast<long>(meth));
int number = my_fptr();
for(int i = 0; i < number; i++){
printf("I will say this 15 times!\n");
}
return 0;
} else{
printf("Unable to VirtualProtect code with execute protection!\n");
return 1;
}
}
You assemble the code using this tool.
While "true" self modifying code in C is impossible (the assembly way feels like slight cheat, because at this point, we're writing self modifying code in assembly and not in C, which was the original question), there might be a pure C way to make the similar effect of statements paradoxically not doing what you think are supposed do to. I say paradoxically, because both the ASM self modifying code and the following C snippet might not superficially/intuitively make sense, but are logical if you put intuition aside and do a logical analysis, which is the discrepancy which makes paradox a paradox.
#include <stdio.h>
#include <string.h>
int main()
{
struct Foo
{
char a;
char b[4];
} foo;
foo.a = 42;
strncpy(foo.b, "foo", 3);
printf("foo.a=%i, foo.b=\"%s\"\n", foo.a, foo.b);
*(int*)&foo.a = 1918984746;
printf("foo.a=%i, foo.b=\"%s\"\n", foo.a, foo.b);
return 0;
}
$ gcc -o foo foo.c && ./foo
foo.a=42, foo.b="foo"
foo.a=42, foo.b="bar"
First, we change the value of foo.a and foo.b and print the struct. Then we change only the value of foo.a, but observe the output.