While working on my compiler I got this error:
Program received signal SIGSEGV, Segmentation fault.
__memcpy_sse2_unaligned () at ../sysdeps/x86_64/multiarch/memcpy-sse2-unaligned.S:33
How do I get details of what went wrong here? I know from the backtrace it's a memcpy line that causes it, but how do I see how the memory is aligned? And how do I know how it should be aligned?
The project is a compiler with an LLVM back-end using the Zend/PHP runtime with the OCaml garbage collector, so there's is a lot of things that can go wrong.
I suspect this line being part of the problem:
zend_string *str = (zend_string *)caml_alloc(ZEND_MM_ALIGNED_SIZE(_STR_HEADER_SIZE + len + 1), 0);
where caml_alloc were pemalloc in the Zend source-code.
The segfault happens when doing 10'000 string concatenations. This is the output from valgrind:
==7501== Invalid read of size 8
==7501== at 0x4C2F790: memcpy##GLIBC_2.14 (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==7501== by 0x4D7E58: subsetphp_concat_function (bindings.c:160)
==7501== by 0x4D7F52: foo (llvm_test.s:21)
==7501== by 0x4D7FA9: main (llvm_test.s:60)
==7501== Address 0x61db938 is 2,660,600 bytes inside a block of size 3,936,288 free'd
==7501== at 0x4C2BDEC: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==7501== by 0x4C2627: do_compaction (in /home/olle/kod/subsetphp/test)
==7501== by 0x4C2735: caml_compact_heap (in /home/olle/kod/subsetphp/test)
==7501== by 0x4D08DF: caml_major_collection_slice (in /home/olle/kod/subsetphp/test)
==7501== by 0x4D2DCF: caml_minor_collection (in /home/olle/kod/subsetphp/test)
==7501== by 0x4D2FBC: caml_check_urgent_gc (in /home/olle/kod/subsetphp/test)
==7501== by 0x4D7C45: subsetphp_string_alloc (bindings.c:90)
==7501== by 0x4D7CEE: subsetphp_string_init (bindings.c:122)
==7501== by 0x4D7DEA: subsetphp_concat_function (bindings.c:149)
==7501== by 0x4D7F52: foo (llvm_test.s:21)
==7501== by 0x4D7FA9: main (llvm_test.s:60)
Any tips appreciated.
Edit:
extern value subsetphp_concat_function(value v1, value v2)
{
CAMLparam2(v1, v2);
zend_string *str1 = Zend_string_val(v1);
zend_string *str2 = Zend_string_val(v2);
size_t str1_len = str1->len;
size_t str2_len = str2->len;
size_t result_len = str1_len + str2_len;
value result = subsetphp_string_init("", result_len, 1);
zend_string *zend_result = Zend_string_val(result);
if (str1_len > SIZE_MAX - str2_len) {
zend_error_noreturn(E_ERROR, "String size overflow");
}
memcpy(zend_result->val, str1->val, str1_len); // This is line 160
memcpy(zend_result->val + str1_len, str2->val, str2_len);
zend_result->len = result_len;
zend_result->val[result_len] = '\0';
CAMLreturn(result);
}
Edit 2:
Since valgrind gives me this line
Address 0x61db938 is 2,660,600 bytes inside a block of size 3,936,288 free'd
I guess I'm trying to copy something that has already been freed, meaning that I don't tell the OCaml GC correctly when something is no longer referenced.
This errors tells you that something bad happen during memcpy, probably something like a null pointer or error in the sizes.
Don't bother with __memcpy_sse2_unaligned, it is an implementation detail of memcpy. memcpy has a lot of different implementation optimized for the different cases and dispatch dynamically to the most efficient one given the context. That one seems to be used when sse2 instructions are available and pointers are not alligned to 16 bytes boundaries (sse2 instructions cannot load unaligned values), which is probably done by copying one byte at a time until a 16 byte boundary is reached then switching to the fast path.
As for the OCaml gc specific details linked with LLVM, you need to be quite carefull to how you handle heap pointers. As you don't tell whether you are using the gcroot mechanism or the new statepoints, I will suppose you are using gcroot.
Since the OCaml gc is a moving collector (moving from minor heap to major heap, and moving during compaction) every allocation can potentially invalidate a pointer. That means that it is usualy unsafe to factorise field access to heap allocated values. For instance this is unsafe:
v = field(0, x)
r = function_call(...)
w = field(0, v)
the function call could do some allocations that could trigger a compaction.
v = field(0, x)
r = function_call(...)
v' = field(0, x)
w = field(0, v')
By the way, I'm not even certain that the gcroot mechanism can correctly handle moving gc (that llvm doesn't optimize things it shouldn"t).
So that usualy means that it's not a good idea to use gcroot with OCaml's GC. The new way is better for that kind of GC, but you still need to be carefull not to access pointer across function calls or allocations.
So your error may be something linked to that kind of problem: the pointer was valid at some point, then a value was moved during compaction that resulted in some gc page being unused, hence freed.
Related
So I'm building a virtual machine, and trying to make it as cross platform as possible, and suddenly encountering a strange error. There is a let instruction for my machine, which allocates memory for a variable in the memory of the machine and assign that variable with a value. In short, the let function calls getAddress to get the address of the variable. getAddress checks if the variable is already defined, and returns the address. If the variable is not defined, getAddress calls memallocate to allocate memory for the variable, and returns the address. Here is the definition of the functions :
static uint16_t memallocate(Machine *m, char *symbol){
uint16_t allocationAddress = getFirstFree(*m);
SymbolTable *newSymbol = (SymbolTable *)malloc(sizeof(SymbolTable));
newSymbol->symbolName = strdup(symbol);
newSymbol->next = NULL;
newSymbol->mema = allocationAddress;
if(m->symbolTable==NULL){
m->symbolTable = newSymbol;
}
else{
SymbolTable *temp = m->symbolTable;
while(temp->next!=NULL)
temp = temp->next;
temp->next = newSymbol;
}
m->memory[allocationAddress].acquired = 1;
m->memory[allocationAddress].data.value = 0;
m->occupiedAddress++;
return allocationAddress;
}
uint16_t getAddress(Machine *m, char *symbol){
SymbolTable *table = m->symbolTable;
while(table!=NULL){
if(strcmp(symbol, table->symbolName)==0){
return table->mema;
}
table = table->next;
}
uint16_t address = memallocate(m, symbol); // Here is the segfault happening
return address;
}
This code compiles and runs pretty well on Linux, but on Windows I'm getting a segfault on the memallocate call. Since memallocate is directly passed the arguments of getAddress, and the arguments both being a pointer, they shouldn't change. But while debugging through CLion, I'm seeing gibberish arguments to the memallocate call, which is indicating some kind of stack violation(may be). Again, it is ONLY happening in Windows. Can anybody tell me what is going wrong with my code?
Full code for the project can be found at GitHub.
I took your code and run it on linux through valgrind:
==13768== Conditional jump or move depends on uninitialised value(s)
==13768== at 0x109ABE: getAddress (in /home/vonaka/VirtualMachine/machine)
==13768== by 0x10B714: let (in /home/vonaka/VirtualMachine/machine)
==13768== by 0x109425: run (in /home/vonaka/VirtualMachine/machine)
==13768== by 0x109F64: main (in /home/vonaka/VirtualMachine/machine)
==13768== Uninitialised value was created by a heap allocation
==13768== at 0x4C2BE7F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd
==13768== by 0x109C2F: main (in /home/vonaka/VirtualMachine/machine)
==13768==
So (luckily for us) it's not a Windows specific problem. The trick is that on the first call of getAddress (when m->symbolTable is NULL) you call getFirstFree(*m) at the beginning of memallocate, but look at this function:
static uint16_t getFirstFree(Machine m) {
uint16_t add = 0;
while(m.memory[add].acquired)
add++;
return add;
}
m.memory[i].acquired for i between 0 and number_of_instructions_in_your_input_file - 1are all equal to 1 as you initialize them in writeInstruction, but m.memory[number_of_instructions_in_your_input_file].acquired is not initialized yet.
So something like this will resolve your problem:
void writeInstruction(Machine *m, uint16_t add, Instruction ins) {
m->memory[add].acquired = 1;
m->memory[add].type = INSTRUCTION;
m->memory[add].data.instruction = ins;
m->occupiedAddress++;
if(add + 1 < NUM_MEM)
m->memory[add + 1].acquired = 0;
}
Or maybe this is more elegant (if it's works):
static uint16_t getFirstFree(Machine m) {
uint16_t add = 0;
while (m.memory[add].acquired && add < m.occupiedAddress)
add++;
return add;
}
Edit:
First of all about your comment:
By default, the members of the structure is initialised as 0
It's just not true!
Now about why you have segfault without malloc and how it's connected with valgrind's warning.
You have variable m of type Machine and some other variables in the stack, m contains Cell memory[NUM_MEM] and there is acquired in each Cell (which are not initialized!). Your input file contains let's say 88 instructions, so first 88 acquired will be correctly initialized after 88 calls of writeInstruction. Then program start to execute your instructions by calling some functions including memallocate and getFirstFree. In this loop:
while(m.memory[add].acquired)
add++;
for any add m.memory[add].acquired very likely can be different from 0, so once add is equal to NUM_MEM you have segfault.
Why it's not happening with malloc? Simply because you are lucky (but it's not a good luck), your heap is 'cleaner' than stack. Why it's happening only in Windows? Because this time you were not so lucky (I don't have segfault even in Windows).
I got some problems when I call json_decref()
#include <jansson.h>
#include <stdio.h>
void main()
{
json_t *aa, *bb, *cc, *dd;
aa = json_load_file ("/home/cuihaikuo/demo.json", JSON_STRICT, NULL);
bb = json_array_get (aa, 0);
if (bb != NULL)
json_decref (bb);
if (aa != NULL)
json_decref (aa);
}
I use vargrind running the program, it says Invalid write of size 8
==2641== Invalid write of size 8
==2641== at 0x4E4201A: json_delete (in /usr/lib/x86_64-linux-gnu/libjansson.so.4.7.0)
==2641== by 0x4006C3: json_decref (in /home/cuihaikuo/chk/test/a)
==2641== by 0x400721: main (in /home/cuihaikuo/chk/test/a)
==2641== Address 0x5412438 is 8 bytes inside a block of size 48 free'd
==2641== at 0x4C2EDEB: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==2641== by 0x4006C3: json_decref (in /home/cuihaikuo/chk/test/a)
==2641== by 0x40070E: main (in /home/cuihaikuo/chk/test/a)
if I comment out json_decref (aa);, it will lead a memory leak, how to explain it and what should I do?
json_array_get() returns a borrowed reference. You must not call json_decref() on the values that are returned as borrowed references, unless you manually call json_incref() to increase the reference count. In your minimalistic example, this is not required, so remove the json_decref (bb); and the code should work as expected.
However, if you need to use bb after you have released aa, then call json_incref (bb); after bb = json_array_get (aa, 0); and later, when you are done using bb, you can call json_decref (bb);
Also, even though json_array_get() doesn't mind being passed a NULL value, it would be nice to check the value of aa before using it, so readers of your code doesn't have to double check the manual to see if it is OK not to check aa before using it in json_array_get().
I have a very large (~1E9) array of objects that I malloc, realloc, and free on iterations of a single thread program.
Specifically,
//(Individual *ind)
//malloc, old implementation
ind->obj = (double *)malloc(sizeof(double)*acb->in.nobj);
//new implementation
ind->obj = (double *)a_allocate(acb->in.nobj*sizeof(double));
void *a_allocate (int siz){
void *buf;
buf = calloc(1,siz);
acb->totmemMalloc+=siz;
if (buf==NULL){
a_throw2("a_allocate...failed to allocate buf...<%d>",siz);
}
return buf;
}
...
//realloc
ind->obj = (double *)a_realloc(ind->obj, acb->in.nobj*sizeof(double));
void *a_realloc (void *bufIn,int siz)
{
void *buf = bufIn;
if (buf==NULL){
a_throw2("a_realloc called with null bufIn...");
}
buf = realloc(buf,siz);
return buf;
}
...
//deallocate
free(ind->obj);
The other three dozen properties are processed similarly.
However, every few test runs, the code fails a heap validation on the deallocation of only this object property (the free() statement). At the time of failure, the ind->obj property is not null and has some valid value.
Is there any obvious problem with what I'm doing?
I'm very new to C and am not entirely sure I'm perform the memory operations correctly.
Thanks!
EDIT: using _CRTLDBG_REPORT_FLAG
HEAP[DEMO.exe]: Heap block at 010172B0 modified at 010172E4 past requested size of 2c
Heap validation is a delayed metric. The Visual Studio debug heap can be used (debug build) with more frequent checks Microsoft : Debug Heap flags.
Alternative using application verifier and turning on heap checking, will help find the point which is causing this.
+-----+----------+-----+ +----+-------------+-----+
| chk | memory |chk2 | | chk| different m | chk2|
+-----+----------+-----+ +----+-------------+-----+
When the system allocates memory, it puts meta- information about the memory before the returned pointer (or maybe after). When these memory pieces get overwritten, then that causes the heap failure.
This may be the memory you are freeing, or the memory which was directly before hand.
Edit - to address comments
A message such as "HEAP[DEMO.exe]: Heap block at 010172B0 modified at 010172E4 past requested size of 2c"
Implies that the memory at 01017280 wrote beyond the end of the allocated memory.
This could be because the amount malloced/realloced was too small, or an error in your loops.
+---+-----------------+----+--------------------------+
|chk|d0|d1|d2|d3|d4|d5|chk2| memory |
+---+-----------------+----+--------------------------+
So if you tried to write into d6 above, that would cause 'chk2' to be overwritten, which is being detected. In this case the difference is small - requested size is 0x2c and the difference = E4 - B0 = 0x34
Turning on these debug checks should change your code to be more crashing and predictable. If there is no randomness in your data, then turn off ASLR (only for debugging) and the addresses being used will be predictable, you can put a breakpoint in the malloc/realloc for a given memory address.
This question has been asked several times, but since I think my situation is I think more specific:
I have a C program, which works perfectly on my OSX system (too huge to copy). I already tested it with Valgrind, and I am not missing any frees /mallocs /or writes, all problems are solved 100%.
When I now run the program over ssh on an external sever, when I run with not that many data (see code below, my_length < 1000), it works without any problem. But with a larger dataset, using the Linux terminal I get this error:
*** Error in `./a.out': free(): invalid next size (fast): 0x00000000016b9ed0 ***
======= Backtrace: =========
/lib64/libc.so.6[0x3e50475cff]
/lib64/libc.so.6[0x3e5047cff8]
./a.out[0x41083c]
./a.out[0x402374]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x3e50421d65]
./a.out[0x400e79]
======= Memory map: ========
00400000-00418000 r-xp 00000000 00:4d 89038
[...]
and when I run it on Solaris it complains :
malloc failed
at a line where I allocate a three dimensional array:
int ***A, ***B, ***C;
A = malloc(sizeof(int**)*2); B = malloc(sizeof(int**)*2); C = malloc(sizeof(int**)*2);
int i;
for (i = 0; i < 2; i++) {
A[i] = malloc(sizeof(int*)* my_length);
B[i] = malloc(sizeof(int*)* my_length);
C[i] = malloc(sizeof(int*)* my_length);
for (j = 0; j <= my_length2; j++) {
A[i][j] = malloc(sizeof(int)* my_length2);
B[i][j] = malloc(sizeof(int)* my_length2);
C[i][j] = malloc(sizeof(int)* my_length2);<== malloc failed here??
}
}
where my_length and my_length2 get really really huge!
I am getting desperate! Does someone have any clue what my problem could be?
There are so many duplicates found for this question that annoyingly, I cannot find the right one for you.
The basic problem is that your program has most definitely written over the memory block tracking information that the malloc/free library uses.
Somewhere in your program is a memory write that is out of bounds.
Ok I found one possible solution, I was increasing my values step by step, and now valgrind reports following:
==3954== Invalid write of size 8
==3954== at 0x344C1B: _platform_memmove$VARIANT$Unknown (in /usr/lib/system/libsystem_platform.dylib)
==3954== by 0x1C4D74: __memcpy_chk (in /usr/lib/system/libsystem_c.dylib)
==3954== by 0x10000B2E4: my_method (delete.c:1461)
==3954== by 0x1000025B3: main (delete.c:365)
==3954== Address 0x1020611a0 is 16 bytes after a block of size 2,096 alloc'd
==3954== at 0x56AA: realloc (vg_replace_malloc.c:698)
==3954== by 0x10000B21E: my_method (delete.c:1458)
==3954== by 0x1000025B3: main (delete.c:365)
And this is the code, because I have no idea why this appears:S
if (temp_length + strlen(new_substring)
> max_seq_lens[i]) {
max_len[i] *= 2;
my_array[i].name = realloc(sizeof(char)* max_seq_lens[i]); <===
}
temp_length += (some_num);
SO here temp_length is saving the current length of my my_array[i].name, I am trying to concatenate a new string (new_substring) and before I concatenate them, I tried to check if the memory is enough, I really don't see my mistake here :S
I am using OpenCV 2.1 with codeblocks (gcc under mingw). Within my code I am trying (for some sane reason) to access the imagedata within IplImage datastructure directly. Kindly refer the code snippet for more details:
int main(void)
{
IplImage* test_image = cvLoadImage("test_image.bmp",CV_LOAD_IMAGE_GRAYSCALE);
int mysize = test_image->height * test_image->widthStep;
char* imagedata_ptr = NULL;
int i = 0;
imagedata_ptr = test_image->imageData;
char* temp_buff = (char *)malloc(sizeof(mysize));
memcpy(temp_buff,imagedata_ptr,mysize);
free(temp_buff);
}
When I run this code it crashes. On running it in the debug mode it generates a SIGTRAP is due to heap corruption. At first I suspected that this might be a compiler related issue and hence tried running the same code in Visual Studio. But it still crashes. Thats the reason I feel it could be an OpenCV related issue.
NOTE: There are no other instances of program open, this is the only code that I am running, no threading etc is being done here.
Awaiting your comments on the same.
Regards,
Saurabh Gandhi
You're not allocating enough memory, this:
char* temp_buff = (char *)malloc(sizeof(mysize))
only allocates sizeof(int) bytes (probably 4) and that's probably a lot less than you need. Then the memcpy right after that will copy test_image->height * test_image->widthStep bytes of data into somewhere that only has space for sizeof(int) bytes and you have now scribbled all over your memory and corrupted your heap.
I'd guess that you really want to say this:
char *temp_buff = malloc(mysize);
And don't cast the return value from malloc, you don't need it and it can hide problems.