C binary reaches 1.2G memory limit - c

Question

I am dealing with a C program that reads in from a file with many
lines of about 60 characters each and allocates the string in memory
by requesting more memory as it reads the file in. After each malloc
request, it checks with a function OOM() if the request for more
memory was successful.
I have tested the program with an increasingly larger input file, and
OOM() reports an "Out of memory" more or less when the memory
usage reaches 1.2G when looking at the top command while the program
is running. This is on a 64bit linux machine with plenty more memory
available. Output from file /my/binary/program:
ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18, not stripped
My question is: why is it reaching the 1.2G limit? I remember my
sysadmin used to say that certain binaries would only be able to use
up to 1.2G, which coincidentally is what I am seeing here.
When I run qsub the same execution on a node of the same 64bit Linux
SGE grid reserving 50GB of memory, it reports that also goes "Out of
memory" and the following SGE log memory footprint:
Max vmem = 2.313G
Any ideas why the program is reaching this memory limit? Are there any
compilation flags I should be aware of which could cause/solve this?
Find below the relevant flags in the current Makefile:
CC = gcc
CFLAGS = -Wall -O3 -funroll-loops -DNDEBUG -fomit-frame-pointer -std=gnu99 -msse2 -Wno-unused-function -Wno-unused-result
CFLAGSSFMT = -msse2 -DHAVE_SSE2 -O9 -finline-functions -fomit-frame-pointer \
-DNDEBUG -fno-strict-aliasing --param max-inline-insns-single=1800 -std=c99
LD = ld
LDFLAGS = -lm -lc -lblas -llapack
INCFLAGS =
DEFINES = -D_GNU_SOURCE -DUSE_BLAS
Some of the relevant code belo:w
in mystring.h:
#ifndef _MYSTRING_H_
#define _MYSTRING_H_
struct __mystring_struct {
char * string;
int len, maxlen;
};
typedef struct __mystring_struct * Mystring;
#define Mystring_size sizeof(struct __mystring_struct)
Mystring new_mystring (const int len);
void free_mystring (Mystring string);
void append_char_to_mystring ( const char c, Mystring string);
char * cstring_of_mystring(const Mystring string);
Mystring mystring_of_cstring (const char * str);
#endif
in mystring.c:
#include <string.h>
#include "mystring.h"
#define OOM(A) { if (NULL==(A) ){fputs("Out of memory\n",stderr); exit(EXIT_FAILURE);} }
static void check_is_mystring (const Mystring string);
static void double_length_of_mystring ( Mystring string);
Later on:
static void double_length_of_mystring (Mystring string){
char * new_mem;
check_is_mystring(string);
new_mem = malloc(string->maxlen * 2 * sizeof(char)); OOM(new_mem);
memcpy (new_mem,string->string,string->len * sizeof(char));
free(string->string);
string->string = new_mem;
string->maxlen *= 2;
check_is_mystring (string);
}

Answer 1

It seems you use int to keep size of your string. In GCC (and in most other PC platform compilers) this type is 32b even on 64b platform. You should use size_t instead.
The mechanism of the failed allocation is as follows:
1.2 GB ~= 1288490189
2 * 1.2 GB ~= 2576980378, which is over 2^31 (2147483648), after overflow you get -1717986918 in 2nd complement arithmetics
When calling malloc, -1717986918 is sign extended to a 64 b, and then casted to unsigned 64b, which gives you 2^64 - 1717986918, which is only marginally less than 2^64, and definitely more than the memory you have in the system.