building large strings of unknown length in c - c

Question

I've no doubt there is an answer to this somewhere, I just can't find it.
I have just returned to c after a long break and am very rusty, so please excuse dumb errors. I need to generate a large (maybe equivilent of 10mb) string. I don't know how long it's going to be until it's built.
I tried the following two approaches to test speed:
int main() {
#if 1
size_t message_len = 1; /* + 1 for terminating NULL */
char *buffer = (char*) malloc(message_len);
for (int i = 0; i < 200000; i++)
{
int size = snprintf(NULL, 0, "%d \n", i);
char * a = malloc(size + 1);
sprintf(a, "%d \n", i);
message_len += 1 + strlen(a); /* 1 + for separator ';' */
buffer = (char*) realloc(buffer, message_len);
strncat(buffer, a, message_len);
}
#else
FILE *f = fopen("test", "w");
if (f == NULL) return -1;
for (int i = 0; i < 200000; i++)
{
fprintf(f, "%d \n", i);
}
fclose(f);
FILE *fp = fopen("test", "r");
fseek(fp, 0, SEEK_END);
long fsize = ftell(f);
fseek(fp, 0, SEEK_SET);
char *buffer = malloc(fsize + 1);
fread(buffer, fsize, 1, f);
fclose(fp);
buffer[fsize] = 0;
#endif
char substr[56];
memcpy(substr, buffer, 56);
printf("%s", substr);
return 1;
}
The first solution of concatenating strings each time took 3.8s, the second of writing to a file then reading took 0.02s.
Surely there is a fast way to build a big string in c without resorting to reading and writing to a file? Am I just doing something very inefficient? If not can I write to some kind of file object, then read it at the end and never save it?
In C# you would use a stringbuffer to avoid the slow concatination, what's the equivilent in c?
Thanks in advance.

Answer 1

You are making life pretty rough with these lines:
for (int i = 0; i < 200000; i++)
{
int size = snprintf(NULL, 0, "%d \n", i); // << executed in first loop only
char * a = malloc(size + 1); // allocate enough space for "0 \n" + 1
sprintf(a, "%d \n", i); // may try to squeeze "199999 \n" into a
message_len += 1 + strlen(a); /* 1 + for separator ';' */
buffer = (char*) realloc(buffer, message_len);
strncat(buffer, a, message_len);
}
You compute size and allocate space for a in the first iteration - then proceed to use it in every subsequent iteration (where i gets bigger, and you will in principle exceed the storage allocated for a). If you did this correctly (allocating size for a in each loop) you would have to free in every loop as well, or create a giant memory leak.
The solution, in C, is to pre-allocate plenty of memory - and only reallocate in emergency. If you know "roughly" how big your string will be, allocate all that memory at once; keep track of how big it is, and add more if you run short. At the end you can always "give back what you didn't use". Too many calls to realloc keep moving memory around (since you often don't have enough contiguous memory available where you were). As #Matt clarified in his comment: there is a real risk that every call to realloc moves the entire block of memory - and as the block gets bigger, that becomes a quadratically increasing load on the system. Here is a possible better solution (complete, tested with small N and BLOCK just to show the principle; you will want to use large N (your value of 200000), and larger BLOCK - and get rid of the printf statements that were there to show things are working ):
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <time.h>
#define N 2000000
#define BLOCK 32
int main(void) {
size_t message_len = BLOCK; //
char *buffer = (char*) malloc(message_len);
int bb;
int i, n=0;
char* a = buffer;
clock_t start, stop;
for(bb = 1; bb < 128; bb *= 2) {
int rCount = 0;
start = clock();
for (i = 0; i < N; i++)
{
a = buffer + n;
n += sprintf(a, "%d \n", i);
if ((message_len - n) < BLOCK*bb) {
rCount++;
message_len += BLOCK*bb;
//printf("increasing buffer\n");
//printf("increased buffer to %ld\n", (long int)message_len);
buffer = realloc(buffer, message_len);
}
}
stop = clock();
printf("\nat the end, buffer length is %d; rCount = %d\n", strlen(buffer), rCount);
// buffer = realloc(buffer, strlen(buffer+1));
//printf("buffer is now: \n%s\n", buffer);
printf("time taken with blocksize = %d: %.1f ms\n", BLOCK*bb, (stop - start) * 1000.0 / CLOCKS_PER_SEC);
}
}
You will want to use a fairly large value for BLOCK - this will limit the number of calls to realloc. I would use something like 100000; you get rid of the space at the end anyway.
EDIT I modified the code I had posted to allow timing of the loop - increasing N to 2 million to get "reasonable times". I also minimized the initial memory allocation (to force a lot of calls to realloc and fixed a bug (when realloc had to move memory, a was no longer pointing to an offset in buffer. That is fixed now by keeping track of the string length so far in n.
This is pretty fast - 450 ms for the smallest block, dropping to 350 ms for larger blocks (2 million numbers). That is comparable (within the resolution of my measurement) to your file read/write operation. But yes - file I/O streaming and associated memory management are highly optimized...

Answer 2

I have left out some details, but my approach is generally like this
create a structure like this one
typedef struct {
char *curr ;
char *start ;
char *end ;
} VBUF ;
write some functions along these lines:
void vbuf_alloc(VBUF *v,int n)
{
v->start = malloc(n) ;
v->end = v->start + n ;
v->curr = v->start ;
}
int vbuf_add(VBUF *v,char *s,int length)
{
if (v->end - v->curr < length) {
vbuf_realloc(v,(v->end - v->start) * 2) ;
}
memcpy(v->curr,s,length) ;
v->curr += length ;
return length ;
}
int vbuf_adds(VBUF *v,char *s)
{
return vbud_add(v,s,strlen(s)) ;
}
You can extend this suite of functions as much as you like.

Answer 3

c has no objects so there's no equivalent to the C# stringbuffer (though in C++ you would use std::string).
You would get a performance boost by not calling realloc on every append, and never calling malloc the way you are.
You can avoid your malloc completely by simply declaring a char[] large enough to print the largest int into; this would avoid the snprintf too, and the size is fairly small.
Instead of constantly calling realloc, you should grow your buffer by some reasonable size… say 4kb (a nice size to correspond with page size), and only grow it again when it comes close to being exhausted (that is, when its current usage is less than the array size you use above from the top).

Answer 4

I suggest instead of realloc on every successive string, attempt to intelligently realloc ahead of time if the length is too short. In other words, avoid realloc whenever possible.
A naive implementation in pseudocode might be something like
Initialize an int/long to "written so far"
Initialize an int/long to remember "buffer size"
Alloc memory for a string up to the "buffer size"
Read in the next chunk into a temporary buffer
Get the "chunk size" from the temporary buffer
If "written so far" + "chunk size" > "buffer size"
Reallocate the chunk to be much bigger (double "buffer size"?)
Set the new "buffer size"
Copy the data from the temporary buffer to "buffer address" + "written so far" + 1
Set "written so far" to "written so far" + "chunk size"
I just threw this together, so there may be indexing errors, but you get the idea: only allocate and copy when you have to, instead of every time through the loop.