I need to read a file with some kind of a matrix from CSV file(number of matrix columns and rows may be different every time) using C.
The file will be something like that:
#,#,#,#,#,#,.,#,.,.,.$
#,.,#,.,.,#,.,#,#,#,#$
#,.,#,.,.,.,.,.,.,#,#$
#,.,#,.,.,#,#,#,#,#,#$
#,.,.,#,.,.,.,.,.,.,#$
#,.,.,.,#,.,#,#,.,.,#$
#,.,.,.,.,#,.,.,.,.,#$
#,.,.,.,.,#,.,.,.,.,#$
#,.,.,.,.,.,.,.,.,.,#$
#,#,#,#,#,#,#,#,#,.,#$
I need to read the file and save it to a two-dimensional array to be able to iterate through it and find the path out of the labyrinth using Lee algorithm.
So I want to do somenthing like:
int fd = open (argv[i], O_RDONLY);
while (read(fd, &ch, 1)) {
here should be some for loops to find the number of colums and rows.
}
Unfortunately, I don't know how to do that if heigth and width of the matrix is unknown.
I was trying to do that:
while (read (fd, &ch, 1)) {
for (int i = 0; arr[i] != '\0'; i++) {
for (int j = 0; j != '\n'; j++) {
somehow save the values, number of columns and rows.
}
}
}
However, number of rows could be greater than number of columns.
Any help will be appreciated
If the size isn't known but has to be determined as you parse the file, then a simple but a bit naive idea would be to use a char** rows = malloc(n); where n is a sufficiently large number to cover most normal use-cases. realloc if you go past n.
Then for each row you read, store it inside rows[i] through another malloc followed by strcpy/memcpy.
A smarter version of the same would be to first read the first row, find the row length and then assume that all rows in the file have that size. You can do do a char (*rows)[n] = malloc (n * (row_length+1) ); to allocate a true 2D array. This has advantages over the char**, since you get a proper cache-friendly 2D array with faster access, faster allocation and less heap fragmentation. See Correctly allocating multi-dimensional arrays for details about that.
Another big advantage of the char (*rows)[n] is that if you know n in advance, you can actually read/fread the whole file in one go, which would be a significant performance boost since file I/O will be the bottleneck in this program.
If you don't know n in advance, you would still have to realloc in case you end up reading more than n rows. So a third option would be to use a linked list, which is probably the worst option since it is slow and adds complexity. The only advantage being that a link list allows you to swiftly add/remove rows on the fly.
Related
I have a program that results in an integer array of variable size and I need to write the elements to a text file. The elements need to be on the same line in the file.
This is a minimal example of what I am currently doing. I'm using the approach in this post https://stackoverflow.com/a/30234430/10163981
FILE *file = fopen("file.txt","w");
int nLines = 1000;
char breakstr[]="\n";
for(; ix<N; ix++){
char s[nLines*13];
for(int jx = 0 ; jx<nLines; jx++){
index += sprintf(&s[index],"%03i %03i %03i ", array[ix][jx], array[ix][jx], array[ix][jx]);
// I need jx:th element in repeats of three, and may need to modify it externally for printing
}
fwrite(s,sizeof(char),strlen(s), file);
fwrite(breakstr,sizeof(char),strlen(breakstr), file);
}
fclose(file);
I am formatting the array contents as a string and using fwrite, as this method has been given to me as a requirement. My problem is that this implementation that I am using is way too slow. I have also tried using shorter strings and writing for each iteration, but this even slower. There is not much I can do with regards to the outer ix loop, as the initial value of ix is variable. I included it for the same of completeness.
nLines is expected to reach as high as 10000 at most.
I'm tasked with having 1,000,000 cards in one input file having a market price and then having the same 1,000,000 cards with a higher price in another input file, and I have to compare both to compute the profit.
A nested for loop of:
for(int i = 0; i < marketPriceCards.size(); i++){
for(int j = 0; j < priceListCards.size(); j++){
compute profit
is O(n^2) which is way too long. I was thinking a hash table but how big would I have to make it? A prime number that's higher than 1000000?
In Java the default load factor is 0.75, so you can create your hashtable at the size of:
1.75 * <size of your data>
and that should be a good start.
By the way, you didn't mention which language are you going to use. In case it's Java you should use HashMap - not Hashtable (just FYI).
I don't understand why you wrote a nested loop since it can be done in one loop O(n).
As your data is recorded in a two big files you need to read them and you need to traverse whole of the both file since you need all numbers.
if the records was less than 100,000 I would suggest to load them both in to memory using mopen() however you have two big files and loading them both into memory is not a clever action. So here it is what I think you should do in case you have text files
cardsFile = fopen ("elapsed.dta", "rt");
priceFile = fopen ("elapsed.dta", "rt");
while(fgets(aCardline, 80, cardsFile) != NULL)
{
sscanf (aCardline, "%ld", &aCard);
fgets(aPriceline, 80, priceFile)
sscanf (aCardline, "%ld", &aPrice);
printf ("Card :%s Price :%ld\n", aCard, aPrice,);
}
I think you have to change the methods which return cards and prices
you may use buffers in case you need to elaborate more the data.
I personally like to store this size of data in a database.
Hope this helps you.
I am looking into how to copy a 2D array of variable width for each row into the GPU.
int rows = 1000;
int cols;
int** host_matrix = malloc(sizeof(*int)*rows);
int *d_array;
int *length;
...
Each host_matrix[i] might have a different length, which I know length[i], and there is where the problem starts. I would like to avoid copying dummy data. Is there a better way of doing it?
According to this thread, that won't be a clever way of doing it:
cudaMalloc(d_array, rows*sizeof(int*));
for(int i = 0 ; i < rows ; i++) {
cudaMalloc((void **)&d_array[i], length[i] * sizeof(int));
}
But I cannot think of any other method. Is there any other smarter way of doing it?
Can it be improved using cudaMallocPitch and cudaMemCpy2D ??
The correct way to allocate an array of pointers for the GPU in CUDA is something like this:
int **hd_array, **d_array;
hd_array = (int **)malloc(nrows*sizeof(int*));
cudaMalloc(d_array, nrows*sizeof(int*));
for(int i = 0 ; i < nrows ; i++) {
cudaMalloc((void **)&hd_array[i], length[i] * sizeof(int));
}
cudaMemcpy(d_array, hd_array, nrows*sizeof(int*), cudaMemcpyHostToDevice);
(disclaimer: written in browser, never compiled, never tested, use at own risk)
The idea is that you assemble a copy of the array of device pointers in host memory first, then copy that to the device. For your hypothetical case with 1000 rows, that means 1001 calls to cudaMalloc and then 1001 calls to cudaMemcpy just to set up the device memory allocations and copy data into the device. That is an enormous overhead penalty, and I would counsel against trying it; the performance will be truly terrible.
If you have very jagged data and need to store it on the device, might I suggest taking a cue of the mother of all jagged data problems - large, unstructured sparse matrices - and copy one of the sparse matrix formats for your data instead. Using the classic compressed sparse row format as a model you could do something like this:
int * data, * rows, * lengths;
cudaMalloc(rows, nrows*sizeof(int));
cudaMalloc(lengths, nrows*sizeof(int));
cudaMalloc(data, N*sizeof(int));
In this scheme, store all the data in a single, linear memory allocation data. The ith row of the jagged array starts at data[rows[i]] and each row has a length of length[i]. This means you only need three memory allocation and copy operations to transfer any amount of data to the device, rather than nrows in your current scheme, ie. it reduces the overheads from O(N) to O(1).
I would put all the data into one array. Then compose another array with the row lengths, so that A[0] is the length of row 0 and so on. so A[i] = length[i]
Then you need just to allocate 2 arrays on the card and call memcopy twice.
Of course it's a little bit of extra work, but i think performance wise it will be an improvement (depending of course on how you use the data on the card)
I am struggling to decide between two optimisations for building a numerical solver for the poisson equation.
Essentially, I have a two dimensional array, of which I require n doubles in the first row, n/2 in the second n/4 in the third and so on...
Now my difficulty is deciding whether or not to use a contiguous 2d array grid[m][n], which for a large n would have many unused zeroes but would probably reduce the chance of a cache miss. The other, and more memory efficient method, would be to dynamically allocate an array of pointers to arrays of decreasing size. This is considerably more efficient in terms of memory storage but would it potentially hinder performance?
I don't think I clearly understand the trade-offs in this situation. Could anybody help?
For reference, I made a nice plot of the memory requirements in each case:
There is no hard and fast answer to this one. If your algorithm needs more memory than you expect to be given then you need to find one which is possibly slower but fits within your constraints.
Beyond that, the only option is to implement both and then compare their performance. If saving memory results in a 10% slowdown is that acceptable for your use? If the version using more memory is 50% faster but only runs on the biggest computers will it be used? These are the questions that we have to grapple with in Computer Science. But you can only look at them once you have numbers. Otherwise you are just guessing and a fair amount of the time our intuition when it comes to optimizations are not correct.
Build a custom array that will follow the rules you have set.
The implementation will use a simple 1d contiguous array. You will need a function that will return the start of array given the row. Something like this:
int* Get( int* array , int n , int row ) //might contain logical errors
{
int pos = 0 ;
while( row-- )
{
pos += n ;
n /= 2 ;
}
return array + pos ;
}
Where n is the same n you described and is rounded down on every iteration.
You will have to call this function only once per entire row.
This function will never take more that O(log n) time, but if you want you can replace it with a single expression: http://en.wikipedia.org/wiki/Geometric_series#Formula
You could use a single array and just calculate your offset yourself
size_t get_offset(int n, int row, int column) {
size_t offset = column;
while (row--) {
offset += n;
n << 1;
}
return offset;
}
double * array = calloc(sizeof(double), get_offset(n, 64, 0));
access via
array[get_offset(column, row)]
I'm writing code for a decision tree in C. Right now it gives me the correct result (0% training error, low test error), but it takes a long time to run.
The problem lies in how often I run qsort. My basic algorithm is this:
for every feature
sort that feature column using qsort
remove duplicate feature values in that column
for every unique feature value
split
determine entropy given that split
save the best feature to split + split value
for every training_example
if training_example's value for best feature < best split value, store in Left[]
else store in Right[]
recursively call this function, using only the Left[] training examples
recursively call this function, using only the Right[] training examples
Because the last two lines are iterative calls, and because the tree can extend for dozens and dozens of branches, the number of calls to qsort is huge (especially for my dataset that has > 1000 features).
My idea to reduce the runtime is to create a 2d array (in a separate function) where each column is a sorted feature column. Then, as long as I maintain a vector of row numbers of the training examples in Left[] and Right[] for each recursive call, I can just call this separate function, grab the rows I want in the pre-sorted feature vector, and save the cost of having to qsort each time.
I'm fairly new to C and so I'm not sure how to code this. In MatLab I can just have a global array that any function can change or access, looking for something like that in C.
Global arrays in C are totally possible. There are actually two ways of doing that. In the first case the dimensions of the array are fixed for the application:
#define NROWS 100
#define NCOLS 100
int array[NROWS][NCOLS];
int main(void)
{
int i, j;
for (i = 0; i < NROWS; i++)
for (j = 0; j < NCOLS; j++)
{
array[i][j] = i+j;
}
return 0;
}
In the second example the dimensions may depend on values from the input.
#include <stdlib.h>
int **array;
int main(void)
{
int nrows = 100;
int ncols = 100;
int i, j;
array = malloc(nrows*sizeof(*array));
for (i = 0; i < nrows; i++)
{
array[i] = malloc(ncols*sizeof(*(array[i])));
for (j = 0; j < ncols; j++)
{
array[i][j] = i+j;
}
}
}
Although the access to the arrays in both examples looks deceivingly similar, the implementation of the arrays is quite different. In the first example the array is located in one piece of memory and the strides to access rows is a whole row. In the second example each row access is a pointer to a row, which is one piece of memory. The various rows can however be located in different areas of the memory. In the second example rows might also have a different length. In that case you would need to store the length of each row somewhere too.
I don't fully understand what you are trying to achieve, because I'm not familiar with the terminology of decision tree, feature and the standard approaches to training sets. But you may also want to have a look at other data structures to maintain sorted data:
http://en.wikipedia.org/wiki/Red–black_tree maintains a more or less balanced and sorted tree.
AVL tree a bit slower but more balanced and sorted tree.
Trie a sorted tree on lists of elements.
Hash function to easily map a complex element to an integral value that can be used to sort the elements. Good for finding exact elements, but there is no real order in the elements itself.
P.S1: Coming from Matlab you may want to consider a different language from C to move to. C++ has standard libraries to support above data structures. Java, Python come to mind or even Haskell if you are daring. Pointer handling in C can be quite tedious and error prone.
P.S2: I'm unable to include a - in a URL on StackOverflow. So the Red-black tree links is a bit off and can't be clicked. If someone can edit my post to fix it, then I would appreciate that.