I am writing a program that goes through an array of ints and calculates stdev to identify outliers in the data. From here, I would like to create a new array with the identified outliers removed in order to recalculate the avg and stdev. Is there a way that I can do this?
There is a pretty simple solution to the problem that involves switching your mindset in the if statement (which isn't actually in a for loop it seems... might want to fix that).
float dataMinusOutliers[n];
int indexTracker = 0;
for (i=0; i<n; i++) {
if (data[i] >= (-2*stdevfinal) && data[i] <= (2*stdevfinal)) {
dataMinusOutliers[indexTracker] = data[i];
indexTracker += 1;
}
}
Note that this isn't particularly scalable and that the dataMinusOutliers array is going to potentially have quite a few unused indices. You can always use indexTracker - 1 to note how large the array actually is though, and create yet another array into which you copy the important values in dataMinusOutliers. Is there likely a more elegant solution? Yes. Does this work given your requirements though? Yup.
Related
my code works as far as i can tell..
I was wondering if it can be done in a better way (better time complexity) and what is the time complexity of my code as im not really sure how to caculate it.
cant change the current array in the question but if there is a faster way to do it by removal i would also like to know, thanks a lot.
int i = 1, j = 0, count = 1;
int arrNew[SIZE] = { NULL };
arrNew[0] = arr1[0];
while(i<size){
if (arr1[i] == arrNew[j]) { // if the element of arr1 is already added, resets j for next iteration and moves to the next element.
j = 0;
i++;
}
else {
if (j == count - 1) { // checks if we reached the end of arrNew and adds missing element.
arrNew[count] = arr1[i];
j = 0;
count++; // this variable makes sure we check only the assigned elements of arrNew.
i++;
}
else // if j < count -1 we didnt finish checking all of arrNew.
j++;
}
}
I was wondering if it can be done in a better way (better time complexity)
It's a little hard to tell what's going on at first, but it looks like you're basically using one loop to do two jobs. You're looping on i to step through the original array, but also using j to scan through the new array for each new element. Effectively, you've got nested loops that both potentially have the same size, so you've got O(n2) complexity.
I'd suggest rewriting your code so that the two loops are explicit. You're not saving any time by making one loop do double duty, and if you come back to this code a month from now you're going to waste a bunch of time trying to remember how it works. Make your code obvious — it's as much about communicating with your future self or your coworkers as with the compiler.
Can you improve on that O(n2) complexity? Yes, definitely. One way is to sort the array, so that duplicate values end up being adjacent to each other in the array. It's then easy to just not copy any values that are the same as the preceding value. I know you can't modify the original array, but you can copy the whole thing, sort it, and then copy that array while removing dupes. That'd give you O(n log n) complexity (if you choose an efficient sorting algorithm). In fact, you could speed that up a bit by combining the sorting and copying -- but you'd still end up with O(n log n) complexity. Another way is to use a hash table: check to see whether the value exists in the table, tossing it if it does, or adding it to the table and copying to the new array if it doesn't. That'd be close to O(n).
I am trying to compare linear memory access to random memory access. I am traversing an array in the order of its indices to log performance of linear memory access. However to log memory's performance with random memory access I want to traverse my array randomly i.e arr[8], arr[17], arr[34], arr[2]...
Can I use pointer chasing to achieve this while ensuring that no index are accessed twice? Is pointer chasing most optimal approach in this case?
If your goal is to show that sequential access is faster than non-sequential access, simply pointer chasing the latter is not a good way to demonstrate that. You would be comparing access via a single pointer plus simple offset against deterrencing one or more pointers before offsetting.
To use pointer chasing, you'd have to apply it to both cases. Here's an example:
int arr[n], i;
int *unshuffled[n];
int *shuffled[n];
for(i = 0; i < n; i++) {
unshuffled[i] = arr + i;
}
/* I'll let you figure out how to randomize your indices */
shuffle(unshuffled, shuffled)
/* Do toning on these two loops */
for(i = 0; i < n; i++) {
do_stuff(*unshuffled[i]);
}
for(i = 0; i < n; i++) {
do_stuff(*shuffled[i]);
}
It you want to time the direct access better though, you could construct some simple formula for advancing the index instead of randomizing the access completely:
for(i = 0; i < n; i++) {
do_stuff(arr[i]);
}
for(i = 0; i < n; i++) {
do_stuff(arr[i / 2 + (i % 2) * (n / 2)]);
}
This will only work properly for even n as shown, but it illustrates the idea. You could go so far as to compensate for the extra flops in computing the index within do_stuff.
Probably the most apples-to-apples test would be to literally access the indices you want, without loops or additional computations:
do_stuff(arr[0]);
do_stuff(arr[1]);
do_stuff(arr[2]);
...
do_stuff(arr[123]);
do_stuff(arr[17]);
do_stuff(arr[566]);
...
Since I'd imagine you'd want to test with large arrays, you can write a program to generate the actual test code for you, and possibly compile and run the result.
I can tell you that for arrays in C the access time is constant regardless of the index being accessed. There will be no difference between accessing them randomly or sequentially other than the fact that randomizing will in itself introduce additional computations.
But, to really answer your question, you would probably be best off to build some kind of lookup array and shuffle it a few times and use that array to get the next index. Obviously, you would be accessing two arrays, one sequentially and another randomly, by doing so, thus making the exercise pretty much useless.
So what I have is an array that's size is decided by me and then the elements in the array are randomly generated. It's supposed to take an integer array,its size, and an integer number
and find how many times the number is present in the array and return that count at the end.I keep trying stuff and nothing seems to be getting me anywhere close to an answer. I was just trying to see if someone could point me in the right direction on where to start
count_numbers(int array[], int size, int z)
Hhave you tried running a loop through the array and trying a match expression to the array value in another loop. This seems like a logic question rather than actual code related. Maybe a search around the internet looking at how to count in arrays could help you.
This should point you in the right direction...
for (int i = 0; i < arraySize; i++) {
if (array[i] == z /*z being your search value**/) {
you may have to alter this a little
//dosomething
// e.g. increment a count here
}
else
do-nothing essentially.
There is a method for checking array size - so don't worry about defining it's size. have a look at the java method for this and use it.
Hope this helps
I'm currently working on a program to solve the red/blue computation; program is written in C.
Description of the problem is here : http://www.cs.utah.edu/~mhall/cs4961f10/CS4961-L9.pdf
tl;dr you have a grid of colors (red/blue/white), first red cells move to the right according to certain rules, then blue cells move down according to other rules.
I've got my program working and giving correct output, and I'm now trying to see if I can't speed it up at all.
Using Intel's VTune Amplifier (this is for a parallel programming course, and we're doing pthreads in visual studio with parallel studio integrated), I've identified that the biggest hotspot in my code is when moving blue cells.
Implementation details: grid is stored as a dynamically allocated int **, set up this way
globalBoard = malloc(sizeof(int *) * size);
for (i = 0; i < size; i++)
{
globalBoard[i] = malloc(sizeof(int) * size);
for (j = 0; j < size; j++)
globalBoard[i][j] = rand() % 3;
}
After some research, I believe the cause of the hotspot (almost 4 times as much CPU time as moving red cells) is cache misses when traversing column by column.
I understand that under the hood, this grid will be stored as a 1d array, so when I move red cells to the right and go row by row, I'm most often checking contiguous values, so the CPU doesn't need to load new values into the cache as often, whereas going column by column results in jumping around through the array by amounts that only increase as the size of the board does.
All that being said, I want this particular section to go faster. Here's the code as it stands now :
void blueStep(int col)
{
int i;
int local[size];
for (i = 0; i < size; local[i] = globalBoard[i++][col]);
for (i = 0; i < size; i++)
{
if (i < size - 1)
{
if (globalBoard[i][col] == 2 && globalBoard[i + 1][col] == 0)
{
local[i++] = 0;
local[i] = 2;
}
}
else
{
if (globalBoard[i][col] == 2 && globalBoard[0][col] == 0)
{
local[i++] = 0;
local[0] = 2;
}
}
}
for (i = 0; i < size; i++)
globalBoard[i][col] = local[i];
}
Here, col is which column to work on and size is how big the grid is (it's always square).
I was thinking that I might be able to do some kind of fancy pointer arithmetic to speed this up, and was reading this : http://www.cs.umd.edu/class/sum2003/cmsc311/Notes/BitOp/pointer.html.
Looking at that, I feel like I might need to change how I declare the grid in order to take advantage of 2d array pointer arithmetic, but I'm still not sure how I would go about traversing columns using that method.
Any help with that, or any other suggestions of fast ways to go through a column are welcome.
UPDATE: After a bit more research and discussion, it would seem my assumptions were incorrect. Turns out it's actually taking almost twice as long to write the results back to the global array than it is to loop over columns, due to false sharing. That said, I'm still somewhat curious to see if there are any better ways of doing column traversal.
I think the answer is to process the grid in tiles. You can do a very quick tile move, either down or right, in a 16x16 or 32x32 tile. They two moves will be effectively the same, and run at the same speed: read all values into XMM registers, process, write. You may want to investigate MASKMOVDQU instruction here. If I understand the nature of the problem, you can overlap tiles by one row/column and this will work okay if you process them in the usual (scan) order. If not, you have to handle stitching the tiles separately.
There is no truly fast way to do this in C code. However, you can try (1) changing your board type to be a unit8_t, (2) replacing all if .. statements with arithmetic, like this: value = (mask & value) | (^mask & newvalue), and (3) turning on maximum loop unrolling and auto-vectorization in the compiler options. This will give you a nice speedup - especially avoiding conditionals.
EDIT In addition to tiles that can fit in registers, you can also do a second level of tiles sized to fit in your cache. I think the combination will run at roughly your memory bandwidth.
EDIT Or, make your board type be two bits: pack four cells to a byte. Goes nicely with the replacing if statements with arithmetic idea :)
I'm writing code for a decision tree in C. Right now it gives me the correct result (0% training error, low test error), but it takes a long time to run.
The problem lies in how often I run qsort. My basic algorithm is this:
for every feature
sort that feature column using qsort
remove duplicate feature values in that column
for every unique feature value
split
determine entropy given that split
save the best feature to split + split value
for every training_example
if training_example's value for best feature < best split value, store in Left[]
else store in Right[]
recursively call this function, using only the Left[] training examples
recursively call this function, using only the Right[] training examples
Because the last two lines are iterative calls, and because the tree can extend for dozens and dozens of branches, the number of calls to qsort is huge (especially for my dataset that has > 1000 features).
My idea to reduce the runtime is to create a 2d array (in a separate function) where each column is a sorted feature column. Then, as long as I maintain a vector of row numbers of the training examples in Left[] and Right[] for each recursive call, I can just call this separate function, grab the rows I want in the pre-sorted feature vector, and save the cost of having to qsort each time.
I'm fairly new to C and so I'm not sure how to code this. In MatLab I can just have a global array that any function can change or access, looking for something like that in C.
Global arrays in C are totally possible. There are actually two ways of doing that. In the first case the dimensions of the array are fixed for the application:
#define NROWS 100
#define NCOLS 100
int array[NROWS][NCOLS];
int main(void)
{
int i, j;
for (i = 0; i < NROWS; i++)
for (j = 0; j < NCOLS; j++)
{
array[i][j] = i+j;
}
return 0;
}
In the second example the dimensions may depend on values from the input.
#include <stdlib.h>
int **array;
int main(void)
{
int nrows = 100;
int ncols = 100;
int i, j;
array = malloc(nrows*sizeof(*array));
for (i = 0; i < nrows; i++)
{
array[i] = malloc(ncols*sizeof(*(array[i])));
for (j = 0; j < ncols; j++)
{
array[i][j] = i+j;
}
}
}
Although the access to the arrays in both examples looks deceivingly similar, the implementation of the arrays is quite different. In the first example the array is located in one piece of memory and the strides to access rows is a whole row. In the second example each row access is a pointer to a row, which is one piece of memory. The various rows can however be located in different areas of the memory. In the second example rows might also have a different length. In that case you would need to store the length of each row somewhere too.
I don't fully understand what you are trying to achieve, because I'm not familiar with the terminology of decision tree, feature and the standard approaches to training sets. But you may also want to have a look at other data structures to maintain sorted data:
http://en.wikipedia.org/wiki/Red–black_tree maintains a more or less balanced and sorted tree.
AVL tree a bit slower but more balanced and sorted tree.
Trie a sorted tree on lists of elements.
Hash function to easily map a complex element to an integral value that can be used to sort the elements. Good for finding exact elements, but there is no real order in the elements itself.
P.S1: Coming from Matlab you may want to consider a different language from C to move to. C++ has standard libraries to support above data structures. Java, Python come to mind or even Haskell if you are daring. Pointer handling in C can be quite tedious and error prone.
P.S2: I'm unable to include a - in a URL on StackOverflow. So the Red-black tree links is a bit off and can't be clicked. If someone can edit my post to fix it, then I would appreciate that.