I am working with four MATLAB arrays of size 169x14, 207x14, 94x14, and 108x14. I would like to produce a single array which has the linear addition of every possible row combination of the four arrays. For example, one such combination may be the 99th row of array1, the 72nd row of array2, 6th row of array3, and 27th row of array 4 added together as a single row. These arrays are named helm, chest, arm, leg - this is for a stat calculator of a video game.
My first attempt at this was the following:
for i = 1:length(lin_helm)
for k = 1:length(lin_arm)
for j = 1:length(lin_leg)
for g = 1:length(lin_leg)
armor_comb = [armor_comb;
i j k g helm_array(i,2:15)+chest_array(j,2:15)+arm_array(k,2:15)+leg_array(g,2:15)];
end
end
end
end
Which uses nested for loops for each array and simply adds the rows together (note that 'lin_X' are just numbered vectors for the row number and the rows of the array are 2:15 because the first column is a row iterator). The first four columns of this result array can be ignored, they are just denoting which rows were selected from the other arrays. To say the least, this is extremely slow.
I then tried omitting the last for loop to instead take the first three selections and add them as an entire matrix to the entire last array. This was done by taking the addition of the first three row selections and using a matrix of ones. I chose to do this for the largest array, chest, to save the most time.
for i = 1:length(lin_helm)
for k = 1:length(lin_arm)
for j = 1:length(lin_leg)
armor_comb = [armor_comb;
i*ones(length(lin_chest),1) j*ones(length(lin_chest),1) k*ones(length(lin_chest),1) lin_chest' ones(length(lin_chest),14).*[helm_array(i,2:15)+leg_array(j,2:15)+arm_array(k,2:15)]+chest_array(:,2:15)];
end
end
end
This was significantly faster, but still extremely slow compared to the total array size needed.
I am not sure how to make this process faster by using matrix math. To generalize my issue, I am trying to find the numerical array of all possible row additions of an AxN, BxN, CxN, and DxN where any given selection takes one row from each array with no repeats.
All online documentation I can find just says to use nested for loops because they assume your array sizes are small. This is unpractical for my application, so I am seeking help on how to use matrices (or another method) to speed up computation time.
For making indexes (the first columns of your final matrix), you can try something like this:
function i=indexes(i1, i2)
i=[kron(i1, ones(size(i2, 1), 1)) kron(ones(size(i1, 1), 1), i2)];
end
If a and b are column vectors of indexes 1, 2, ..., then indexes(a, b) will be the pairs of index combos, and you can repeat for additional indexing columns, e.g., indexes(indexes(a, b), c).
If you have the indexes, say ii, you can add up what you want with something like
array1(ii(:, 1), 2:15) + array2(ii(:, 2), 2:15)
Prepend with ii if you really need to.
This will be much faster than a naive loop like you have initially. E.g., on my somewhat old Matlab, this:
n=10;
a=(1:2*n)';
b=(1:3*n)';
c=(1:5*n)';
tic
ii=indexes(indexes(a,b),c);
toc
tic
jj=[];
k=1;
for i1=1:length(a)
for i2=1:length(b)
for i3=1:length(c)
jj(k, :)=[i1 i2 i3];
k=k+1;
end
end
end
toc
gives
Elapsed time is 0.003514 seconds.
Elapsed time is 0.754066 seconds.
If you pre-allocate the storage for the loop case like jj=zeros(size(ii));, that's also significantly faster, though still slower than the kron-based approach, like with n=100:
Elapsed time is 3.323197 seconds.
Elapsed time is 9.825276 seconds.
Related
I've just started using for loops in matlab in programming class and the basic stuff is doing me fine, However I've been asked to "Use loops to create a 3 x 5 matrix in which the value of each element is its row number to the power of its column number divided by the sum of its row number and column number for example the value of element (2,3) is (2^3 / 2+3) = 1.6
So what sort of looping do I need to use to enable me to start new lines to form a matrix?
Since you need to know the row and column numbers (and only because you have to use loops), for-loops are a natural choice. This is because a for-loop will automatically keep track of your row and column number for you if you set it up right. More specifically, you want a nested for loop, i.e. one for loop within another. The outer loop might loop through the rows and the inner loop through the columns for example.
As for starting new lines in a matrix, this is extremely bad practice to do in a loop. You should rather pre-allocate your matrix. This will have a major performance impact on your code. Pre-allocation is most commonly done using the zeros function.
e.g.
num_rows = 3;
num_cols = 5;
M = zeros(num_rows,num_cols); %// Preallocation of memory so you don't grow your matrix in your loop
for row = 1:num_rows
for col = 1:num_cols
M(row,col) = (row^col)/(row+col);
end
end
But the most efficient way to do it is probably not to use loops at all but do it in one shot using ndgrid:
[R, C] = ndgrid(1:num_rows, 1:num_cols);
M = (R.^C)./(R+C);
The command bsxfun is very helpful for such problems. It will do all the looping and preallocation for you.
eg:
bsxfun(#(x,y) x.^y./(x+y), (1:3)', 1:5)
I have several arrays that are calculated example a,b and c (there are more than three) are calculated: Please note this is just an example the numbers are much larger and are not so basic
a=[1,2,3,4,5] b=[10,20,30,40,50] c=[100,200,300,400,500] and I want a for loop that inserts zeros into it so I can have the new_abc array steps look like.
1st for loop step new_abc=[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]
2nd for loop step new_abc=[1,0,0,2,0,0,3,0,0,4,0,0,5,0,0]
3rd for loop step new_abc=[1,10,0,2,20,0,3,30,0,4,40,0,5,50,0]
4th for loop step new_abc=[1,10,100,2,20,200,3,30,300,4,40,400,5,50,500]
how can I do this with a for loop?
I started with the code below which gives me the zeros
a=[1,2,3,4,5]
new_abc=zeros(1,length(a)*(3));
But I'm not sure how to place the values of the array a b and c using a for loopinto the correct locations ofnew_abc
I know I could place all the arrays into one large array and do a reshape but the calculated arrays I use become to large and I run out of ram, so reading / calculating each array and inserting them into one common array new_abcusing a for loop works best.
I'm running octave 3.8.1 which is like matlab.
This should do it. You can put a,b,c into a cell array. (you can also put them in a matrix...)
new_abc = zeros(1, 3*numel(a));
in = {a, b, c};
for k = 1:3
new_abc(k:3:end) = in{k};
end
I have 2 arrays, A and B. I want to form a new array C with same dimension as B where each element will show SUM(A) for A > B
Below is my working code
A = [1:1:1000]
B=[1:1:100]
for n = 1:numel(B)
C(n) = sum(A(A>B(n)));
end
However, when A has millions of rows and B has thousands, and I have to do similar calculations for 20 array-couples,it takes insane amount of time.
Is there any faster way?
For example, histcounts is pretty fast, but it counts, rather than summing.
Thanks
Depending on the size of your arrays (and your memory limitations), the following code might be slightly faster:
C = A*bsxfun(#gt,A',B);
Though it's vectorized, however, it seems to be bottlenecked (perhaps) by the allocation of memory. I'm looking to see if I can get a further speedup. Depending on your input vector size, I've seen up to a factor of 2 speedup for large vectors.
Here's a method that is a bit quicker, but I'm sure there is a better way to solve this problem.
a=sort(A); %// If A and B are already sorted then this isn't necessary!
b=sort(B);
c(numel(B))=0; %// Initialise c
s=cumsum(a,2,'reverse'); %// Get the partial sums of a
for n=1:numel(B)
%// Pull out the sum for elements in a larger than b(n)
c(n)=s(find(a>b(n),1,'first'));
end
According to some very rough tests, this seems to run a bit better than twice as fast as the original method.
You had the right ideas with histcounts, as you are basically "accumulating" certain A elements based on binning. This binning operation could be done with histc. Listed in this post is a solution that starts off with similar steps as listed in #David's answer and then uses histc to bin and sum up selective elements from A to get us the desired output and all of it in a vectorized manner. Here's the implementation -
%// Sort A and B and also get sorted B indices
sA = sort(A);
[sB,sortedB_idx] = sort(B);
[~,bin] = histc(sB,sA); %// Bin sorted B onto sorted A
C_out = zeros(1,numel(B)); %// Setup output array
%// Take care of the case when all elements in B are greater than A
if sA(1) > sB(end)
C_out(:) = sum(A);
end
%// Only do further processing if there is at least one element in B > any element in A
if any(bin)
csA = cumsum(sA,'reverse'); %// Reverse cumsum on sorted A
%// Get sum(A(A>B(n))) for every n, but for sorted versions
valid_mask = cummax(bin) - bin ==0;
valid_mask2 = bin(valid_mask)+1 <= numel(A);
valid_mask(1:numel(valid_mask2)) = valid_mask2;
C_out(valid_mask) = csA(bin(valid_mask)+1);
%// Rearrange C_out to get back in original unsorted version
[~,idx] = sort(sortedB_idx);
C_out = C_out(idx);
end
Also, please remember when comparing the result from this method with the one from the original for-loop version that there would be slight variations in output as this vectorized solution uses cumsum which computes a running summation and as such would have large cumulatively summed numbers being added to individual elements that are comparatively very small, whereas the for-loop version
would sum only selective elements. So, floating-precision issues would come up there.
MATLAB is well-known for being column-major. Consequently, manipulating entries of an array that are in the same column is faster than manipulating entries that are on the same row.
In that case, why do so many built-in functions, such as linspace and logspace, output row vectors rather than column vectors? This seems to me like a de-optimization...
What, if any, is the rationale behind this design decision?
It is a good question. Here are some ideas...
My first thought was that in terms of performance and contiguous memory, it doesn't make a difference if it's a row or a column -- they are both contiguous in memory. For a multidimensional (>1D) array, it is correct that it is more efficient to index a whole column of the array (e.g. v(:,2)) rather than a row (e.g. v(2,:)) or other dimension because in the row (non-column) case it is not accessing elements that are contiguous in memory. However, for a row vector that is 1-by-N, the elements are contiguous because there is only one row, so it doesn't make a difference.
Second, it is simply easier to display row vectors in the Command Window, especially since it wraps the rows of long arrays. With a long column vector, you will be forced to scroll for much shorter arrays.
More thoughts...
Perhaps row vector output from linspace and logspace is just to be consistent with the fact that colon (essentially a tool for creating linearly spaced elements) makes a row:
>> 0:2:16
ans =
0 2 4 6 8 10 12 14 16
The choice was made at the beginning of time and that was that (maybe?).
Also, the convention for loop variables could be important. A row is necessary to define multiple iterations:
>> for k=1:5, k, end
k =
1
k =
2
k =
3
k =
4
k =
5
A column will be a single iteration with a non-scalar loop variable:
>> for k=(1:5)', k, end
k =
1
2
3
4
5
And maybe the outputs of linspace and logspace are commonly looped over. Maybe? :)
But, why loop over a row vector anyway? Well, as I say in my comments, it's not that a row vector is used for loops, it's that it loops through the columns of the loop expression. Meaning, with for v=M where M is a 2-by-3 matrix, there are 3 iterations, where v is a 2 element column vector in each iteration. This is actually a good design if you consider that this involves slicing the loop expression into columns (i.e. chunks of contiguous memory!).
I have a bit of a technical issue, but I feel like it should be possible with MATLAB's powerful toolset.
What I have is a random n by n matrix of 0's and w's, say generated with
A=w*(rand(n,n)<p);
A typical value of w would be 3000, but that should not matter too much.
Now, this matrix has two important quantities, the vectors
c = sum(A,1);
r = sum(A,2)';
These are two row vectors, the first denotes the sum of each column and the second the sum of each row.
What I want to do next is randomize each value of w, for example between 0.5 and 2. This I would do as
rand_M = (0.5-2).*rand(n,n) + 0.5
A_rand = rand_M.*A;
However, I don't want to just pick these random numbers: I want them to be such that for every column and row, the sums are still equal to the elements of c and r. So to clean up the notation a bit, say we define
A_rand_c = sum(A_rand,1);
A_rand_r = sum(A_rand,2)';
I want that for all j = 1:n, A_rand_c(j) = c(j) and A_rand_r(j) = r(j).
What I'm looking for is a way to redraw the elements of rand_M in a sort of algorithmic fashion I suppose, so that these demands are finally satisfied.
Now of course, unless I have infinite amounts of time this might not really happen. I therefore accept these quantities to fall into a specific range: A_rand_c(j) has to be an element of [(1-e)*c(j),(1+e)*c(j)] and A_rand_r(j) of [(1-e)*r(j),(1+e)*r(j)]. This e I define beforehand, say like 0.001 or something.
Would anyone be able to help me in the process of finding a way to do this? I've tried an approach where I just randomly repick the numbers, but this really isn't getting me anywhere. It does not have to be crazy efficient either, I just need it to work in finite time for networks of size, say, n = 50.
To be clear, the final output is the matrix A_rand that satisfies these constraints.
Edit:
Alright, so after thinking a bit I suppose it might be doable with some while statement, that goes through every element of the matrix. The difficult part is that there are four possibilities: if you are in a specific element A_rand(i,j), it could be that A_rand_c(j) and A_rand_r(i) are both too small, both too large, or opposite. The first two cases are good, because then you can just redraw the random number until it is smaller than the current value and improve the situation. But the other two cases are problematic, as you will improve one situation but not the other. I guess it would have to look at which criteria is less satisfied, so that it tries to fix the one that is worse. But this is not trivial I would say..
You can take advantage of the fact that rows/columns with a single non-zero entry in A automatically give you results for that same entry in A_rand. If A(2,5) = w and it is the only non-zero entry in its column, then A_rand(2,5) = w as well. What else could it be?
You can alternate between finding these single-entry rows/cols, and assigning random numbers to entries where the value doesn't matter.
Here's a skeleton for the process:
A_rand=zeros(size(A)) is the matrix you are going to fill
entries_left = A>0 is a binary matrix showing which entries in A_rand you still need to fill
col_totals=sum(A,1) is the amount you still need to add in every column of A_rand
row_totals=sum(A,2) is the amount you still need to add in every row of A_rand
while sum( entries_left(:) ) > 0
% STEP 1:
% function to fill entries in A_rand if entries_left has rows/cols with one nonzero entry
% you will need to keep looping over this function until nothing changes
% update() A_rand, entries_left, row_totals, col_totals every time you loop
% STEP 2:
% let (i,j) be the indeces of the next non-zero entry in entries_left
% assign a random number to A_rand(i,j) <= col_totals(j) and <= row_totals(i)
% update() A_rand, entries_left, row_totals, col_totals
end
update()
A_rand(i,j) = random_value;
entries_left(i,j) = 0;
col_totals(j) = col_totals(j) - random_value;
row_totals(i) = row_totals(i) - random_value;
end
Picking the range for random_value might be a little tricky. The best I can think of is to draw it from a relatively narrow distribution centered around N*w*p where p is the probability of an entry in A being nonzero (this would be the average value of row/column totals).
This doesn't scale well to large matrices as it will grow with n^2 complexity. I tested it for a 200 by 200 matrix and it worked in about 20 seconds.