I am working on some Matlab homework and I was having issues conceptualizing the way that it addresses matrices. In Matlab the matrix is address in d(row,col) format.
I have been programming for a while and have always tended to think of a one dimensional array as a horizontal structure with a second dimension extending out from below.
Which of these is a "more correct" way of thinking about an array data structure from the computer's point of view
Good question +1.
Purely from a Matlab programming perspective, it is best to think of a matrix as a sequence of column vectors. Why? Because this is how Matlab allocates them to your computers memory. That is, two sequential elements in any given column of a matrix will be allocated next to each other in memory. This is sometimes referred to as "column-major order", and is used in languages such as Fortran, R, and Julia. The opposite is, unsurprisingly, called "row-major order", and is used in C and Python.
The implication of this is that Matlab will be much faster at performing operations on the columns of a matrix than on the rows. #angainor provided a great answer to a question of mine a few months ago that demonstrates this fact. Based on #angainor's insight, here is a useful speed test to run:
M = 1000; %# Number of iterations over each method
T = 1000; %# Number of rows
N = 1000; %# Number of columns
X = randn(T, N); %# Random matrix
%# Loop over the rows of a matrix and perform a sum operation on each row vector
tic
for m = 1:M
for t = 1:T
sum(X(t, :));
end
end
toc
%# Loop over the columns of a matrix and perform a sum operation on each column vector
tic
for m = 1:M
for n = 1:N
sum(X(:, n));
end
end
toc
On my machine, the outcome of the test is:
Elapsed time is 9.371870 seconds. %# Looping over rows
Elapsed time is 1.943970 seconds. %# Looping over columns
In other words, operations performed on columns are almost 5 times faster than operations performed on rows!
From a mathematical perspective I don't trust myself to give a good answer. You could probably get some great insights from math.stackexchange.
Related
I am working with four MATLAB arrays of size 169x14, 207x14, 94x14, and 108x14. I would like to produce a single array which has the linear addition of every possible row combination of the four arrays. For example, one such combination may be the 99th row of array1, the 72nd row of array2, 6th row of array3, and 27th row of array 4 added together as a single row. These arrays are named helm, chest, arm, leg - this is for a stat calculator of a video game.
My first attempt at this was the following:
for i = 1:length(lin_helm)
for k = 1:length(lin_arm)
for j = 1:length(lin_leg)
for g = 1:length(lin_leg)
armor_comb = [armor_comb;
i j k g helm_array(i,2:15)+chest_array(j,2:15)+arm_array(k,2:15)+leg_array(g,2:15)];
end
end
end
end
Which uses nested for loops for each array and simply adds the rows together (note that 'lin_X' are just numbered vectors for the row number and the rows of the array are 2:15 because the first column is a row iterator). The first four columns of this result array can be ignored, they are just denoting which rows were selected from the other arrays. To say the least, this is extremely slow.
I then tried omitting the last for loop to instead take the first three selections and add them as an entire matrix to the entire last array. This was done by taking the addition of the first three row selections and using a matrix of ones. I chose to do this for the largest array, chest, to save the most time.
for i = 1:length(lin_helm)
for k = 1:length(lin_arm)
for j = 1:length(lin_leg)
armor_comb = [armor_comb;
i*ones(length(lin_chest),1) j*ones(length(lin_chest),1) k*ones(length(lin_chest),1) lin_chest' ones(length(lin_chest),14).*[helm_array(i,2:15)+leg_array(j,2:15)+arm_array(k,2:15)]+chest_array(:,2:15)];
end
end
end
This was significantly faster, but still extremely slow compared to the total array size needed.
I am not sure how to make this process faster by using matrix math. To generalize my issue, I am trying to find the numerical array of all possible row additions of an AxN, BxN, CxN, and DxN where any given selection takes one row from each array with no repeats.
All online documentation I can find just says to use nested for loops because they assume your array sizes are small. This is unpractical for my application, so I am seeking help on how to use matrices (or another method) to speed up computation time.
For making indexes (the first columns of your final matrix), you can try something like this:
function i=indexes(i1, i2)
i=[kron(i1, ones(size(i2, 1), 1)) kron(ones(size(i1, 1), 1), i2)];
end
If a and b are column vectors of indexes 1, 2, ..., then indexes(a, b) will be the pairs of index combos, and you can repeat for additional indexing columns, e.g., indexes(indexes(a, b), c).
If you have the indexes, say ii, you can add up what you want with something like
array1(ii(:, 1), 2:15) + array2(ii(:, 2), 2:15)
Prepend with ii if you really need to.
This will be much faster than a naive loop like you have initially. E.g., on my somewhat old Matlab, this:
n=10;
a=(1:2*n)';
b=(1:3*n)';
c=(1:5*n)';
tic
ii=indexes(indexes(a,b),c);
toc
tic
jj=[];
k=1;
for i1=1:length(a)
for i2=1:length(b)
for i3=1:length(c)
jj(k, :)=[i1 i2 i3];
k=k+1;
end
end
end
toc
gives
Elapsed time is 0.003514 seconds.
Elapsed time is 0.754066 seconds.
If you pre-allocate the storage for the loop case like jj=zeros(size(ii));, that's also significantly faster, though still slower than the kron-based approach, like with n=100:
Elapsed time is 3.323197 seconds.
Elapsed time is 9.825276 seconds.
I've just started using for loops in matlab in programming class and the basic stuff is doing me fine, However I've been asked to "Use loops to create a 3 x 5 matrix in which the value of each element is its row number to the power of its column number divided by the sum of its row number and column number for example the value of element (2,3) is (2^3 / 2+3) = 1.6
So what sort of looping do I need to use to enable me to start new lines to form a matrix?
Since you need to know the row and column numbers (and only because you have to use loops), for-loops are a natural choice. This is because a for-loop will automatically keep track of your row and column number for you if you set it up right. More specifically, you want a nested for loop, i.e. one for loop within another. The outer loop might loop through the rows and the inner loop through the columns for example.
As for starting new lines in a matrix, this is extremely bad practice to do in a loop. You should rather pre-allocate your matrix. This will have a major performance impact on your code. Pre-allocation is most commonly done using the zeros function.
e.g.
num_rows = 3;
num_cols = 5;
M = zeros(num_rows,num_cols); %// Preallocation of memory so you don't grow your matrix in your loop
for row = 1:num_rows
for col = 1:num_cols
M(row,col) = (row^col)/(row+col);
end
end
But the most efficient way to do it is probably not to use loops at all but do it in one shot using ndgrid:
[R, C] = ndgrid(1:num_rows, 1:num_cols);
M = (R.^C)./(R+C);
The command bsxfun is very helpful for such problems. It will do all the looping and preallocation for you.
eg:
bsxfun(#(x,y) x.^y./(x+y), (1:3)', 1:5)
I have 2 arrays, A and B. I want to form a new array C with same dimension as B where each element will show SUM(A) for A > B
Below is my working code
A = [1:1:1000]
B=[1:1:100]
for n = 1:numel(B)
C(n) = sum(A(A>B(n)));
end
However, when A has millions of rows and B has thousands, and I have to do similar calculations for 20 array-couples,it takes insane amount of time.
Is there any faster way?
For example, histcounts is pretty fast, but it counts, rather than summing.
Thanks
Depending on the size of your arrays (and your memory limitations), the following code might be slightly faster:
C = A*bsxfun(#gt,A',B);
Though it's vectorized, however, it seems to be bottlenecked (perhaps) by the allocation of memory. I'm looking to see if I can get a further speedup. Depending on your input vector size, I've seen up to a factor of 2 speedup for large vectors.
Here's a method that is a bit quicker, but I'm sure there is a better way to solve this problem.
a=sort(A); %// If A and B are already sorted then this isn't necessary!
b=sort(B);
c(numel(B))=0; %// Initialise c
s=cumsum(a,2,'reverse'); %// Get the partial sums of a
for n=1:numel(B)
%// Pull out the sum for elements in a larger than b(n)
c(n)=s(find(a>b(n),1,'first'));
end
According to some very rough tests, this seems to run a bit better than twice as fast as the original method.
You had the right ideas with histcounts, as you are basically "accumulating" certain A elements based on binning. This binning operation could be done with histc. Listed in this post is a solution that starts off with similar steps as listed in #David's answer and then uses histc to bin and sum up selective elements from A to get us the desired output and all of it in a vectorized manner. Here's the implementation -
%// Sort A and B and also get sorted B indices
sA = sort(A);
[sB,sortedB_idx] = sort(B);
[~,bin] = histc(sB,sA); %// Bin sorted B onto sorted A
C_out = zeros(1,numel(B)); %// Setup output array
%// Take care of the case when all elements in B are greater than A
if sA(1) > sB(end)
C_out(:) = sum(A);
end
%// Only do further processing if there is at least one element in B > any element in A
if any(bin)
csA = cumsum(sA,'reverse'); %// Reverse cumsum on sorted A
%// Get sum(A(A>B(n))) for every n, but for sorted versions
valid_mask = cummax(bin) - bin ==0;
valid_mask2 = bin(valid_mask)+1 <= numel(A);
valid_mask(1:numel(valid_mask2)) = valid_mask2;
C_out(valid_mask) = csA(bin(valid_mask)+1);
%// Rearrange C_out to get back in original unsorted version
[~,idx] = sort(sortedB_idx);
C_out = C_out(idx);
end
Also, please remember when comparing the result from this method with the one from the original for-loop version that there would be slight variations in output as this vectorized solution uses cumsum which computes a running summation and as such would have large cumulatively summed numbers being added to individual elements that are comparatively very small, whereas the for-loop version
would sum only selective elements. So, floating-precision issues would come up there.
I have a bit of a technical issue, but I feel like it should be possible with MATLAB's powerful toolset.
What I have is a random n by n matrix of 0's and w's, say generated with
A=w*(rand(n,n)<p);
A typical value of w would be 3000, but that should not matter too much.
Now, this matrix has two important quantities, the vectors
c = sum(A,1);
r = sum(A,2)';
These are two row vectors, the first denotes the sum of each column and the second the sum of each row.
What I want to do next is randomize each value of w, for example between 0.5 and 2. This I would do as
rand_M = (0.5-2).*rand(n,n) + 0.5
A_rand = rand_M.*A;
However, I don't want to just pick these random numbers: I want them to be such that for every column and row, the sums are still equal to the elements of c and r. So to clean up the notation a bit, say we define
A_rand_c = sum(A_rand,1);
A_rand_r = sum(A_rand,2)';
I want that for all j = 1:n, A_rand_c(j) = c(j) and A_rand_r(j) = r(j).
What I'm looking for is a way to redraw the elements of rand_M in a sort of algorithmic fashion I suppose, so that these demands are finally satisfied.
Now of course, unless I have infinite amounts of time this might not really happen. I therefore accept these quantities to fall into a specific range: A_rand_c(j) has to be an element of [(1-e)*c(j),(1+e)*c(j)] and A_rand_r(j) of [(1-e)*r(j),(1+e)*r(j)]. This e I define beforehand, say like 0.001 or something.
Would anyone be able to help me in the process of finding a way to do this? I've tried an approach where I just randomly repick the numbers, but this really isn't getting me anywhere. It does not have to be crazy efficient either, I just need it to work in finite time for networks of size, say, n = 50.
To be clear, the final output is the matrix A_rand that satisfies these constraints.
Edit:
Alright, so after thinking a bit I suppose it might be doable with some while statement, that goes through every element of the matrix. The difficult part is that there are four possibilities: if you are in a specific element A_rand(i,j), it could be that A_rand_c(j) and A_rand_r(i) are both too small, both too large, or opposite. The first two cases are good, because then you can just redraw the random number until it is smaller than the current value and improve the situation. But the other two cases are problematic, as you will improve one situation but not the other. I guess it would have to look at which criteria is less satisfied, so that it tries to fix the one that is worse. But this is not trivial I would say..
You can take advantage of the fact that rows/columns with a single non-zero entry in A automatically give you results for that same entry in A_rand. If A(2,5) = w and it is the only non-zero entry in its column, then A_rand(2,5) = w as well. What else could it be?
You can alternate between finding these single-entry rows/cols, and assigning random numbers to entries where the value doesn't matter.
Here's a skeleton for the process:
A_rand=zeros(size(A)) is the matrix you are going to fill
entries_left = A>0 is a binary matrix showing which entries in A_rand you still need to fill
col_totals=sum(A,1) is the amount you still need to add in every column of A_rand
row_totals=sum(A,2) is the amount you still need to add in every row of A_rand
while sum( entries_left(:) ) > 0
% STEP 1:
% function to fill entries in A_rand if entries_left has rows/cols with one nonzero entry
% you will need to keep looping over this function until nothing changes
% update() A_rand, entries_left, row_totals, col_totals every time you loop
% STEP 2:
% let (i,j) be the indeces of the next non-zero entry in entries_left
% assign a random number to A_rand(i,j) <= col_totals(j) and <= row_totals(i)
% update() A_rand, entries_left, row_totals, col_totals
end
update()
A_rand(i,j) = random_value;
entries_left(i,j) = 0;
col_totals(j) = col_totals(j) - random_value;
row_totals(i) = row_totals(i) - random_value;
end
Picking the range for random_value might be a little tricky. The best I can think of is to draw it from a relatively narrow distribution centered around N*w*p where p is the probability of an entry in A being nonzero (this would be the average value of row/column totals).
This doesn't scale well to large matrices as it will grow with n^2 complexity. I tested it for a 200 by 200 matrix and it worked in about 20 seconds.
Searching around here one finds many questions how one can convert cell arrays of doubles into one big matrix.
In my application I have a two dimensional cell array (lets call it celldata of size m times n) of all same sized double matrices (lets say of size a times b).
I want to convert that data structure into one bit 4D double (m times n times a times b).
At the moment I do that by
reshape(cat(3,celldata{:}),m,n,a,b)
but maybe there are other methods doing that directly? Maybe with a call like
cat([3 4],celldata{:,:})
or similar.
I think
cell2mat(permute(celldata, [3 4 1 2]))
will do the trick. However,
%// create some bogus data
m = 1.1e2;
n = 1.2e2;
a = 1.3e2;
b = 1.4e2;
celldata = cellfun(#(~) randi(10, a,b, 'uint8'), cell(m,n), 'UniformOutput', false);
%// new method
tic
cell2mat(permute(celldata, [3 4 1 2]));
toc
%// your current method
tic
reshape(cat(3,celldata{:}),m,n,a,b);
toc
Results:
Elapsed time is 1.745495 seconds. % cell2mat/permute
Elapsed time is 0.305368 seconds. % reshape/cat
cell2mat is a matlab m-file (with necessary inefficiencies in the loop due to compatibility issues), while reshape and cat are built-ins. This is where that difference comes from.
I'd stick with your current method :)
Now, I'm asking you why you'd want to do this convesion in the first place. Is it an indexing problem? Because
celldata{x,y}(w,z)
prevents you from having to do the conversion, so you can index like
converted_celldata(x,y,w,z)
I don't see other reasons, because matrix/vector operations don't work anyway on 4D arrays...