I've been trying for a while to figure out the most efficient way of handling a two dimensional array (i.e. a matrix) that contains a variable.
I construct a large matrix by concatenating smaller arrays, weighted with a variable. As a simplified example, this is what I currently do.
function myMatrix(x::Float64)
M = vcat(hcat(x*A, C), hcat(C, 2*x*B))
return M
end
const A = ones(2,2)
const B = ones(2,2)
const C = zeros(2,2)
y_values = collect(0:0.01:10)
for y in y_values
eivals, eivecs = eig(myMatrix(y))
dosomething(eivals,eivecs)
end
The problem: I have more (and more complex, non-diagonal) matrices and the final size is pretty large. At this moment, I call the function a few hundred times. Does anyone have a suggestion on what to do to make this process more efficient in terms of runtime?
Thanks in advance
EDIT: Based on the comment below, this does not answer the question.
Do you mean constructing the Matrix? One way of doing that which is ideomatic is to use a block Matrix:
M = [x*A C
C 2x*B]
Not sure if that's the most efficient in terms of runtime, sorry.
This solution, using preallocation as proposed somewhere in the comments, does not solve my problem directly (because of the way I construct the matrices) but it may still be useful for some people reading this. Also I don't guarantee for the initially used statement of "most efficient", as this seems to widely depend on your purposes, size of the matrices and so on. The method is also mentioned in the Performance Tips section of Julia.
Because there has been some confusion, consider the following example:
function myMatrix(x::Float64)
M = vcat(hcat(x*A, C), hcat(D, 2*x*B))
return M
end
function doSomething(A::Array{Float64,2})
nothing
end
const ArraySize = 1000
const A = ones(ArraySize,ArraySize)
const B = ones(ArraySize,ArraySize)
const C = rand(ArraySize,ArraySize)
const D = rand(ArraySize,ArraySize)
for i = 1:1000
ret = myMatrix( convert(Float64,i) )
doSomething(ret)
end
This does literally nothing but construct a BlockMatrix(function) from initial matrices depending on one parameter. I was thinking this repeated construction is redundant and indeed, one can preallocate the memory for the matrix by writing:
function xinc!(ret::Array{T,2}, x::T) where T
ret[1:ArraySize, 1:ArraySize] = x*A
ret[1:ArraySize, ArraySize+1:2*ArraySize] = C
ret[ArraySize+1:2*ArraySize, 1:ArraySize] = D
ret[ArraySize+1:2*ArraySize, ArraySize+1:2*ArraySize] = 2*x*B
nothing
end
function doSomething(A::Array{Float64,2})
nothing
end
const ArraySize = 1000
const A = ones(ArraySize,ArraySize)
const B = ones(ArraySize,ArraySize)
const C = rand(ArraySize,ArraySize)
const D = rand(ArraySize,ArraySize)
ret = Array{Float64}(2*ArraySize, 2*ArraySize)
for i = 1:1000
xinc!(ret, convert(Float64,i))
doSomething(ret)
end
For me, the second code executes in 9.866s while the first takes up 38.076s.
Edit: In response to the previous comment, if i write
function xinc!(ret::Array{T,2}, x::T) where T
ret = [x*A C
D 2*x*B]
nothing
end
the code takes 16.173s to execute. I don't know why, but this way of assigning the matrix is substantially slower.
Related
In common arrays the intersection function would have worked great, but for tall arrays this function has not been implemented yet. So is there an other way to find the common elements of these two?
Use a for-loop to do the checks
n = 1e4;
m = 1e3; %Batchsize
a = tall(randi(n,n,1));
b = tall(randi(n,1,n));
idx = tall(false(n,1));
for ii = 1:n/m
idx = idx | any(a==b(1,(ii-1)*m+1:ii*m),2);
end
inter = a(idx);
Where m controls how much of the array you can load into the memory at the time. The more the faster, but you will need to have enough memory for it.
So I want to decompose array to multiple variables.
For example,
I have 'data' array of (136,9) size which is of double type.
I want to decompose the values of data(1,:) to multiple variables something like below:
[frm_id,seq_id,xmin,ymin,w,h,temp1,temp2,temp3] = data(1,:);
In python it was straightforward, but above code gives following error in matlab:
Insufficient number of outputs from right hand side of equal sign to satisfy
assignment.
I can go with something like
frm_id = data(1,1);
seq_id = data(1,2);
%ect
But I do believe there must be matlab (more neat) way to do this operation.
Thanks!
You can use num2cell to convert the matrix to a cell array then copy contents of the cell to each variable:
C = num2cell(data,1);
[frm_id,seq_id,xmin,ymin,w,h,temp1,temp2,temp3] = C{:};
I can only suggest you to create a function like this:
function [frm_id,seq_id,xmin,ymin,w,h,temp1,temp2,temp3] = myfunction (data)
frm_id = data(:,1);
seq_id = data(:,2);
xmin = data(:,3);
ymin = data(:,4);
w = data(:,5);
h = data(:,6);
temp1 = data(:,7);
temp2 = data(:,8);
temp3 = data(:,9);
so in your main code
[frm_id,seq_id,xmin,ymin,w,h,temp1,temp2,temp3] = myfunction(data);
Is it possible to find the difference beetwen two arrays of different size?
My problem is that I have two arrays, that scaled are pretty similar and I need the error in each point.
The data look like this:-
Yaw data is much bigger than Yaw Ref.
You could take a very naive approach and simply pad each element of the reference array. That is fairly simple to do:
n = length(yaw)/length(yaw_ref);
yaw_ref_pad = zeros(length(yaw), 1);
for j = 1:length(yaw_ref)-1
yaw_ref_pad((n*j):(n*(j+1)) = yaw_ref(j);
end
You could also do something more adaptive, which may or may not be what you want. This approach uses the derivatives to determine where the padded reference should switch. This might be considered a bit circular, since your system looks like an overdamped PID system and this uses the output to seed the input.
yaw_ref_pad = zeros(length(yaw), 1);
[x, peaks] = findpeaks(diff(yaw));
for j = 1:length(peaks)-1
yaw_ref_pad(peaks(j):peaks(j+1)) = yaw_ref(j);
end
Either way, after filling yaw_ref_pad, your result is simply
error = yaw_ref_pad - yaw;
I want to slice an 4D-array into n parts along the 5th Dimension in order to use it in parfor:
X(:,:,:,particles)-->X(:,:,:,particles/n,n)
The Problem is that X is so big that I run out of memory if I start writing it into a new variable, so i want to basically do:
X = cat(5,X(:,:,:,1:particles/n),X(:,:,:,particles/n+1:2*particles/n),...)
I am doing this with
sliced = 'cat(5'
for i=1:n
sliced = strcat(2,sliced,sprintf(',X(:,:,:,(1+(%i-1)*%i):%i*%i)',i,particles/n,i,particles/n))
end
sliced = strcat(2,sliced,')');
X = eval(sliced);
I get:
Error: The input character is not valid in MATLAB statements or expressions.
If i print out the contents of sliced and comment everything and paste the printout of sliced manually into eval('...') it works.
Anyone got a solution for my problem or another way of slicing a 4D array without using additional memory?
Thanks
You can use reshape, which must not use any additional memory -
sz_X = size(X) %// get size
X = reshape(X,sz_X(1),sz_X(2),sz_X(3),sz_X(4)/n,[]); %// reshape and save
%// into same variable and as such must be memory efficient
Ok. I just got things mixed up cat and strcat are not the same... oops :o
n = 4;
particles = 200;
X = rand(6,6,6,particles);
sliced = sprintf('X = cat(5');
for i = 1:n
sliced = cat(2,sliced,sprintf(',X(:,:,:,(1+(%i-1)*%i):%i*%i)',i,particles/n,i,particles/n));
end
sliced = cat(2,sliced,sprintf(');'));
eval(sliced);
works just fine. If somebody has got a better way to slice without memory usage - please feel free to post...
I have a routine that generates a series of data vectors, one iteration at a time. I would like to find a way to "grow" either a list or a matrix that holds these vectors. I tried to create a list,
PROTECT( myList = allocVector( VECSXP, 1 ) )
But is there a way to grow the list, by pushing a vector element in the end?
Also, I wouldn't mind using a matrix, since the vectors I generate are of the same length.
Rf_lengthgets in Rinternals.h; implemented in builtin.c:lengthgets. The returned pointer needs to be PROTECTed, so one pattern is
SEXP myList;
PROTECT_INDEX ipx;
PROTECT_WITH_INDEX(myList = allocVector( VECSXP, 1 ), &ipx);
REPROTECT(mylist = Rf_lengthgets(mylist, 100), ipx);
If one were growing a list based on some unknown stopping condition, the approach might be like in R, with pre-allocate and fill followed by extension; the following is psuedo-code:
const int BUF_SIZE = 100;
PROTECT_INDEX ipx;
SEXP myList;
int i, someCondition = 1;
PROTECT_WITH_INDEX(myList=allocVector(VECSXP, BUF_SIZE), &ipx);
for (i = 0; some_condition; ++i) {
if (Rf_length(myList) == i) {
const int len = Rf_length(myList) + BUF_SIZE;
REPROTECT(myList = Rf_lengthgets(mYlist, BUF_SIZE), &ipx);
}
PROTECT(result = some_calculation();
SET_VECTOR_ELT(myList, i, result);
UNPROTECT(1);
// set some_condition
}
Rf_lengthgets(myList, i); // no need to re-PROTECT; we're leaving C
UNPROTECT(1)
return myList;
This performs a deep copy of myList, so can become expensive and in some ways if ht emain objective to evaluate some_calculation, then it seems like it's easier and not too much less efficient to do the pre-allocate and extend operations in an R loop, calling some_calculation and doing assignment inside the loop.
This is IMHO a good example of where C++ beats C hands-down.
In C++, you can use a STL container (such as vector) and easily insert elements one at a time using push_back(). You never use malloc or free (or new and delete), and you never touch pointers. There is just no way to do that in C.
As well, you can make use of the Rcpp interface between R and C++ which makes getting the data you have grown in C++ over to R a lot easier.