Related
I basically want to use the function ismember, but for a range. For example, I want to know what data points in array1 are within n distance to array2, for each element in array2.
I have the following:
array1 = [1,2,3,4,5]
array2 = [2,2,3,10,20,40,50]
I want to know what values in array2 are <= 2 away from array1:
indices(1,:) (where array1(1) = 1) = [1 1 1 0 0 0 0]
indices(2,:) (where array1(2) = 2) = [1 1 1 0 0 0 0]
indices(3,:) (where array1(3) = 3) = [1 1 1 0 0 0 0]
indices(4,:) (where array1(4) = 4) = [1 1 1 0 0 0 0]
indices(5,:) (where array1(5) = 5) = [0 0 1 0 0 0 0]
Drawbacks:
My array1 is 496736 elements, my array2 is 9268 elements, so aI would rather not use a loop.
Looping is a valid option here. Intialise an array output to the size of array1 X array2, then loop over all elements in array1 an subtract array2 from that, then check whether the absolute value is less than or equal to 2:
array1 = [1,2,3,4,5];
array2 = [2,2,3,10,20,40,50];
output = zeros(numel(array1), numel(array2),'logical');
for ii = 1:numel(array1)
output(ii,:) = abs(array1(ii)-array2)<=2;
end
output =
1 1 1 0 0 0 0
1 1 1 0 0 0 0
1 1 1 0 0 0 0
1 1 1 0 0 0 0
0 0 1 0 0 0 0
i.e. loops are not the problem.
Thanks to Rahnema1's suggestion, you can initialise output directly as a logical matrix:
output = zeros(numel(array1),numel(array2),'logical');
whose size is just 4.3GB.
On timings: Hans' code runs in a matter of seconds for array1 = 5*rand(496736,1); array2 = 25*rand(9286,1);, the looped solution takes about 15 times longer. Both solutions are equal to one another. obcahrdon's ismembertol solution is somewhere in between on my machine.
On RAM usage:
Both implicit expansion, as per Hans' answer, as well as the loop suggested in mine work with just 4.3GB RAM on your expanded problem size (496736*9286)
pdist2 as per Luis' answer and bsxfun as per Hans' on the other hand try to create an intermediate double matrix of 34GB (which doesn't even fit in my RAM, so I cannot compare timings).
obchardon's ismembertol solution outputs a different form of the solution, and takes ~5.04GB (highly dependent on the amount of matches found, the more, the larger this number will be).
In general this leads me to the conclusion that implicit expansion should be your option of choice, but if you have R2016a or earlier, ismembertol or a loop is the way to go.
Using implicit expansion, introduced in MATLAB R2016b, you can simply write:
abs(array1.' - array2) <= 2
ans =
1 1 1 0 0 0 0
1 1 1 0 0 0 0
1 1 1 0 0 0 0
1 1 1 0 0 0 0
0 0 1 0 0 0 0
For earlier MATLAB versions, you can get this using the bsxfun function:
abs(bsxfun(#minus, array1.', array2)) <= 2
ans =
1 1 1 0 0 0 0
1 1 1 0 0 0 0
1 1 1 0 0 0 0
1 1 1 0 0 0 0
0 0 1 0 0 0 0
Hope that helps!
P.S. On the "MATLAB is slow for loops" myth, please have a look at that blog post for example.
EDIT: Please read Adriaan's answer on the RAM consumption using this and/or his approach!
If you have the Statistics Toolbox, you can use pdist2 to compute the distance matrix:
result = pdist2(array1(:), array2(:)) <= 2;
As noted by #Adriaan, this is not efficient for large input sizes. In that case a better approach is a loop with preallocation of the logical matrix output, as in his answer.
You can also use ismembertol with some specific option:
A = [1,2,3,4,5];
B = [2,2,3,10,20,40,5000];
tol = 2;
[~,ind] = ismembertol([A-tol;A+tol].',[B-tol;B+tol].',tol, 'ByRows', true, ...
'OutputAllIndices', true, 'DataScale', [1,Inf])
It will create a 5x1 cell array containing the corresponding linear indice
ind{1} = [1,2,3]
ind{2} = [1,2,3]
...
ind{5} = [3]
In this case using linear indices instead of logical indices will greatly reduce the memory consumption.
I would like to create a square matrix of size n x n where the diagonal elements as well as the left-diagonal are all equal to 1. The rest of the elements are equal to 0.
For example, this would be the expected result if the matrix was 5 x 5:
1 0 0 0 0
1 1 0 0 0
0 1 1 0 0
0 0 1 1 0
0 0 0 1 1
How could I do this in MATLAB?
Trivial using the tril function:
tril(ones(n),0) - tril(ones(n),-2)
And if you wanted a thicker line of 1s just adjust that -2:
n = 10;
m = 4;
tril(ones(n),0) - tril(ones(n),-m)
If you prefer to use diag like excaza suggested then try
diag(ones(n,1)) + diag(ones(n-1,1),-1)
but you can't control the 'thickness' of the stripe this way. However, for a thickness of 2, it might perform better. You'd have to test it though.
You can also use spdiags too to create that matrix:
n = 5;
v = ones(n,1);
d = full(spdiags([v v], [-1 0], n, n));
We get:
>> d
d =
1 0 0 0 0
1 1 0 0 0
0 1 1 0 0
0 0 1 1 0
0 0 0 1 1
The first two lines define the desired size of the matrix, assuming a square n x n as well as a vector of all ones that is of length n x 1. We then call spdiags to define where along the diagonal of this matrix this vector will be populating. We want to define the main diagonal to have all ones as well as the diagonal to the left of the main diagonal, or -1 away from the main diagonal. spdiags will adjust the total number of elements for the diagonal away from the main to compensate.
We also ensure that the output is of size n x n, but this matrix is actually sparse . We need to convert the matrix to full to complete the result.,
With a bit of indices juggling, you can also do this:
N = 5;
ind = repelem(1:N, 2); % [1 1 2 2 3 3 ... N N]
M = full(sparse(ind(2:end), ind(1:end-1), 1))
Simple approach using linear indexing:
n = 5;
M = eye(n);
M(2:n+1:end) = 1;
This can also be done with bsxfun:
n = 5; %// matrix size
d = [0 -1]; %// diagonals you want set to 1
M = double(ismember(bsxfun(#minus, 1:n, (1:n).'), d));
For example, to obtain a 5x5 matrix with the main diagonal and the two diagonals below set to 1, define n=5 and d = [0 -1 -2], which gives
M =
1 0 0 0 0
1 1 0 0 0
1 1 1 0 0
0 1 1 1 0
0 0 1 1 1
Suppose I have the following matrix
1 1 0 0 0
1 1 0 0 0
0 0 1 1 1
0 0 1 1 1
0 0 1 1 1
The result would be
{[1,2],[3,4,5]}
How would I implement this?
I have an ugly solution involving a loop that runs through the diagonal (except (1,1)) and checks whether the element directly left is 0. If not, that is the start of a new cluster.
Is there a prettier solution?
EDIT: current solution:
n = size(input, 2);
result = cell(1,n);
result{1} = 1;
counter = 1;
for i = 2:n
if input(i,i-1) ~= 1
counter = counter + 1;
end
result{counter} = [result{counter} i];
end
result = result(~cellfun('isempty',result));
use unique with 'rows' argument on the matrix transposed
In Matlab, say I have the following matrix, which represents a population of 10 individuals:
pop = [0 0 0 0 0; 1 1 1 0 0; 1 1 1 1 1; 1 1 1 0 0; 0 0 0 0 0; 0 0 0 0 0; 1 0 0 0 0; 1 1 1 1 1; 0 0 0 0 0; 0 0 0 0 0];
Where rows of ones and zeros define 6 different 'types' of individuals.
a = [0 0 0 0 0];
b = [1 0 0 0 0];
c = [1 1 0 0 0];
d = [1 1 1 0 0];
e = [1 1 1 1 0];
f = [1 1 1 1 1];
I want to define the proportion/frequency of a, b, c, d, e and f in pop.
I want to end up with the following list:
a = 0.5;
b = 0.1;
c = 0;
d = 0.2;
e = 0;
f = 0.2;
One way I can think of is by summing the rows, then counting the number of times each appears, and then sorting and indexing
sum_pop = sum(pop')';
x = unique(sum_pop);
N = numel(x);
count = zeros(N,1);
for l = 1:N
count(l) = sum(sum_pop==x(l));
end
pop_frequency = [x(:) count/10];
But this doesn't quite get me what I want (i.e. when frequency = 0) and it seems there must be a faster way?
You can use pdist2 (Statistics Toolbox) to get all frequencies:
indiv = [a;b;c;d;e;f]; %// matrix with all individuals
result = mean(pdist2(pop, indiv)==0, 1);
This gives, in your example,
result =
0.5000 0.1000 0 0.2000 0 0.2000
Equivalently, you can use bsxfun to manually compute pdist2(pop, indiv)==0, as in Divakar's answer.
For the specific individuals in your example (that can be identified by the number of ones) you could also do
result = histc(sum(pop, 2), 0:size(pop,2)) / size(pop,1);
There is some functionality in unique that can be used for this. If
[q,w,e] = unique(pop,'rows');
q is the matrix of unique rows, w is the index of the row first appears in the matrix. The third element e contains indices of q so that pop = q(e,:). Armed with this, the rest of the problem should be straight forward. The probability of a value in e should be the probability that this row appears in pop.
The counting can be done with histc
histc(e,1:max(e))/length(e)
and the non occuring rows can be found with
ismember(a,q,'rows')
There is of course other ways as well, maybe (probably) faster ways, or oneliners. Why I post this is because it provides a way that is easy to understand, readable and that does not require any special toolboxes.
EDIT
This example gives expected output
a = [0,0,0,0,0;1,0,0,0,0;1,1,0,0,0;1,1,1,0,0;1,1,1,1,0;1,1,1,1,1]; % catenated a-f
[q,w,e] = unique(pop,'rows');
prob = histc(e,1:max(e))/length(e);
out = zeros(size(a,1),1);
out(ismember(a,q,'rows')) = prob;
Approach #1
With bsxfun -
A = cat(1,a,b,c,d,e,f)
out = squeeze(sum(all(bsxfun(#eq,pop,permute(A,[3 2 1])),2),1))/size(pop,1)
Output -
out =
0.5000
0.1000
0
0.2000
0
0.2000
Approach #2
If those elements are binary numbers, you can convert them into decimal format.
Thus, decimal format for pop becomes -
>> bi2de(pop)
ans =
0
7
31
7
0
0
1
31
0
0
And that of the concatenated array, A becomes -
>> bi2de(A)
ans =
0
1
3
7
15
31
Finally, you need to count the decimal formatted numbers from A in that of pop, which you can do with histc. Here's the code -
A = cat(1,a,b,c,d,e,f)
out = histc(bi2de(pop),bi2de(A))/size(pop,1)
Output -
out =
0.5000
0.1000
0
0.2000
0
0.2000
I think ismember is the most direct and general way to do this. If your groups were more complicated, this would be the way to go:
population = [0,0,0,0,0; 1,1,1,0,0; 1,1,1,1,1; 1,1,1,0,0; 0,0,0,0,0; 0,0,0,0,0; 1,0,0,0,0; 1,1,1,1,1; 0,0,0,0,0; 0,0,0,0,0];
groups = [0,0,0,0,0; 1,0,0,0,0; 1,1,0,0,0; 1,1,1,0,0; 1,1,1,1,0; 1,1,1,1,1];
[~, whichGroup] = ismember(population, groups, 'rows');
freqOfGroup = accumarray(whichGroup, 1)/size(groups, 1);
In your special case the groups can be represented by their sums, so if this generic solution is not fast enough, use the sum-histc simplification Luis used.
I want to find the lengths of all series of ones and zeros in a logical array in MATLAB. This is what I did:
A = logical([0 0 0 1 1 1 1 0 1 1 0 0 0 0 0 0 1 1 1 1 1]);
%// Find series of ones:
csA = cumsum(A);
csOnes = csA(diff([A 0]) == -1);
seriesOnes = [csOnes(1) diff(csOnes)];
%// Find series of zeros (same way, using ~A)
csNegA = sumsum(~A);
csZeros = csNegA(diff([~A 0]) == -1);
seriesZeros = [csZeros(1) diff(csZeros)];
This works, and gives seriesOnes = [4 2 5] and seriesZeros = [3 1 6]. However it is rather ugly in my opinion.
I want to know if there is a better way to do this. Performance is not an issue as this is inexpensive (A is no longer than a few thousand elements). I am looking for code clarity and elegance.
If nothing better can be done, I'll just put this in a little helper function so I don't have to look at it.
You could use an existing code for run-length-encoding, which does the (ugly) work for you and then filter out your vectors yourself. This way your helper function is rather general and its functionality is evident from the name runLengthEncode.
Reusing code from this answer:
function [lengths, values] = runLengthEncode(data)
startPos = find(diff([data(1)-1, data]));
lengths = diff([startPos, numel(data)+1]);
values = data(startPos);
You would then filter out your vectors using:
A = logical([0 0 0 1 1 1 1 0 1 1 0 0 0 0 0 0 1 1 1 1 1]);
[lengths, values] = runLengthEncode(A);
seriesOnes = lengths(values==1);
seriesZeros = lengths(values==0);
You can try this:
A = logical([0 0 0 1 1 1 1 0 1 1 0 0 0 0 0 0 1 1 1 1 1]);
B = [~A(1) A ~A(end)]; %// Add edges at start/end
edges_indexes = find(diff(B)); %// find edges
lengths = diff(edges_indexes); %// length between edges
%// Separate zeros and ones, to a cell array
s(1+A(1)) = {lengths(1:2:end)};
s(1+~A(1)) = {lengths(2:2:end)};
This strfind (works wonderfully with numeric arrays as well as string arrays) based approach could be easier to follow -
%// Find start and stop indices for ones and zeros with strfind by using
%// "opposite (0 for 1 and 1 for 0) sentients"
start_ones = strfind([0 A],[0 1]) %// 0 is the sentient here and so on
start_zeros = strfind([1 A],[1 0])
stop_ones = strfind([A 0],[1 0])
stop_zeros = strfind([A 1],[0 1])
%// Get lengths of islands of ones and zeros using those start-stop indices
length_ones = stop_ones - start_ones + 1
length_zeros = stop_zeros - start_zeros + 1