I have an array in Matlab. I numbered every entry in array with natural number. So I formed equivalence relation in array.
For example,
array = [1 2 3 5 6 7]
classes = [1 2 1 1 3 3].
I want to get cell array: i-th cell array's position is connected with i-th entry of initial array and shows, which elements are in the one class with this entry. For the example above, I would get:
{[1 3 5], [2], [1 3 5], [1 3 5], [6 7], [6 7]}
It can be done easily with for-loop, but is there any other solution? It will be good if it works faster than O(n^2), where n is the size of initial array.
Edit.
Problem will be solved, if I know the approach to split sorted array into cells with indeces of equal elements by O(n).
array = [1 1 1 2 3 3]
groups = {[1 2 3], [4], [5 6]}
Not sure about complexity, but accumarray with cell output is useful for splitting up the array based on unique values of the classes:
data = sortrows([classes; array].',1) %' stable w.r.t. array
arrayPieces = accumarray(data(:,1),data(:,2)',[],#(x){x.'})
classElements = arrayPieces(classes).'
Regarding sorted array splitting into cells of indeces:
>> array = [1 1 1 2 3 3]
>> arrayinds = accumarray(array',1:numel(array),[],#(x){x'})' %' transpose for rows
arrayinds =
[1x3 double] [4] [1x2 double]
>> arrayinds{:}
ans =
1 2 3
ans =
4
ans =
5 6
I don't know how to do this without for-loops entirely, but you can use a combination of sort, diff, and find to organize and partition the equivalence class identifiers. That'll give you a mostly vectorized solution, where the M-code level for-loop is O(n) where n is the number of classes, not the length of the whole input array. This should be pretty fast in practice.
Here's a rough example using some index munging. Be careful; there's probably an off-by-one edge case bug in there somewhere since I just banged this out.
function [eqVals,eqIx] = equivsets(a,x)
%EQUIVSETS Find indexes of equivalent values
[b,ix] = sort(x);
ixEdges = find(diff(b)); % identifies partitions between equiv classes
ix2 = [0 ixEdges numel(ix)];
eqVals = cell([1 numel(ix2)-1]);
eqIx = cell([1 numel(ix2)-1]);
% Map back to original input indexes and values
for i = 1:numel(ix2)-1
eqIx{i} = ix((ix2(i)+1):ix2(i+1));
eqVals{i} = a(eqIx{i});
end
I included the indexes in the output because they're often more useful than the values themselves. You'd call it like this.
% Get indexes of occurrences of each class
equivs = equivsets(array, classes)
% You can expand that to get equivalences for each input element
equivsByValue = equivs(classes)
It's a lot more efficient to build the lists for each class first and then expand them out to match the input indexes. Not only do you have to do the work just once, but when you use the b = a(ix) to expand a small cell array to a larger one, Matlab's copy-on-write optimization will end up reusing the memory for the underlying numeric mxArrays so you get a more compact representation in memory.
This transformation pops up a lot when working with unique() or databases. For decision support systems and data warehouse style things I've worked with, it happens all over the place. I wish it were built in to Matlab. (And maybe it's been added to one of the db or timeseries toolboxes in recent years; I'm a few versions behind.)
Realistically, if performance of this is critical for your code, you might also look at dropping down to Java or C MEX functions and implementing it there. But if your data sets are low cardinality - that is, have a small number of classes/distinct values, like numel(unique(classes)) / numel(array) tends to be less than 0.1 or so - the M-code implementation will probably be just fine.
For the second question:
array = [1 1 1 2 3 3]; %// example data
Use diff to find the end of each run of equal values, and from that build the groups:
ind = [0 find(diff([array NaN])~=0)];
groups = arrayfun(#(n) ind(n)+1:ind(n+1), 1:numel(ind)-1, 'uni', 0);
Same approach using unique:
[~, ind] = unique(array);
ind = [0 ind];
groups = arrayfun(#(n) ind(n)+1:ind(n+1), 1:numel(ind)-1, 'uni', 0);
I haven't tested if the complexity is O(n), though.
Related
Say that I have a batch of arrays, and I would like to alter them based on conditions of particular values located by indices.
For example, say that I would like to increase and decrease particular values if the difference between those values are less than two.
For a single 1D array it can be done like this
import numpy as np
single2 = np.array([8, 8, 9, 10])
if abs(single2[1]-single2[2])<2:
single2[1] = single2[1] - 1
single2[2] = single2[2] + 1
single2
array([ 8, 7, 10, 10])
But I do not know how to do it for batch of arrays. This is my initial attempt
import numpy as np
single1 = np.array([6, 0, 3, 7])
single2 = np.array([8, 8, 9, 10])
single3 = np.array([2, 15, 15, 20])
batch = np.array([
np.copy(single1),
np.copy(single2),
np.copy(single3),
])
if abs(batch[:,1]-batch[:,2])<2:
batch[:,1] = batch[:,1] - 1
batch[:,2] = batch[:,2] + 1
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
Looking at np.any and np.all, they are used to create an array of booleans values, and I am not sure how they could be used in the code snippet above.
My second attempt uses np.where, using the method described here for comparing particular values of a batch of arrays by creating new versions of the arrays with values added to the front/back of the arrays.
https://stackoverflow.com/a/71297663/3259896
In the case of the example, I am comparing values that are right next to each other, so I created copies that shift the arrays forwards and backwards by 1. I also use only the particular slice of the array that I am comparing, since the other numbers would also be used in the comparison in np.where.
batch_ap = np.concatenate(
(batch[:, 1:2+1], np.repeat(-999, 3).reshape(3,1)),
axis=1
)
batch_pr = np.concatenate(
(np.repeat(-999, 3).reshape(3,1), batch[:, 1:2+1]),
axis=1
)
Finally, I do the comparisons, and adjust the values
batch[:, 1:2+1] = np.where(
abs(batch_ap[:,1:]-batch_ap[:,:-1])<2,
batch[:, 1:2+1]-1,
batch[:, 1:2+1]
)
batch[:, 1:2+1] = np.where(
abs(batch_pr[:,1:]-batch_pr[:,:-1])<2,
batch[:, 1:2+1]+1,
batch[:, 1:2+1]
)
print(batch)
[[ 6 0 3 7]
[ 8 7 10 10]
[ 2 14 16 20]]
Though I am not sure if this is the most computationally efficient nor programmatically elegant method for this task. Seems like a lot of operations and code for the task, but I do not have a strong enough mastery of numpy to be certain about this.
This works
mask = abs(batch[:,1]-batch[:,2])<2
batch[mask,1] -= 1
batch[mask,2] += 1
I have a strange feeling this is a very easy problem to solve but I'm not finding a good way of doing this without using brute force or dynamic programming. Here it goes:
Given N arrays of ordered and monotonic values, find the set of positions for each array i1, i2 ... in that minimises pair-wise difference of values at those indexes between all arrays. In other words, find the positions for all arrays whose values are closest to each other. Multiple solutions may exist and arrays may or may not be equally sized.
If A denotes the list of all arrays, the pair-wise difference is given by the sum of absolute differences between all values at the given indexes between all different arrays, as so:
An example, 3 arrays a, b and c:
a = [20 29 30 32 33]
b = [28 29 30 32 33]
c = [10 12 28 31 32 33]
The best alignment for this array would be a[3] b[3] c[4] or a[4] b[4] c[5], because (32,32,32) and (33,33,33) are all equal values and have, therefore minimum pairwise difference between each other. (Assuming array index starts at 0)
This is a common problem in bioinformatics thats usually solved with Dynamic Programming, but due to the fact this is an ordered sequence, I think there's somehow a way of exploiting this notion of order. I first thought about doing this pairwise, but this does not guarantee the global optimum because the best local answer might not be the best global answer.
This is meant to be language agnostic, but I don't really mind an answer for a specific language, as long as there is no loss of generality. I know Dynamic Programming is an option here, but I have a feeling there's an easier way to do this?
The tricky thing is parsing the arrays so that at some point you're guaranteed to be considering the set of indices that realize the pairwise min. Using a min heap on the values doesn't work. Counterexample with 4 arrays: [0,5], [1,2], [2], [2]. We start with a d(0,1,2,2) = 7, optimal is d(0,2,2,2) = 6, but the min heap moves us from 7 to d(5,1,2,2) = 12, then d(5,2,2,2) = 9.
I believe (but haven't proved) that if we alway increment the index that improves pairwise distance the most (or degrades it the least), we're guaranteed to visit every local min and the global min.
Assuming n total elements across k arrays:
Simple approach: we repeatedly get the pairwise distance deltas (delta wrt. incrementing each index), increment the best one, and any time doing so switch us from improvement to degradation (i.e. a local minimum) we calculate the pairwise distance. All this is O(k^2) per increment for a total running time of O((n-k) * (k^2)).
With O(k^2) storage, we could keep an array where (i,j) stores the pairwise distance delta achieve by increment the index of array i wrt. array j. We also store the column sums. Then on incrementing an index we can update the appropriate row & column & column sums in O(k). This gives us a running time of O((n-k)*k)
To just complete Dave's answer, here is the pseudocode of the delta algorithm:
initialise index_table to 0's where each row i denotes the index for the ith array
initialise delta_table with the corresponding cost of incrementing index of ith array and keeping the other indexes at their current values
cur_cost <- cost of current index table
best_cost <- cur_cost
best_solutions <- list with the current index table
while (can_at_least_one_index_increase)
i <- index whose delta is lowest
increment i-th entry of the index_table
if cost(index_table) < cur_cost
cur_cost = cost(index_table)
best_solutions = {} U {index_table}
if cost(index_table) = cur_cost
best_solutions = best_solutions U {index_table}
update delta_table
Important Note: During an iteration, some index_table entries might have already reached the maximum value for that array. Whenever updating the delta_table, it is necessary to never pick those values, otherwise this will result in a Array Out of Bounds,Segmentation Fault or undefined behaviour. A neat trick is to simply check which indexes are already at max and set a sufficiently large value, so they are never picked. If no index can increase anymore, the loop will end.
Here's an implementation in Python:
def align_ordered_sequences(arrays: list):
def get_cost(index_table):
n = len(arrays)
if n == 1:
return 0
sum = 0
for i in range(0, n-1):
for j in range(i+1, n):
v1 = arrays[i][index_table[i]]
v2 = arrays[j][index_table[j]]
sum += math.sqrt((v1 - v2) ** 2)
return sum
def compute_delta_table(index_table):
# Initialise the delta table: we switch each index element to 1, call
# the cost method and then revert the change, this avoids having to
# create copies, which decreases performance unnecessarily
delta_table = []
for i in range(n):
if index_table[i] + 1 >= len(arrays[i]):
# Implementation detail: if the index is outside the bounds of
# array i, choose a "large enough" number
delta_table.append(999999999999999)
else:
index_table[i] = index_table[i] + 1
delta_table.append(get_cost(index_table))
index_table[i] = index_table[i] - 1
return delta_table
def can_at_least_one_index_increase(index_table):
answer = False
for i in range(len(arrays)):
if index_table[i] < len(arrays[i]) - 1:
answer = True
return answer
n = len(arrays)
index_table = [0] * n
delta_table = compute_delta_table(index_table)
best_solutions = [index_table.copy()]
cur_cost = get_cost(index_table)
best_cost = cur_cost
while can_at_least_one_index_increase(index_table):
i = delta_table.index(min(delta_table))
index_table[i] = index_table[i] + 1
new_cost = get_cost(index_table)
# A new best solution was found
if new_cost < cur_cost:
cur_cost = new_cost
best_solutions = [index_table.copy()]
# A new solution with the same cost was found
elif new_cost == cur_cost:
best_solutions.append(index_table.copy())
# Update the delta table
delta_table = compute_delta_table(index_table)
return best_solutions
And here are some examples:
>>> print(align_ordered_sequences([[0,5], [1,2], [2], [2]]))
[[0, 1, 0, 0]]
>> print(align_ordered_sequences([[3, 5, 8, 29, 40, 50], [1, 4, 14, 17, 29, 50]]))
[[3, 4], [5, 5]]
Note 2: this outputs indexes not the actual values of each array.
I am sure this question must be answered somewhere else but I can't seem to find the answer.
Given a matrix M, what is the most efficient/succinct way to return two matrices respectively containing the row and column indices of the elements of M.
E.g.
M = [1 5 ; NaN 2]
and I want
MRow = [1 1; 2 2]
MCol = [1 2; 1 2]
One way would be to do
[MRow, MCol] = find(ones(size(M)))
MRow = reshape(MRow, size(M))
MCol = reshape(MCol, size(M))
But this does not seem particular succinct nor efficient.
This essentially amounts to building a regular grid over possible values of row and column indices. It can be achieved using meshgrid, which is more effective than using find as it avoids building the matrix of ones and trying to "find" a result that is essentially already known.
M = [1 5 ; NaN 2];
[nRows, nCols] = size(M);
[MCol, MRow] = meshgrid(1:nCols, 1:nRows);
Use meshgrid:
[mcol, mrow] = meshgrid(1:size(M,2),1:size(M,1))
I am currently looking for an efficient way to slice multidimensional matrices in MATLAB. Ax an example, say I have a multidimensional matrix such as
A = rand(10,10,10)
I would like obtain a subset of this matrix (let's call it B) at certain indices along each dimension. To do this, I have access to the index vectors along each dimension:
ind_1 = [1,4,5]
ind_2 = [1,2]
ind_3 = [1,2]
Right now, I am doing this rather inefficiently as follows:
N1 = length(ind_1)
N2 = length(ind_2)
N3 = length(ind_3)
B = NaN(N1,N2,N3)
for i = 1:N1
for j = 1:N2
for k = 1:N3
B(i,j,k) = A(ind_1(i),ind_2(j),ind_3(k))
end
end
end
I suspect there is a smarter way to do this. Ideally, I'm looking for a solution that does not use for loops and could be used for an arbitrary N dimensional matrix.
Actually it's very simple:
B = A(ind_1, ind_2, ind_3);
As you see, Matlab indices can be vectors, and then the result is the Cartesian product of those vector indices. More information about Matlab indexing can be found here.
If the number of dimensions is unknown at programming time, you can define the indices in a cell aray and then expand into a comma-separated list:
ind = {[1 4 5], [1 2], [1 2]};
B = A(ind{:});
You can reference data in matrices by simply specifying the indices, like in the following example:
B = A(start:stop, :, 2);
In the example:
start:stop gets a range of data between two points
: gets all entries
2 gets only one entry
In your case, since all your indices are 1D, you could just simply use:
C = A(x_index, y_index, z_index);
function prealloc()
situation=zeros(Int64,3^5,5);
i=1;
for north=0:2, south=0:2, east=0:2, west=0:2, current=0:2
situation[i,:]=[north, south, east, west, current]
if situation[i,:]=[2, 2, 2, 2, 2]
elseif situation[i,:]=[2, 2, 2, 2, 1]
elseif situation[i,:]=[2, 2, 2, 2, 0]`enter code here`
end
i+=1
end
situation
end
How can I eliminate the row which equal those if conditions from the array which called situation
First things first: the code in your question doesn't run (for several reasons). When posting code in questions, it is good form to put it in a "working example" form, where users can copy and paste it into their editor of choice and it will work without the user having to make educated guesses as to what you are actually trying to do. This is probably one reason the question has received down-votes.
With that out of the way, there are two approaches to accomplish what you are trying to do:
1) Construct your matrix without the indicated rows in the first step. Then you don't need to worry about "deleting the rows" later on. For situations as simple as the one in the question, you could just do something like this:
function prealloc()
x = zeros(Int, 3^5 - 3, 5)
i = 1
for n=0:2, s=0:2, ea=0:2, w=0:2, cur=0:2
if !([n, s, ea, w, cur] == [2, 2, 2, 2, 2] || [n, s, ea, w, cur] == [2, 2, 2, 2, 1] || [n, s, ea, w, cur] == [2, 2, 2, 2, 0])
x[i, :] = [n, s, ea, w, cur]
i += 1
end
end
return(x)
end
Notice I'm using Int, not Int64. This will not affect performance, and it means your code will run on both 32-bit and 64-bit architectures.
Another style tip. Don't use semi-colons to end lines. This is a Matlab quirk, and it is not needed in Julia.
2) As other users have suggested, you could construct the entire matrix (including the undesirable rows), and then remove them at a later point. Of course, this necessitates re-allocating the entire matrix, and so is somewhat inefficient (note, you can remove elements of vectors in place, i.e. without re-allocation, but not any arrays of dimension 2 or greater). In this case, to encourage code re-use, it makes sense to break the routine down into three separate functions. First, we allocate the entire matrix:
function prealloc1()
x = zeros(Int64,3^5,5)
i = 1
for north=0:2, south=0:2, east=0:2, west=0:2, current=0:2
x[i,:]=[north, south, east, west, current]
i += 1
end
return(x)
end
Next, we obtain a vector of indices that we wish to remove. We do this as its own step because we only want to re-allocate the matrix once, rather than re-allocating every time we find a new row we want to delete. For your situation, you could use a function like this:
function findCondition(x::Matrix{Int})
inds = Array(Int, 0)
for i = 1:size(x, 1)
if x[i, :] == [2 2 2 2 2]
push!(inds, i)
elseif x[i, :] == [2 2 2 2 1]
push!(inds, i)
elseif x[i, :] == [2 2 2 2 0]
push!(inds, i)
end
end
return(inds)
end
Notice that in my comparison statements in this function I use [2 2 2 2 2] instead of [2, 2, 2, 2, 2]. This is because the first construct is a 2-dimensional array (type Matrix) while the second is 1-dimensional (type Vector). Since x[i, :] is of type Matrix, the difference is important.
Finally, we need to re-allocate the matrix without the offending rows. As user #Matt B. suggests, this can be done with the following one-liner function:
removeIndices(x::Matrix{Int}, inds::Vector{Int}) = x[setdiff(IntSet(1:size(x, 1)), IntSet(inds)), :]
Note, applying setdiff to IntSet here is fast because by construction inds will already be sorted in ascending order.
You cannot just delete a row in Julia, the only way to do it, is to create a copy of the array without the row you want to delete. And I think that's not internally implemented and it's intentional.
So you will have to do to it manually, something like this will create a copy of situation without the row i (which is not the same as saying that it will delete row i).
situation = vcat(situation[1:i-1,:],situation[i+1:end,:])
also, this will actually change the dimensions of situation in each iteration, so be careful with that...
Also2, your loop will finish in a bounds error since eventually it will be off limits of your array, maybe you could write something like this to end your loop.
if i = length(situation)
break
else
i += 1
end
Ultimately, you can make a function delrow and call it from within your loop:
function delrow(array,row)
return vcat(array[1:row-1,:],array[row+1:end,:])
end
then call situation = delrow(situation,i)