Related
Consider array a, holding some of the permutations of 1,2,3,4. (my actual arrays may be larger)
import numpy as np
n = 3
a = np.array([[1, 2, 3, 4],
[1, 2, 4, 3],
[1, 3, 4, 2],
[1, 4, 3, 2],
[2, 3, 4, 1],
[2, 4, 3, 1],
[3, 1, 4, 2],
[4, 1, 2, 3]]))
I want to identify sets of n rows (in this example, n=3) where each row position holds unique values.
In this example, the output would be:
out = [[0, 4, 7],
[2, 5, 7],
[3, 4, 7]]
The 1st row of out indicates that a[0], a[4], and a[7] have unique values in each row position
When n = 2, there are 11 row pairs that match the criteria: [[0,4], [0,6], [0,7], [1,5] ...etc
When n = 4, there are 0 rows that match the criteria.
I'm new enough to python that I can't find a good way to approach this situation.
Solving this problem efficiently is far from not easy. Indeed, the brute-force solution consisting is using n nested loop is very inefficient: its complexity is O(c r! / (r-n)!) where r is the number of rows of a and c is the number of columns of a (note that ! is the factorial). Since r is a number of permutation which already grow experientially with the number of unique items in a, this means the complexity of this solution is really bad.
A more efficient solution (but still not great) is to pick a row, filter the other rows that can match with it (ie. there is no items at the same position that are equal), and then recursively do the same thing n times (the picked row are only the one that are filtered). The several sets of row indices can be appended in a list during the recursion. It is hard to evaluate the complexity of this solution, but it is far much faster in practice since most rows hopefully does not match together and the filtered rows tends to decrease exponentially too. That being said, the complexity is certainly still exponential since the size of the output appears to grow exponentially too and the output needs to be written.
Here is the implementation:
def recursiveFindSets(a, i, n, availableRows, rowIndices, results):
if availableRows.size == 0:
return
for k in availableRows:
# Save the current choice
rowIndices[i] = k
# The next selected rows needs to be bigger than `k` so to prevent replicates
newAvailableRows = availableRows[availableRows > k]
# If there is no solutions with a[k], then choose another
if newAvailableRows.size == 0:
continue
# Find rows that contains different items of a[i]
goodMatches = np.all(a[newAvailableRows] != a[k], axis=1)
# Find the location relative to `a` and not `a[availableRows]`
newAvailableRows = newAvailableRows[goodMatches]
# If there is no solutions with a[k], then choose another
if newAvailableRows.size == 0:
continue
if i == n-2:
# Generate some solutions from `newAvailableRows`
for k2 in newAvailableRows:
rowIndices[i+1] = k2
results.append(rowIndices.copy())
elif i < n-2:
recursiveFindSets(a, i+1, n, newAvailableRows, rowIndices, results)
def findSets(a, n):
availableRows = np.arange(a.shape[0], dtype=int) # Filter
rowIndices = np.empty(n, dtype=np.int_) # Current set of row indices
results = [] # List of all the sets
recursiveFindSets(a, 0, n, availableRows, rowIndices, results)
if len(results) == 0:
return np.empty((0, n), dtype=int)
return np.vstack(results)
findSets(a, 3)
# Output:
# array([[0, 4, 7],
# [2, 5, 7],
# [3, 4, 7]])
You can reduce this problem to finding all cliques of size n in an undirected graph. Nodes in the graph are given by row indices of a. There is an edge between i and j if (a[i] != a[j]).all().
Here is one implementation based on networkx. A function enumerate_all_cliques(g) iterates over cliques in g in order of increasing size. We discard all cliques of size less than n, keep those of size n, and stop once the first clique of size greater than n is found or cliques run out.
from itertools import combinations
import networkx as nx
def f(arr, n):
nodes = np.arange(arr.shape[0])
g = nx.Graph()
g.add_nodes_from(nodes)
for i, j in combinations(nodes, 2):
if (arr[i] != arr[j]).all():
g.add_edge(i, j)
for c in nx.algorithms.clique.enumerate_all_cliques(g):
lc = len(c)
if lc == n:
yield c
elif lc > n:
break
print(list(f(a, 3)))
# [[0, 4, 7], [2, 5, 7], [3, 4, 7]]
Here is another approach: find all maximal cliques and yield all subsets of size n from each clique. This can lead to double-counting, hence set is used before the return statement.
def f(arr, n):
nodes = np.arange(arr.shape[0])
g = nx.Graph()
g.add_nodes_from(nodes)
for i, j in combinations(nodes, 2):
if (arr[i] != arr[j]).all():
g.add_edge(i, j)
cliques = set()
# iterate over maximal cliques
for c in nx.algorithms.clique.find_cliques(g):
# update the clique set subsets of c
cliques.update(map(frozenset, combinations(c, n)))
# return all cliques of size n without doublecounting
return [list(c) for c in cliques]
print(f(a, 3))
# [[2, 5, 7], [3, 4, 7], [0, 4, 7]]
The performance of either approach will vary depending on input values.
I receive 3 integers, "x", "y" and "z". Then a last integer which is "n".
I have to find all possible combinations between x, y and z as in a permutation, but I only need to save those ones where the sum of all values is not equal to n (x + y + z != n).
I have done this trough a big for loop. As you can see, the print output confirms that this logic works, so all the possible combinations are found with any 3 integers you use. Each combination is temporary saved on a list which is constantly overwritten.
I filter the combinations so you can see with the print function only those which sum of the values on the list is not n. However, when I want to save each combination found, it just saves repeated values and it seems that the append method is not working as I need to.
# Example:
#
# Input: x = 1
# y = 1
# z = 1
# n = 2
#
# Output = [[0,0,0],[0,0,1],[0,1,0],[1,0,0],[1,1,1]]
x = int(input())
y = int(input())
z = int(input())
n = int(input())
list_x = [x for x in range(x+1)] #numbers from 0 to x
list_y = [y for y in range(y+1)] #numbers from 0 to y
list_z = [z for z in range(z+1)] #numbers from 0 to z
results = []
#Big for loop which makes the permutation
combination = [0,0,0]
for num_x in list_x:
combination[0] = num_x
for num_y in list_y:
combination[1] = num_y
for num_z in list_z:
combination[2]=num_z
if sum(combination) != n: #Find combinations where total sum != to n
print(combination) #print the combination found
results.append(combination) #save the combination in a list of lists
print(results) #print the lists of lists
Any kind of help is really appreciated. Thank you.
It's because combination is a reference to an underlying list, and that's what you're storing in the results (the reference, not the list). This is one of those "a-ha" moments that all Python coders experience, when they start to fully grok the language :-)
Every time you change an element of the list (as opposed to the actual list), every reference to it seems to change as well, because there's only one list.
This makes it appear that you're changing things that have already been set at an earlier time but that's not actually the case.
You can see that in action with:
>>> xyzzy = [1, 2, 3] # The list.
>>> plugh = xyzzy # Copies reference, not list content.
>>> xyzzy[1] = 99 # Changes list content (for BOTH references).
>>> xyzzy ; plugh # Result: they are both changed.
[1, 99, 3]
[1, 99, 3]
>>> xyzzy = [4, 5, 6] # Makes xyzzy reference point to NEW list.
>>> xyzzy ; plugh # Result: they are different.
[4, 5, 6]
[1, 99, 3]
To fix that, you can just get rid of combination and just use something like:
for num_x in list_x:
for num_y in list_y:
for num_z in list_z:
if num_x + num_y + num_z != n:
results.append([num_x, num_y, num_z])
print(results[-1])
That creates and appends a new list each time, so changes won't seem to travel back through time.
And, for what it's worth, this can also be done with a more Pythonic list comprehension. Using your supplied test data, dummied up:
list_x, list_y, list_z, n = [0, 1], [0, 1], [0, 1], 2
result = [[x, y, z] \
for x in list_x \
for y in list_y \
for z in list_z \
if x + y + z != n]
print(result)
I've split the comprehension across multiple lines for readability but it could just as easily go on one line, and it gives the expected output as per below:
[[0, 0, 0], [0, 0, 1], [0, 1, 0], [1, 0, 0], [1, 1, 1]]
I have the task of selecting p% of elements within a given numpy array. For example,
# Initialize 5 x 3 array-
x = np.random.randint(low = -10, high = 10, size = (5, 3))
x
'''
array([[-4, -8, 3],
[-9, -1, 5],
[ 9, 1, 1],
[-1, -1, -5],
[-1, -4, -1]])
'''
Now, I want to select say p = 30% of the numbers in x, so 30% of numbers in x is 5 (rounded up).
Is there a way to select these 30% of numbers in x? Where p can change and the dimensionality of numpy array x can be 3-D or maybe more.
I am using Python 3.7 and numpy 1.18.1
Thanks
You can use np.random.choice to sample without replacement from a 1d numpy array:
p = 0.3
np.random.choice(x.flatten(), int(x.size * p) , replace=False)
For large arrays, the performance of sampling without replacement can be pretty bad, but there are some workarounds.
You can randome choice 0,1 and usenp.nonzero and boolean indexing:
np.random.seed(1)
x[np.nonzero(np.random.choice([1, 0], size=x.shape, p=[0.3,0.7]))]
Output:
array([ 3, -1, 5, 9, -1, -1])
I found a way of selecting p% of numpy elements:
p = 20
# To select p% of elements-
x_abs[x_abs < np.percentile(x_abs, p)]
# To select p% of elements and set them to a value (in this case, zero)-
x_abs[x_abs < np.percentile(x_abs, p)] = 0
It seems I just cannot solve this in Numpy: I have a matrix, with an arbitrary number of dimensions, ordered in an arbitrary way. Inside this matrix, there is always one dimension I am interested in (as I said, the position of this dimension is not always the same). Now, I want to find the first nonzero value along this dimension. In fact, I need the index of that value to perform some operations on the value itself.
An example: if my matrix a is n x m x p and the dimension I am interested in is number 1, I would do something like:
for ii in xrange(a.shape[0]):
for kk in xrange(a.shape[2]):
myview = np.squeeze(a[ii, :, kk])
firsti = np.nonzero(myview)[0][0]
myview[firsti] = dostuff
Apart from performance considerations, I really do not know how to do this having different number of dimensions, and having the dimension I am interested in an arbitrary position.
You can abuse np.argmax for your purpose. Here, you can specify the axis which you are interested in, where 0 is along columns, 1 is along rows, and so on. You just need an array which contains the same value for all elements that are not zero. You can achieve that by doing a != 0, as this will contain False (meaning 0) for all zero-elements and True (meaning 1) for all non-zero-elements. Now np.argmax(a != 0, axis=1) would give you the first non-zero element along the 1 axis.
For example:
import numpy as np
a = np.array([[0, 1, 4],[1, 0, 2],[0, 0, 1]])
# a = [[0, 1, 4],
# [1, 0, 2],
# [0, 0, 1]]
print(np.argmax(a!=0, axis=0))
# >>> array([1, 0, 0]) -> along columns
print(np.argmax(a!=0, axis=1))
# >>> array([1, 0, 2]) -> along rows
This will also work for higher dimension, but the output is less instructive, as it is hard to imagine.
Few days ago I asked a question how to find the eigenvalues of a large sparse matrix. I got no answers, so I decided to describe a potential solution.
One question remains:
Can I use the python implementation of ARPACK
to compute the eigenvalues of a asymmetric sparse matrix.
At the beginning I would like to say that it is not at all necessary to call the subroutines of ARPACK directly using FOTRAN driver program. That is quite difficult and I never got it going. But one can do the following:
#
OPTION 1: Python
#
One can install numpy and scipy and run the following code:
import numpy as np
from scipy.linalg import eigh
from scipy.sparse.linalg import eigsh
from scipy.sparse import *
from scipy import *
# coordinate format storage of the matrix
# rows
ii = array([0, 0, 1, 1, 1, 2, 2, 2, 3, 3, 4])
# cols.
jj = array([0, 1, 0, 1, 2, 1, 2, 3, 2, 3, 4])
# and the data
data=array([1.,-1.,-1., 2.,-2.,-2., 1., 1., 1., 1., 1.])
# now put this into sparse storage (CSR-format)
m=csr_matrix( (data,(ii,jj)), shape=(5,5) )
# you can check what you did
matrix([[ 1, -1, 0, 0, 0],
[-1, 2, -2, 0, 0],
[ 0, -2, 1, 1, 0],
[ 0, 0, 1, 1, 0],
[ 0, 0, 0, 0, 1]])
# the real part starts here
evals_large, evecs_large = eigsh(m, 4, which='LM')
# print the largest 4 eigenvalues
print evals_all
# and the values are
[-1.04948118 1. 1.48792836 3.90570354]
Well this is all very nice, specially because it spears us the joy of reading the very "well written" manual of ARPACK.
I have a problem with this, I think that it doesn't work with asymmetric matrices. At least comparing the results to matlab was not very convincing.
#
OPTION 2: MATLAB
#
% put your data in a file "matrix.dat"
% row col. data
% note that indexing starts at "1"
1 1 1.
1 2 -1.
......
load matrix.dat
M = spconvert(matrix)
[v,d] = eig(M)
% v - contains the eigenvectors
% d - contains the eigenvalues
I think that using matlab is way simpler and works for asymmetric matrices. Well I have a 500000x500000 sparse matrix, so whether this will work in matlab .... Is another cup of tea! I have to note that using python I was able to load matrix of this size and compute it's eigenvalues without too much of a trouble.
Cheers,