Related
I am trying to manually convert a BGR image to HSV. I need to find the maximum pixel value each of 3 image channels (numPy arrays) and create a new array which contains the maximum of the 3 channels.
def convertBGRtoHSV(image):
# normalize image
scaledImage = image // 256
# split image into 3 channels
B, G, R = cv2.split(scaledImage)
# find the shape of each array
heightB, widthB = B.shape
V = []
for h_i in range(0, height):
for w_i in range(0, width):
V[h_i][w_i] = max(B[h_i][w_i], G[h_i][w_i], R[h_i][w_i])
I am getting this error: IndexError: list index out of range
I know this loop is incorrect. I know to access the value of a pixel in an array you must say the location as such as x[:,:] but I am not sure how to loop over all the pixels of each image and make a new array with the max value of each array element.
If possible I would like to know how to use a numPy "Vectorized Operation" to accomplish this as well as the for loop.
There is a builtin function for element-wise maximum:
V = np.maximum(np.maximum(R, G), B)
... and you are done
Following up on my comment:
import cv2
import numpy as np
image = cv2.imread(image)
height, width, _ = image.shape
# initialize your output array 'v'
v = np.zeros((height, width))
# loop over each index in ranges dictated by the image shape
for row in range(height):
for col in range(width):
# assign the maximum value across the 3rd dimension (color channel)
# from the original image to your output array
v[row, col] = max(image[row, col, :])
I'm essentially trying to accomplish this and then this but with a 3D matrix, say (128,128,60,6). The 4th dimension is an array vector that represents the diffusion array at that voxel, e.g.:
d[30,30,30,:] = [dxx, dxy, dxz, dyy, dyz, dzz] = D_array
Where dxx etc. are diffusion for a particular direction. D_array can also be seen as a triangular matrix (since dxy == dyx etc.). So I can use those 2 other answers to get from D_array to D_square, e.g.
D_square = [[dxx, dxy, dxz], [dyx, dyy, dyz],[dzx, dzy, dzz]]
I can't seem to figure out the next step however - how to apply that unit transformation of a D_array into D_square to the whole 3D volume.
Here's the code snippet that works on a single tensor:
#this solves an linear eq. that provides us with diffusion arrays at each voxel in a 3D space
D = np.einsum('ijkt,tl->ijkl',X,bi_plus)
#our issue at this point is we have a vector that represents a triangular matrix.
# first make a tri matx from the vector, testing on unit tensor first
D_tri = np.zeros((3,3))
D_array = D[30][30][30]
D_tri[np.triu_indices(3)] = D_array
# then getting the full sqr matrix
D_square = D_tri.T + D_tri
np.fill_diagonal(D_square, np.diag(D_tri))
So what would be the numpy-way of formulating that unit transformation of the Diffusion tensor to the whole 3D volume all at once?
Approach #1
Here's one using row, col indices from triu_indices for indexing along last two axes into an initialized output array -
def squareformnd_rowcol_integer(ar, n=3):
out_shp = ar.shape[:-1] + (n,n)
out = np.empty(out_shp, dtype=ar.dtype)
row,col = np.triu_indices(n)
# Get a "rolled-axis" view with which the last two axes come to the front
# so that we could index into them just like for a 2D case
out_rolledaxes_view = out.transpose(np.roll(range(out.ndim),2,0))
# Assign permuted version of input array into rolled output version
arT = np.moveaxis(ar,-1,0)
out_rolledaxes_view[row,col] = arT
out_rolledaxes_view[col,row] = arT
return out
Approach #2
Another one with the last two axes merged into one and then indexing with linear indices -
def squareformnd_linear_integer(ar, n=3):
out_shp = ar.shape[:-1] + (n,n)
out = np.empty(out_shp, dtype=ar.dtype)
row,col = np.triu_indices(n)
idx0 = row*n+col
idx1 = col*n+row
ar2D = ar.reshape(-1,ar.shape[-1])
out.reshape(-1,n**2)[:,idx0] = ar2D
out.reshape(-1,n**2)[:,idx1] = ar2D
return out
Approach #3
Finally altogether a new method using masking and should be better with performance as most masking based ones are when it comes to indexing -
def squareformnd_masking(ar, n=3):
out = np.empty((n,n)+ar.shape[:-1] , dtype=ar.dtype)
r = np.arange(n)
m = r[:,None]<=r
arT = np.moveaxis(ar,-1,0)
out[m] = arT
out.swapaxes(0,1)[m] = arT
new_axes = range(out.ndim)[2:] + [0,1]
return out.transpose(new_axes)
Timings on (128,128,60,6) shaped random array -
In [635]: ar = np.random.rand(128,128,60,6)
In [636]: %timeit squareformnd_linear_integer(ar, n=3)
...: %timeit squareformnd_rowcol_integer(ar, n=3)
...: %timeit squareformnd_masking(ar, n=3)
10 loops, best of 3: 103 ms per loop
10 loops, best of 3: 103 ms per loop
10 loops, best of 3: 53.6 ms per loop
A vectorized way to do it:
# Gets the triangle matrix
d_tensor = np.zeros(128, 128, 60, 3, 3)
triu_idx = np.triu_indices(3)
d_tensor[:, :, :, triu_idx[0], triu_idx[1]] = d
# Make it symmetric
diagonal = np.zeros(128, 128, 60, 3, 3)
idx = np.arange(3)
diagonal[:, :, :, idx, idx] = d_tensor[:, :, :, idx, idx]
d_tensor = np.transpose(d_tensor, (0, 1, 2, 4, 3)) + d_tensor - diagonal
I have two numpy arrays of shape arr1=(~140000, 3) and arr2=(~450000, 10). The first 3 elements of each row, for both the arrays, are coordinates (z,y,x). I want to find the rows of arr2 that have the same coordinates of arr1 (which can be considered a subgroup of arr2).
for example:
arr1 = [[1,2,3],[1,2,5],[1,7,8],[5,6,7]]
arr2 = [[1,2,3,7,66,4,3,44,8,9],[1,3,9,6,7,8,3,4,5,2],[1,5,8,68,7,8,13,4,53,2],[5,6,7,6,67,8,63,4,5,20], ...]
I want to find common coordinates (same first 3 elements):
list_arr = [[1,2,3,7,66,4,3,44,8,9], [5,6,7,6,67,8,63,4,5,20], ...]
At the moment I'm doing this double loop, which is extremely slow:
list_arr=[]
for i in arr1:
for j in arr2:
if i[0]==j[0] and i[1]==j[1] and i[2]==j[2]:
list_arr.append (j)
I also tried to create (after the 1st loop) a subarray of arr2, filtering it on the value of i[0] (arr2_filt = [el for el in arr2 if el[0]==i[0]). This speed a bit the operation, but it still remains really slow.
Can you help me with this?
Approach #1
Here's a vectorized one with views -
# https://stackoverflow.com/a/45313353/ #Divakar
def view1D(a, b): # a, b are arrays
a = np.ascontiguousarray(a)
b = np.ascontiguousarray(b)
void_dt = np.dtype((np.void, a.dtype.itemsize * a.shape[1]))
return a.view(void_dt).ravel(), b.view(void_dt).ravel()
a,b = view1D(arr1,arr2[:,:3])
out = arr2[np.in1d(b,a)]
Approach #2
Another with dimensionality-reduction for ints -
d = np.maximum(arr2[:,:3].max(0),arr1.max(0))
s = np.r_[1,d[:-1].cumprod()]
a,b = arr1.dot(s),arr2[:,:3].dot(s)
out = arr2[np.in1d(b,a)]
Improvement #1
We could use np.searchsorted to replace np.in1d for both of the approaches listed earlier -
unq_a = np.unique(a)
idx = np.searchsorted(unq_a,b)
idx[idx==len(a)] = 0
out = arr2[unq_a[idx] == b]
Improvement #2
For the last improvement on using np.searchsorted that also uses np.unique, we could use argsort instead -
sidx = a.argsort()
idx = np.searchsorted(a,b,sorter=sidx)
idx[idx==len(a)] = 0
out = arr2[a[sidx[idx]]==b]
You can do it with the help of set
arr = np.array([[1,2,3],[4,5,6],[7,8,9]])
arr2 = np.array([[7,8,9,11,14,34],[23,12,11,10,12,13],[1,2,3,4,5,6]])
# create array from arr2 with only first 3 columns
temp = [i[:3] for i in arr2]
aset = set([tuple(x) for x in arr])
bset = set([tuple(x) for x in temp])
np.array([x for x in aset & bset])
Output
array([[7, 8, 9],
[1, 2, 3]])
Edit
Use list comprehension
l = [list(i) for i in arr2 if i[:3] in arr]
print(l)
Output:
[[7, 8, 9, 11, 14, 34], [1, 2, 3, 4, 5, 6]]
For integers Divakar already gave an excellent answer. If you want to compare floats you have to consider e.g. the following:
1.+1e-15==1.
False
1.+1e-16==1.
True
If this behaviour could lead to problems in your code I would recommend to perform a nearest neighbour search and probably check if the distances are within a specified threshold.
import numpy as np
from scipy import spatial
def get_indices_of_nearest_neighbours(arr1,arr2):
tree=spatial.cKDTree(arr2[:,0:3])
#You can check here if the distance is small enough and otherwise raise an error
dist,ind=tree.query(arr1, k=1)
return ind
I have a file which contains a 209091 element 1D binary array representing the global land area
which can be downloaded from here:
ftp://sidads.colorado.edu/DATASETS/nsidc0451_AMSRE_Land_Parms_v01/AMSRE_flags_2002/
I want to create a full from the 1D data arrays using provided ancillary row and column files .globland_r and globland_c which can be downloaded from here:
ftp://sidads.colorado.edu/DATASETS/nsidc0451_AMSRE_Land_Parms_v01/AMSRE_ancil/
There is a code written in Matlab for this purpose and I want to translate this Matlab code to R but I do not know Matlab
function [gridout, EASE_r, EASE_s] = mkgrid_global(x)
%MKGRID_GLOBAL(x) Creates a matrix for mapping
% gridout = mkgrid_global(x) uses the 2090887 element array (x) and returns
%Load ancillary EASE grid row and column data, where <MyDir> is the path to
%wherever the globland_r and globland_c files are located on your machine.
fid = fopen('C:\MyDir\globland_r','r');
EASE_r = fread(fid, 209091, 'int16');
fclose(fid);
fid = fopen('C:\MyDir\globland_c','r');
EASE_s = fread(fid, 209091, 'int16');
fclose(fid);
gridout = NaN.*zeros(586,1383);
%Loop through the elment array
for i=1:1:209091
%Distribute each element to the appropriate location in the output
%matrix (but MATLAB is
%(1,1)
end
EDit following the solution of #mdsumner:
The files MLLATLSB and MLLONLSB (4-byte integers) contain latitude and longitude (multiply by 1e-5) for geo-locating the full global EASE grid matrix (586×1383)
MLLATLSB and MLLONLSB can be downloaded from here:
ftp://sidads.colorado.edu/DATASETS/nsidc0451_AMSRE_Land_Parms_v01/AMSRE_ancil/
## the sparse dims, literally the xcol * yrow indexes
dims <- c(1383, 586)
cfile <- "ftp://sidads.colorado.edu/DATASETS/nsidc0451_AMSRE_Land_Parms_v01/AMSRE_ancil/globland_c"
rfile <- "ftp://sidads.colorado.edu/DATASETS/nsidc0451_AMSRE_Land_Parms_v01/AMSRE_ancil/globland_r"
## be nice, don't abuse this
col <- readBin(cfile, "integer", n = prod(dims), size = 2, signed = FALSE)
row <- readBin(rfile, "integer", n = prod(dims), size = 2, signed = FALSE)
## example data file
fdat <- "ftp://sidads.colorado.edu/DATASETS/nsidc0451_AMSRE_Land_Parms_v01/AMSRE_flags_2002/flags_2002170A.bin"
dat <- readBin(fdat, "integer", n = prod(dims), size = 1, signed = FALSE)
## now get serious
m <- matrix(as.integer(NA), dims[2L], dims[1L])
m[cbind(row + 1L, col + 1L)] <- dat
image(t(m)[,dims[2]:1], col = rainbow(length(unique(m)), alpha = 0.5))
Maybe we can reconstruct this map projection too.
flon <- "MLLONLSB"
flat <- "MLLATLSB"
## the key is that these are integers, floats scaled by 1e5
lon <- readBin(flon, "integer", n = prod(dims), size = 4) * 1e-5
lat <- readBin(flat, "integer", n = prod(dims), size = 4) * 1e-5
## this is all we really need from now on
range(lon)
range(lat)
library(raster)
library(rgdal) ## need for coordinate transformation
ex <- extent(projectExtent(raster(extent(range(lon), range(lat)), crs = "+proj=longlat"), "+proj=cea"))
grd <- raster(ncols = dims[1L], nrows = dims[2L], xmn = xmin(ex), xmx = xmax(ex), ymn = ymin(ex), ymx = ymax(ex), crs = "+proj=cea")
There is probably an "out by half pixel" error in there, left as an exercise.
Test
plot(setValues(grd, m), col = rainbow(max(m, na.rm = TRUE), alpha = 0.5))
Hohum
library(maptools)
data(wrld_simpl)
plot(spTransform(wrld_simpl, CRS(projection(grd))), add = TRUE)
We can now save the valid cellnumbers to match our "grd" template, then read any particular dat-file and just populate the template with those values based on cellnumbers. Also, it seems someone trod nearly this path earlier but not much was gained:
How to identify lat and long for a global matrix?
I have a set of around 500 (x,y,z) real values. Since I will need to bin the values based on their (x,y) coordinates, I stripped the z values and stored in on a seperate list. I am left with only the x,y values; I rescaled and rounded them to index pairs in the range of, 1..100 range.
Now I want to populate an array with the z values in a 100x100 matrix at the particular (x,y) coordinates.
More precisely,
I have a set of values for example : data = {{2.62399, 0.338057, 2.09629}, {1.8424, 0.135817, 3.21925}, {0.702257, 1.14502, 3.9335}...
I stripped it of its zvalues and store it in zvalues list:
zvalues = {2.09629, 3.21925, 3.9335....
I rounded, rescaled and created a new array of indices
indices = {{53, 7}, {37, 3}, {14, 23}...
I want to create a new 100x100 matrix and place the zvalues on the coordinates corresponding to the indices matrix
For example, in pseudocode
For (int i = 1, i < 101, i++){
NewArray(indices[i]) = zvalues[i];
}
The first time the loop will run, it should do NewArray(53,7) = 2.09629.
I want to know the syntax to loop through the indices array and populate the 2 dimensional 100x100 NewArray with zvalues
to follow your basic approach you need to initialize the array:
newArray=Table[,{100},{100}]
then in the loop the syntax is:
newArray[[indices[[i,1]],indices[[i,2]]]]=zdata[[i]]
note the double square brackets for referencing parts of arrays (or lists in Mathematica terminology)
A better approach would be to create a SparseArray, which for one thing would not require pre-initialization, or even knowing the dimensions in advance.
Finally in mathematica you can usually use an object oriented approach, avioding the "do" loop all together:
data = {{1.5, 1.1, 1.1}, {2.2, 2.2, 2.2}, {1.01, 2.3, 1.2}};
m1 = Table[, {2}, {2}];
(m1[[Floor[#[[1]]], Floor[#[[2]]]]] = #[[3]]) & /# data;
m1
m2 = SparseArray[ Floor[#[[1 ;; 2]]] -> #[[3]] & /# data , Automatic,];
Normal[m2]
{{1.1, 1.2}, {Null, 2.2}}
{{1.1, 1.2}, {Null, 2.2}}
While I don't understand why you want to create a new way of indexing your array, this will do what you want :
data = {{2.62399, 0.338057, 2.09629}, {1.8424, 0.135817, 3.21925}, {0.702257, 1.14502, 3.9335}};
zvalues = {2.09629, 3.21925, 3.9335};
indices = {{53, 7}, {37, 3}, {14, 23}};
newArray[xIndex_, yIndex_]:=Take[data, Position[indices, {xIndex, yIndex}][[1, 1]]][[1, 3]]
newArray[53, 7]
(* 2.09629 *)