Spent a while this morning looking for a generalized question to point duplicates to for questions about as_strided and/or how to make generalized window functions. There seem to be a lot of questions on how to (safely) create patches, sliding windows, rolling windows, tiles, or views onto an array for machine learning, convolution, image processing and/or numerical integration.
I'm looking for a generalized function that can accept a window, step and axis parameter and return an as_strided view for over arbitrary dimensions. I will give my answer below, but I'm interested if anyone can make a more efficient method, as I'm not sure using np.squeeze() is the best method, I'm not sure my assert statements make the function safe enough to write to the resulting view, and I'm not sure how to handle the edge case of axis not being in ascending order.
DUE DILIGENCE
The most generalized function I can find is sklearn.feature_extraction.image.extract_patches written by #eickenberg (as well as the apparently equivalent skimage.util.view_as_windows), but those are not well documented on the net, and can't do windows over fewer axes than there are in the original array (for example, this question asks for a window of a certain size over just one axis). Also often questions want a numpy only answer.
#Divakar created a generalized numpy function for 1-d inputs here, but higher-dimension inputs require a bit more care. I've made a bare bones 2D window over 3d input method, but it's not very extensible.
EDIT JAN 2020: Changed the iterable return from a list to a generator to save memory.
EDIT OCT 2020: Put the generator in a separate function, since mixing generators and return statements doesn't work intiutively.
Here's the recipe I have so far:
def window_nd(a, window, steps = None, axis = None, gen_data = False):
"""
Create a windowed view over `n`-dimensional input that uses an
`m`-dimensional window, with `m <= n`
Parameters
-------------
a : Array-like
The array to create the view on
window : tuple or int
If int, the size of the window in `axis`, or in all dimensions if
`axis == None`
If tuple, the shape of the desired window. `window.size` must be:
equal to `len(axis)` if `axis != None`, else
equal to `len(a.shape)`, or
1
steps : tuple, int or None
The offset between consecutive windows in desired dimension
If None, offset is one in all dimensions
If int, the offset for all windows over `axis`
If tuple, the steps along each `axis`.
`len(steps)` must me equal to `len(axis)`
axis : tuple, int or None
The axes over which to apply the window
If None, apply over all dimensions
if tuple or int, the dimensions over which to apply the window
gen_data : boolean
returns data needed for a generator
Returns
-------
a_view : ndarray
A windowed view on the input array `a`, or `a, wshp`, where `whsp` is the window shape needed for creating the generator
"""
ashp = np.array(a.shape)
if axis != None:
axs = np.array(axis, ndmin = 1)
assert np.all(np.in1d(axs, np.arange(ashp.size))), "Axes out of range"
else:
axs = np.arange(ashp.size)
window = np.array(window, ndmin = 1)
assert (window.size == axs.size) | (window.size == 1), "Window dims and axes don't match"
wshp = ashp.copy()
wshp[axs] = window
assert np.all(wshp <= ashp), "Window is bigger than input array in axes"
stp = np.ones_like(ashp)
if steps:
steps = np.array(steps, ndmin = 1)
assert np.all(steps > 0), "Only positive steps allowed"
assert (steps.size == axs.size) | (steps.size == 1), "Steps and axes don't match"
stp[axs] = steps
astr = np.array(a.strides)
shape = tuple((ashp - wshp) // stp + 1) + tuple(wshp)
strides = tuple(astr * stp) + tuple(astr)
as_strided = np.lib.stride_tricks.as_strided
a_view = np.squeeze(as_strided(a,
shape = shape,
strides = strides))
if gen_data :
return a_view, shape[:-wshp.size]
else:
return a_view
def window_gen(a, window, **kwargs):
#Same docstring as above, returns a generator
_ = kwargs.pop(gen_data, False)
a_view, shp = window_nd(a, window, gen_data = True, **kwargs)
for idx in np.ndindex(shp):
yield a_view[idx]
Some test cases:
a = np.arange(1000).reshape(10,10,10)
window_nd(a, 4).shape # sliding (4x4x4) window
Out: (7, 7, 7, 4, 4, 4)
window_nd(a, 2, 2).shape # (2x2x2) blocks
Out: (5, 5, 5, 2, 2, 2)
window_nd(a, 2, 1, 0).shape # sliding window of width 2 over axis 0
Out: (9, 2, 10, 10)
window_nd(a, 2, 2, (0,1)).shape # tiled (2x2) windows over first and second axes
Out: (5, 5, 2, 2, 10)
window_nd(a,(4,3,2)).shape # arbitrary sliding window
Out: (7, 8, 9, 4, 3, 2)
window_nd(a,(4,3,2),(1,5,2),(0,2,1)).shape #arbitrary windows, steps and axis
Out: (7, 5, 2, 4, 2, 3) # note shape[-3:] != window as axes are out of order
Related
I'm currently writing a tensorflow program that requires multiplying a batch of 2-D tensors (a 3-D tensor of shape [None,...]) with a 2-D matrix W. This requires turning W into a 3-D matrix, which requires knowing the batch size.
I have not been able to do this; tf.batch_matmul is no longer usable, x.get_shape().as_list()[0] returns None, which is invalid for a reshaping/tiling operation. Any suggestions? I've seen some people use config.cfg.batch_size, but I don't know what that is.
Solution is to use a combination of tf.shape (which returns the shape at runtime) and tf.tile (which accepts the dynamic shape).
x = tf.placeholder(shape=[None, 2, 3], dtype=tf.float32)
W = tf.Variable(initial_value=np.ones([3, 4]), dtype=tf.float32)
print(x.shape) # Dynamic shape: (?, 2, 3)
batch_size = tf.shape(x)[0] # A tensor that gets the batch size at runtime
W_expand = tf.expand_dims(W, axis=0)
W_tile = tf.tile(W_expand, multiples=[batch_size, 1, 1])
result = tf.matmul(x, W_tile) # Can multiply now!
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
feed_dict = {x: np.ones([10, 2, 3])}
print(sess.run(batch_size, feed_dict=feed_dict)) # 10
print(sess.run(result, feed_dict=feed_dict).shape) # (10, 2, 4)
As a simplified example, suppose I have a dataset composed of 40 sorted values. The values of this example are all integers, though this is not necessarily the case for the actual dataset.
import numpy as np
data = np.linspace(1,40,40)
I am trying to find the maximum value inside the dataset for certain window sizes. The formula to compute the window sizes yields a pattern that is best executed with arrays (in my opinion). For simplicity sake, let's say the indices denoting the window sizes are a list [1,2,3,4,5]; this corresponds to window sizes of [2,4,8,16,32] (the pattern is 2**index).
## this code looks long because I've provided docstrings
## just in case the explanation was unclear
def shapeshifter(num_col, my_array=data):
"""
This function reshapes an array to have 'num_col' columns, where
'num_col' corresponds to index.
"""
return my_array.reshape(-1, num_col)
def looper(num_col, my_array=data):
"""
This function calls 'shapeshifter' and returns a list of the
MAXimum values of each row in 'my_array' for 'num_col' columns.
The length of each row (or the number of columns per row if you
prefer) denotes the size of each window.
EX:
num_col = 2
==> window_size = 2
==> check max( data[1], data[2] ),
max( data[3], data[4] ),
max( data[5], data[6] ),
.
.
.
max( data[39], data[40] )
for k rows, where k = len(my_array)//num_col
"""
my_array = shapeshifter(num_col=num_col, my_array=data)
rows = [my_array[index] for index in range(len(my_array))]
res = []
for index in range(len(rows)):
res.append( max(rows[index]) )
return res
So far, the code is fine. I checked it with the following:
check1 = looper(2)
check2 = looper(4)
print(check1)
>> [2.0, 4.0, ..., 38.0, 40.0]
print(len(check1))
>> 20
print(check2)
>> [4.0, 8.0, ..., 36.0, 40.0]
print(len(check2))
>> 10
So far so good. Now here is my problem.
def metalooper(col_ls, my_array=data):
"""
This function calls 'looper' - which calls
'shapeshifter' - for every 'col' in 'col_ls'.
EX:
j_list = [1,2,3,4,5]
==> col_ls = [2,4,8,16,32]
==> looper(2), looper(4),
looper(8), ..., looper(32)
==> shapeshifter(2), shapeshifter(4),
shapeshifter(8), ..., shapeshifter(32)
such that looper(2^j) ==> shapeshifter(2^j)
for j in j_list
"""
res = []
for col in col_ls:
res.append(looper(num_col=col))
return res
j_list = [2,4,8,16,32]
check3 = metalooper(j_list)
Running the code above provides this error:
ValueError: total size of new array must be unchanged
With 40 data points, the array can be reshaped into 2 columns of 20 rows, or 4 columns of 10 rows, or 8 columns of 5 rows, BUT at 16 columns, the array cannot be reshaped without clipping data since 40/16 ≠ integer. I believe this is the problem with my code, but I do not know how to fix it.
I am hoping there is a way to cutoff the last values in each row that do not fit in each window. If this is not possible, I am hoping I can append zeroes to fill the entries that maintain the size of the original array, so that I can remove the zeroes after. Or maybe even some complicated if - try - break block. What are some ways around this problem?
I think this will give you what you want in one step:
def windowFunc(a, window, f = np.max):
return np.array([f(i) for i in np.split(a, range(window, a.size, window))])
with default f, that will give you a array of maximums for your windows.
Generally, using np.split and range, this will let you split into a (possibly ragged) list of arrays:
def shapeshifter(num_col, my_array=data):
return np.split(my_array, range(num_col, my_array.size, num_col))
You need a list of arrays because a 2D array can't be ragged (every row needs the same number of columns)
If you really want to pad with zeros, you can use np.lib.pad:
def shapeshifter(num_col, my_array=data):
return np.lib.pad(my_array, (0, num_col - my.array.size % num_col), 'constant', constant_values = 0).reshape(-1, num_col)
Warning:
It is also technically possible to use, for example, a.resize(32,2) which will create an ndArray padded with zeros (as you requested). But there are some big caveats:
You would need to calculate the second axis because -1 tricks don't work with resize.
If the original array a is referenced by anything else, a.resize will fail with the following error:
ValueError: cannot resize an array that references or is referenced
by another array in this way. Use the resize function
The resize function (i.e. np.resize(a)) is not equivalent to a.resize, as instead of padding with zeros it will loop back to the beginning.
Since you seem to want to reference a by a number of windows, a.resize isn't very useful. But it's a rabbit hole that's easy to fall into.
EDIT:
Looping through a list is slow. If your input is long and windows are small, the windowFunc above will bog down in the for loops. This should be more efficient:
def windowFunc2(a, window, f = np.max):
tail = - (a.size % window)
if tail == 0:
return f(a.reshape(-1, window), axis = -1)
else:
body = a[:tail].reshape(-1, window)
return np.r_[f(body, axis = -1), f(a[tail:])]
Here's a generalized way to reshape with truncation:
def reshape_and_truncate(arr, shape):
desired_size_factor = np.prod([n for n in shape if n != -1])
if -1 in shape: # implicit array size
desired_size = arr.size // desired_size_factor * desired_size_factor
else:
desired_size = desired_size_factor
return arr.flat[:desired_size].reshape(shape)
Which your shapeshifter could use in place of reshape
I want to draw ROC curves with pRoC.
However for some reason there is extra empty space on either side of the x-axis and I cannot remove it with xlim. Some example code:
library(pROC)
n = c(4, 3, 5)
b = c(TRUE, FALSE, TRUE)
df = data.frame(n, b)
rocobj <- plot.roc(df$b, df$n, percent = TRUE, main="ROC", col="#1c61b6", add=FALSE)
I tried the pROC help file, but that doesn't really help me. Even more puzzling is to me that the Y-axis is OK looking...
I really appreciate your help!
Make sure the plotting device is square and adjust the margins so that top + bottom == left + right:
library(pROC)
png("test.png", width = 480, height = 480)
par(mar = c(4, 4, 4, 4)+.1)
n = c(4, 3, 5)
b = c(TRUE, FALSE, TRUE)
rocobj <- plot.roc(b, n, percent = TRUE, main="ROC", col="#1c61b6", add=FALSE)
dev.off()
An other answer, if you don't mind to have distorted axis, is to use the asp parameter. By default it is set to 1, ensuring both axis have the same scale and the ROC curve is squared*, but you can turn it off with asp = NA:
library(pROC)
par(mar = c(4, 4, 4, 4)+.1)
n = c(4, 3, 5)
b = c(TRUE, FALSE, TRUE)
rocobj <- plot.roc(b, n, percent = TRUE, main="ROC", col="#1c61b6", add=FALSE, asp = NA)
* Having a squared ROC curve is important if you want to interpret it visually. For instance, you may want to compare several local maximas by their distance to the diagonal: you can only do that if the two axis have the same scale. So if you want to do that make sure to follow my other answer.
There is yet a third answer, which takes the margins out of the plotting region, so it will automatically look squared, even when the device isnt. This is done by setting the graphical parameter pty to "s":
library(pROC)
par(pty = "s")
n = c(4, 3, 5)
b = c(TRUE, FALSE, TRUE)
rocobj <- plot.roc(b, n, percent = TRUE, main="ROC", col="#1c61b6", add=FALSE, asp = NA)
(I added a black frame to visualize what's going on)
I have two 2D Theano tensors, call them x_1 and x_2, and suppose for the sake of example, both x_1 and x_2 have shape (1, 50). Now, to compute their mean squared error, I simply run:
T.sqr(x_1 - x_2).mean(axis = -1).
However, what I wanted to do was construct a new tensor that consists of their mean squared error in chunks of 10. In other words, since I'm more familiar with NumPy, what I had in mind was to create the following tensor M in Theano:
M = [theano.tensor.sqr(x_1[:, i:i+10] - x_2[:, i:i+10]).mean(axis = -1) for i in xrange(0, 50, 10)]
Now, since Theano doesn't have for loops, but instead uses scan (which map is a special case of), I thought I would try the following:
sequence = T.arange(0, 50, 10)
M = theano.map(lambda i: theano.tensor.sqr(x_1[:, i:i+10] - x_2[:, i:i+10]).mean(axis = -1), sequence)
However, this does not seem to work, as I get the error:
only integers, slices (:), ellipsis (...), numpy.newaxis (None) and integer or boolean arrays are valid indices
Is there a way to loop through the slices using theano.scan (or map)? Thanks in advance, as I'm new to Theano!
Similar to what can be done in numpy, a solution would be to reshape your (1, 50) tensor to a (1, 10, 5) tensor (or even a (10, 5) tensor), and then to compute the mean along the second axis.
To illustrate this with numpy, suppose I want to compute means by slices of 2
x = np.array([0, 2, 0, 4, 0, 6])
x = x.reshape([3, 2])
np.mean(x, axis=1)
outputs
array([ 1., 2., 3.])
function prealloc()
situation=zeros(Int64,3^5,5);
i=1;
for north=0:2, south=0:2, east=0:2, west=0:2, current=0:2
situation[i,:]=[north, south, east, west, current]
if situation[i,:]=[2, 2, 2, 2, 2]
elseif situation[i,:]=[2, 2, 2, 2, 1]
elseif situation[i,:]=[2, 2, 2, 2, 0]`enter code here`
end
i+=1
end
situation
end
How can I eliminate the row which equal those if conditions from the array which called situation
First things first: the code in your question doesn't run (for several reasons). When posting code in questions, it is good form to put it in a "working example" form, where users can copy and paste it into their editor of choice and it will work without the user having to make educated guesses as to what you are actually trying to do. This is probably one reason the question has received down-votes.
With that out of the way, there are two approaches to accomplish what you are trying to do:
1) Construct your matrix without the indicated rows in the first step. Then you don't need to worry about "deleting the rows" later on. For situations as simple as the one in the question, you could just do something like this:
function prealloc()
x = zeros(Int, 3^5 - 3, 5)
i = 1
for n=0:2, s=0:2, ea=0:2, w=0:2, cur=0:2
if !([n, s, ea, w, cur] == [2, 2, 2, 2, 2] || [n, s, ea, w, cur] == [2, 2, 2, 2, 1] || [n, s, ea, w, cur] == [2, 2, 2, 2, 0])
x[i, :] = [n, s, ea, w, cur]
i += 1
end
end
return(x)
end
Notice I'm using Int, not Int64. This will not affect performance, and it means your code will run on both 32-bit and 64-bit architectures.
Another style tip. Don't use semi-colons to end lines. This is a Matlab quirk, and it is not needed in Julia.
2) As other users have suggested, you could construct the entire matrix (including the undesirable rows), and then remove them at a later point. Of course, this necessitates re-allocating the entire matrix, and so is somewhat inefficient (note, you can remove elements of vectors in place, i.e. without re-allocation, but not any arrays of dimension 2 or greater). In this case, to encourage code re-use, it makes sense to break the routine down into three separate functions. First, we allocate the entire matrix:
function prealloc1()
x = zeros(Int64,3^5,5)
i = 1
for north=0:2, south=0:2, east=0:2, west=0:2, current=0:2
x[i,:]=[north, south, east, west, current]
i += 1
end
return(x)
end
Next, we obtain a vector of indices that we wish to remove. We do this as its own step because we only want to re-allocate the matrix once, rather than re-allocating every time we find a new row we want to delete. For your situation, you could use a function like this:
function findCondition(x::Matrix{Int})
inds = Array(Int, 0)
for i = 1:size(x, 1)
if x[i, :] == [2 2 2 2 2]
push!(inds, i)
elseif x[i, :] == [2 2 2 2 1]
push!(inds, i)
elseif x[i, :] == [2 2 2 2 0]
push!(inds, i)
end
end
return(inds)
end
Notice that in my comparison statements in this function I use [2 2 2 2 2] instead of [2, 2, 2, 2, 2]. This is because the first construct is a 2-dimensional array (type Matrix) while the second is 1-dimensional (type Vector). Since x[i, :] is of type Matrix, the difference is important.
Finally, we need to re-allocate the matrix without the offending rows. As user #Matt B. suggests, this can be done with the following one-liner function:
removeIndices(x::Matrix{Int}, inds::Vector{Int}) = x[setdiff(IntSet(1:size(x, 1)), IntSet(inds)), :]
Note, applying setdiff to IntSet here is fast because by construction inds will already be sorted in ascending order.
You cannot just delete a row in Julia, the only way to do it, is to create a copy of the array without the row you want to delete. And I think that's not internally implemented and it's intentional.
So you will have to do to it manually, something like this will create a copy of situation without the row i (which is not the same as saying that it will delete row i).
situation = vcat(situation[1:i-1,:],situation[i+1:end,:])
also, this will actually change the dimensions of situation in each iteration, so be careful with that...
Also2, your loop will finish in a bounds error since eventually it will be off limits of your array, maybe you could write something like this to end your loop.
if i = length(situation)
break
else
i += 1
end
Ultimately, you can make a function delrow and call it from within your loop:
function delrow(array,row)
return vcat(array[1:row-1,:],array[row+1:end,:])
end
then call situation = delrow(situation,i)