In python 3 (3.6.5), I have data in a (much longer) numpy array looking like this:
data = np.array([[16347, 0, 60],[16353, 0, 92],[16382, 0, 1],[17867, 0, 2],[20188, 0, 3],[21459, 0, 512],[21873, 0, 71],[22031, 0, 4],[23072, 0, 61],[25378, 0, 60],[25385, 0, 82],[25410, 0, 1],[26895, 0, 2],[29233, 0, 3],[31695, 0, 71],[31845, 0, 4],[32886, 0, 61],[35069, 0, 60],[35075, 0, 90],[35104, 0, 1]])
The first two columns can be ignored for the point of this question. In the third, I would like to replace all 2 entries with a value in the same column, 2 rows before. For instance, in the example data there is a 2 on the 4th row, and it should be replaced by the number 92in row 2. Similarly, the 2 on row 13 needs to be replaced by 82 on line 11, and so on.
In short, I need to search for all 2 entries in a column within a numpy array, and replace them for whatever value was on the same column 2 rows before.
I'd appreciate any tips or ideas.
Thanks!
You can use the np.where() and np.roll() to do this:
data[:,-1]=where(data[:,-1]==2,np.roll(data[:,-1],2),data[:,-1])
Here
data[:,-1]
Isolates the 3rd column you are interested in, and where() returns an array filled with values that depend on the condition. If the condition that the value is equal to 2 is True, it returns the corresponding value from
np.roll(data[:,-1],2)
which is the original column shifted forward by 2, with the last two values no becoming the first two values.
There result for the input array:
[[16347 0 60]
[16353 0 92]
[16382 0 1]
[17867 0 2]
[20188 0 3]
[21459 0 512]
[21873 0 71]
[22031 0 4]
[23072 0 61]
[25378 0 60]
[25385 0 82]
[25410 0 1]
[26895 0 2]
[29233 0 3]
[31695 0 71]
[31845 0 4]
[32886 0 61]
[35069 0 60]
[35075 0 90]
[35104 0 1]]
is the desired:
[[16347 0 60]
[16353 0 92]
[16382 0 1]
[17867 0 92]
[20188 0 3]
[21459 0 512]
[21873 0 71]
[22031 0 4]
[23072 0 61]
[25378 0 60]
[25385 0 82]
[25410 0 1]
[26895 0 82]
[29233 0 3]
[31695 0 71]
[31845 0 4]
[32886 0 61]
[35069 0 60]
[35075 0 90]
[35104 0 1]]
Related
I have an array made of 0 and 1. I want to calculate a cumulative sum of all consecutive 1 with a reset each time a 0 is met, using numpy as I have thousands of arrays of thousands of lines and columns.
I can do it with loops but I suspect it will not be efficient.
Would you have a smarter and quick way to run it on the array.
Here is short example of the input and the expected output:
import numpy as np
arr_in = np.array([[1,1,1,1,1,1], [0,0,0,0,0,0], [1,0,1,0,1,1], [0,1,1,1,0,0]])
print(arr_in)
print("expected result:")
arr_out = np.array([[1,2,3,4,5,6], [0,0,0,0,0,0], [1,0,1,0,1,2], [0,1,2,3,0,0]])
print(arr_out)
When you run it:
[[1 1 1 1 1 1]
[0 0 0 0 0 0]
[1 0 1 0 1 1]
[0 1 1 1 0 0]]
expected result:
[[1 2 3 4 5 6]
[0 0 0 0 0 0]
[1 0 1 0 1 2]
[0 1 2 3 0 0]]
With numba.vectorize you can define a custom numpy ufunc to use for accumulation.
import numba as nb # v0.56.4, no support for numpy >= 1.22.0
import numpy as np # v1.21.6
#nb.vectorize([nb.int64(nb.int64, nb.int64)])
def reset_cumsum(x, y):
return x + y if y else 0
arr_in = np.array([[1,1,1,1,1,1],
[0,0,0,0,0,0],
[1,0,1,0,1,1],
[0,1,1,1,0,0]])
reset_cumsum.accumulate(arr_in, axis=1)
Output
array([[1, 2, 3, 4, 5, 6],
[0, 0, 0, 0, 0, 0],
[1, 0, 1, 0, 1, 2],
[0, 1, 2, 3, 0, 0]])
You can compute the cumsum for the 1s, then identify the 0s and forward-fill the cumulated sum to subtract it:
# identify 0s
mask = arr_in==0
# get classical cumsum
cs = arr_in.cumsum(axis=1)
# ffill the cumsum value on 1s
# subtract from cumsum
out = cs-np.maximum.accumulate(np.where(mask, cs, 0), axis=1)
Output:
[[1 2 3 4 5 6]
[0 0 0 0 0 0]
[1 0 1 0 1 2]
[0 1 2 3 0 0]]
Output on second example:
[[1 2 3 4 5 6 0 1]
[0 1 2 0 0 0 1 0]]
I have the following array:
[[1 2 1 0 2 0]
[1 2 1 0 2 0]
[1 2 1 0 2 0]
[1 2 1 0 2 0]
[0 1 2 1 0 0]
[0 1 2 1 0 0]
[0 0 1 0 1 0]
[0 0 0 1 1 0]
[0 0 0 0 1 0]
[0 0 0 0 0 1]]
I need to add a column to this array that adds a number whenever the values in the rows change starting with number 3. So the result would look like this:
[[1 2 1 0 2 0 3]
[1 2 1 0 2 0 3]
[1 2 1 0 2 0 3]
[1 2 1 0 2 0 3]
[0 1 2 1 0 0 4]
[0 1 2 1 0 0 4]
[0 0 1 0 1 0 5]
[0 0 0 1 1 0 6]
[0 0 0 0 1 0 7]
[0 0 0 0 0 1 8]]
Thank you
If a is your array as:
a = np.array([[1, 2, 1, 0, 2, 0], [1, 2, 1, 0, 2, 0], [1, 2, 1, 0, 2, 0], [1, 2, 1, 0, 2, 0],
[0, 1, 2, 1, 0, 0], [0, 1, 2, 1, 0, 0], [0, 0, 1, 0, 1, 0], [0, 0, 0, 1, 1, 0],
[0, 0, 0, 0, 1, 0], [0, 0, 0, 0, 0, 1]])
using the following code will get you the results:
n = 3
a = a.tolist()
for i, j in enumerate(a):
if i == 0:
j.append(n)
elif i > 0 and j == a[i-1][:-1]:
j.append(n)
else:
n += 1
j.append(n)
# a = np.array(a)
which will give:
[[1 2 1 0 2 0 3]
[1 2 1 0 2 0 3]
[1 2 1 0 2 0 3]
[1 2 1 0 2 0 3]
[0 1 2 1 0 0 4]
[0 1 2 1 0 0 4]
[0 0 1 0 1 0 5]
[0 0 0 1 1 0 6]
[0 0 0 0 1 0 7]
[0 0 0 0 0 1 8]]
Consider a (m x n) matrix of only 0s and 1s, with m potentially large.
julia> rand([0, 1], 5, 3)
5×3 Array{Int64,2}:
0 1 1
0 0 0
0 1 1
1 0 0
1 0 1
Is there an efficient way to count the number of occurrences and track the indices for each unique row?
For example, the first row above occurs twice, at indices 1 and 3. I am trying to build a sort of contingency table.
Thanks
This is one of the approaches that is based only on functionalities provided in Julia Base:
julia> x = rand([0, 1], 20, 3)
20×3 Matrix{Int64}:
1 0 1
1 1 1
0 0 0
1 0 0
0 0 1
1 0 1
0 0 0
1 0 1
0 0 0
0 0 1
1 1 0
0 1 1
0 1 1
1 0 0
0 0 0
0 0 0
0 0 1
0 1 0
1 1 0
1 0 0
julia> d = Dict()
Dict{Any, Any}()
julia> for (i, r) in enumerate(eachrow(x))
push!(get!(d, r, Int[]), i)
end
julia> d
Dict{Any, Any} with 8 entries:
[1, 1, 1] => [2]
[0, 0, 0] => [3, 7, 9, 15, 16]
[0, 0, 1] => [5, 10, 17]
[1, 1, 0] => [11, 19]
[1, 0, 0] => [4, 14, 20]
[0, 1, 1] => [12, 13]
[1, 0, 1] => [1, 6, 8]
[0, 1, 0] => [18]
and now using the SplitApplyCombine.jl package:
julia> using SplitApplyCombine
julia> group(i -> view(x, i, :), axes(x, 1))
8-element Dictionaries.Dictionary{Any, Vector{Int64}}
[1, 0, 1] │ [1, 6, 8]
[1, 1, 1] │ [2]
[0, 0, 0] │ [3, 7, 9, 15, 16]
[1, 0, 0] │ [4, 14, 20]
[0, 0, 1] │ [5, 10, 17]
[1, 1, 0] │ [11, 19]
[0, 1, 1] │ [12, 13]
[0, 1, 0] │ [18]
How can I reset all values in a column from a negative number to the top to zero in an array?
data = np.array([[1, 1, 1, 2], [0, 1, 0, -1], [-1, 0, 1, 0], [1, 1, 1, 1]])
resetneg_data = np.where(data<0, 0, data)
print(resetnet_data)
This gives me:
[[1 1 1 2]
[0 1 0 0]
[0 0 1 0]
[1 1 1 1]]
But what I want is:
[[0 1 1 0]
[0 1 0 0]
[0 0 1 0]
[1 1 1 1]]
That is, zero where negative, and zero everywhere above the negative. But not zero above other zeros. So that if a column drops below zero in a row, all the rows above it reset to zero.
Can I mask the values somehow by finding the specific ranges:
mask_end = np.where(data < 0)
print(mask_end)
gives:
(array([1, 2]), array([3, 0]))
maybe... use those values to replace to that row in a column with zeros?
# find values that are smaller than 0 from bottom up along with values above negatives
mask = np.minimum.accumulate(data[::-1])[::-1] < 0
# set value at mask positions as 0
data[mask] = 0
data
#[[0 1 1 0]
# [0 1 0 0]
# [0 0 1 0]
# [1 1 1 1]]
Given an 10 x N matrix of 1's and 0s, such as:
1 1 1 1 1 1 1 1
1 1 0 1 1 0 0 1
0 0 0 1 1 0 0 1
0 0 0 1 1 0 0 0
1 0 0 0 0 1 0 0
1 0 1 1 1 1 1 1
1 0 1 1 1 1 1 1
1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1
notes:
the zeroes in a column are always between two runs of consecutive 1s. for example, a column such as 1 1 1 0 1 0 1 1 1 1 is not permitted
there must be at least a gap of one zero in each column, ie a column such as: 1 1 1 1 1 1 1 1 1 1 is not allowed
I want to find the longest consecutive streak of zeroes from left to right. In this case, that would be 4, which corresponds to the path starting in the second column of the 5th row from the top,
The second longest is 3 and there are 3 examples of that.
I'm a bit stumped on this, especially for very large N (~10M). I am looking for suggestions for the right approach/data structure to use or a similar problem and the algorithm used there. Another potential way to model the problem is to represent the problem using two lists:
L1 = [2, 2, 1, 4, 4, 1, 1, 3]
L2 = [6, 3, 5, 5, 5, 5, 5, 5]
but still not quite sure how to come up with an efficient solution
The solution using itertools.groupby(), sum() and max() functions:
import itertools
m = [
[1, 1, 1, 1, 1, 1, 1, 1],
[1, 1, 0, 1, 1, 0, 0, 1],
[0, 0, 0, 1, 1, 0, 0, 1],
[0, 0, 0, 1, 1, 0, 0, 0],
[1, 0, 0, 0, 0, 1, 0, 0],
[1, 0, 1, 1, 1, 1, 1, 1],
[1, 0, 1, 1, 1, 1, 1, 1],
[1, 1, 1, 1, 1, 1, 1, 1],
[1, 1, 1, 1, 1, 1, 1, 1],
[1, 1, 1, 1, 1, 1, 1, 1]
]
max_zero_len = max(sum(1 for z in g if z == 0)
for l in m for k,g in itertools.groupby(l))
print(max_zero_len)
The output:
4
for l in m for k,g in itertools.groupby(l) - will generate a separate group for each consecutive sequences of 1 and 0 values for each nested list. (like [1,1], [0], [1,1], [0,0] ...)
sum(1 for z in g if z == 0) - considers only 0's sequences and counts its length using sum function
max(...) - gets the maximum length among zero(0) sequences