I am having a headache with a numba loop and a 1d numpy array and I cannot seem to find an explanation.
Basically, my goal is to pass to modify a numpy array in parallel with numba loop using a function both passed as arguments. It works well with a 2d numpy array but for some reasons, it does not with a simple 1d numpy array. This is the code to reproduce the issue:
import numpy as np
import numba as nb
size = 10
# Define a 1d numpy array
vec_test_1 = np.zeros(size)
# Fill the 1d array with values
for i in range(size):
vec_test_1[i] = float(i)
# Function that modifies and element
#nb.jit(nopython = True)
def func1(xyz):
xyz = xyz + 2.47
# Loop with numba to modify all elements of the array
#nb.jit(nopython = True, parallel = True)
def loop_numba(vec, func):
for i in nb.prange(len(vec)):
func(vec[i])
loop_numba(vec_test_1, func1)
The vec_test_1 is unchanged after this loop:
array([0., 1., 2., 3., 4., 5., 6., 7., 8., 9.])
when it should be:
array([ 2.47, 3.47, 4.47, 5.47, 6.47, 7.47, 8.47, 9.47, 10.47,
11.47])
What surprises me is that it works well when the array that is passed as argument is a 2d array. I am able to modify all its element with the numba loop.
Could anyone help me to understand this issue?
You have to define a return value since you make a copy of the individual elements when passing to the function.
Explanation: Found here.
Basically: you pass a single, immutable element to the function, so it is passed by copy (a copy is created, which is changed in the function). If you do it with a 2D array, to python this is a mutable object, so it is passed by reference. If you operate on it now, the underlying reference is changed, and this is visible in the result outside of the function.
import numpy as np
import numba as nb
size = 10
# Define a 1d numpy array
vec_test_1 = np.arange(size, dtype=np.float32)
# Function that modifies and element
#nb.jit(nopython = True)
def func1(xyz):
xyz = xyz + 2.47
return xyz
# Loop with numba to modify all elements of the array
#nb.jit(nopython = True, parallel = True)
def loop_numba(vec, func):
for i in nb.prange(len(vec)):
vec[i] = func(vec[i])
loop_numba(vec_test_1, func1)
In [2]: vec_test_1
Out[2]:
array([ 2.47, 3.47, 4.47, 5.47, 6.47, 7.47, 8.47, 9.47, 10.47,
11.47], dtype=float32)
Also: I changed your vector initialization to np.arange(size, dtype=float) to make it easier to understand.
Related
enter image description here
I am getting odd behavior with Jupyter/Numpy/Tranpose()/1D Arrays.
I found another post where transpose() will not transpose a 1D array, but in previous Jupyter notebooks, it does.
I have an example where it is inconsistent, and I do not understand:
Please see the picture attached of my jupyter notebook if 2 more or less identical arrays with 2 different outputs.
It seems it IS and IS NOT transposing the 1D array. Inconsistency is bad
outputs is (1000,) and (1,1000), why does this occur?
# GENERATE WAVEORM:
#---------------------------------------------------------------------------------------------------
N = 1000
fxc = []
fxn = []
for t in range(0,N):
fxc.append(A1*m.sin(2.0*pi*50.0*dt*t) + A2*m.sin(2.0*pi*120.0*dt*t))
fxn.append(A1*m.sin(2.0*pi*50.0*dt*t) + A2*m.sin(2.0*pi*120.0*dt*t) + 5*np.random.normal(u,std,size=1))
#---------------------------------------------------------------------------------------------------
# TAKE TRANSPOSE:
#---------------------------------
fc = np.transpose(np.array(fxc))
fn = np.transpose(np.array(fxn))
#---------------------------------
# PRINT DIMENSION:
#---------------------------------
print(fc.shape)
print(fn.shape)
#---------------------------------
Remove size=1 from your call to numpy.random.normal. Then it will return a scalar instead of a 1-d array of length 1.
For example,
In [2]: np.random.normal(0, 3, size=1)
Out[2]: array([0.47058288])
In [3]: np.random.normal(0, 3)
Out[3]: 4.350733438283539
Using size=1 in your code is a problem, because it results in fxn being a list of 1-d arrays (e.g. something like [[0.123], [-.4123], [0.9455], ...]. When NumPy converts that to an array, it has shape (N, 1). Transposing such an array results in the shape (1, N).
fxc, on the other hand, is a list of scalars (e.g. something like [0.123, 0.456, ...]). When converted to a NumPy array, it will be a 1-d array with shape (N,). NumPy's transpose operation swaps dimensions, but it does not create new dimensions, so transposing a 1-d array does nothing.
I am trying to print two different lists with numpy and pandas respectively.
The strange thing is that I can only print one list at a time by commenting the other one with all its accosiated code. Do mumpy and pandas have any dependcies?
import numpy as np
import pandas as pd
np.array = []
for i in range(7):
np.array.append([])
np.array[i] = i
values = np.array
print(np.power(np.array,3))
df = pd.DataFrame({'X':[78,85,96,80,86], 'Y':[84,94,89,83,86],'Z':[86,97,96,72,83]})
print(df)
I'm not sure what you mean by "I can only print one list at a time by commenting the other one with all its accosiated code", but any strange behavior you're seeing probably comes from you assigning to np.array. You should name your variable something different, e. g. array. Perhaps you were trying to do this:
arr = []
for i in range(7):
arr.append([])
arr[i] = i
values = np.array(arr)
I stuck with a simple question in NumPy. I have an array of zero values. Once I generate a new value I would like to add it one by one.
arr=array([0,0,0])
# something like this
l=[1,5,10]
for x in l:
arr.append(x) # from python logic
so I would like to add one by one x into array, so I would get: 1st iteration arr=([1,0,0]); 2d iteration arr=([1,5,0]); 3rd arr=([1,5,10]);
Basically I need to substitute zeros with new values one by one in NumPy (I am learning NumPy!!!!!!).
I checked many of NumPy options like np.append (it adds to existing values new values), but can't find the right.
thank you
There are a few things to pick up with numpy:
you can generate the array full of zeros with
>>> np.zeros(3)
array([ 0., 0., 0.])
You can get/set array elements with indexing as with lists etc:
arr[2] = 7
for i, val in enumerate([1, 5, 10]):
arr[i] = val
Or, if you want to fill with array with something like a list, you can directly use:
>>> np.array([1, 5, 10])
array([ 1, 5, 10])
Also, numpy's signature for appending stuff to an array is a bit different:
arr = np.append(arr, 7)
Having said that, you should just consider diving into Numpy's own userguide.
for the following code:
from array import *
x=[]
x.append(0.232)
print (x)
for i in range(25):
x[i+1]=(1/(i+1))-5*x[i]
I have this error:
x[i+1]=(1/(i+1))-5*x[i]
IndexError: list assignment index out of range
This may be happening because I have defined x to be an empty array. But how do I define the array and perform the same operation otherwise?
list is not designed for efficient mathematical operations and therefore its better to use numpy arrays for doing mathematical operations. However, if you want to use list, you may define a list initialized with n zero's using
x=[0]*n
x[0] = 0.232
x[1] = ....
....
Remember, that a multidimensional list created using above approach will refer to same element in the array! For example:
l = [0,0,0]*5
will be creating five same list's inside another list not separate list's. So its a bad idea to create multidimensional array like this!
A better way would be to create arrays using numpy using following code:
from numpy import empty, zeros
x = empty(n) # or # x = zeros(n)
x[0] = 0.232
x[1] = ....
....
and
l = empty((3,5)) # or # l = zeros((3,5))
for a array with 3 rows and 5 columns.
Is it possible to convert a one dimensional array
a = np.array([1,2,3])
to a two dimensional array that is equivalent to
b = np.array([[1],[2],[3]])
without creating a copy of the array and also
with b being contiguous?
You can do this using np.newaxis and the T transpose method.
import numpy as np
a = np.array([1,2,3])
a = a[np.newaxis].T
print(a)
# [[1]
# [2]
# [3]]
Reshaping the array does not copy data (where possible*) and retains C contiguity (usually):
>>> a = np.array([1,2,3])
>>> b = a.reshape(3, 1)
>>> b
array([[1],
[2],
[3]])
>>> b.flags
C_CONTIGUOUS : True
F_CONTIGUOUS : False
OWNDATA : False
WRITEABLE : True
ALIGNED : True
UPDATEIFCOPY : False
*Edit: Reshaping often creates a view of an array, but this is not always possible (docs). There are ways to check that the data has not been copied and that a and b share the same underlying data. For example see here and here.
In the above case, trying a couple of tests showed that reshape created a view of a (no data copied):
>>> a.data == b.data
True
>>> np.may_share_memory(a, b) # function from linked answer above
True