Write into re-opend NetCDF4 file - arrays

I'm trying to write numbers into an array with an unlimited dimensions. The file I've created is structured like this :
import netCDF4 as nc4
rootgrp = nc4.Dataset("test.nc",'a',format="NETCDF4")
mgrp= rootgrp.createGroup('Flex')
mgrp.createDimension('pv',None)
mgrp.createDimension('s',4)
a = mgrp.createVariable('fill',"f8",('pv','s'))
rootgrp.close()
Now I'm trying to fill this array like this :
while i<10:
f = nc4.Dataset("test.nc",'r+',format="NETCDF4")
fgrp= f.groups['Flex']
fgrp['fill'][i][0] = i
print(fgrp['fill'][i][:])
f.groups['Flex'].variables['fill'][i][3] = i
f.close()
i=i+1
But I'm always getting a 'dimension out of bounds' error even though it's telling me that I've no dimension limit. Even if I use an array with fixed 100x4 dimension i still get the same error.
Would appreciate any kind of help.

This line is the problem:
fgrp['fill'][i][0] = i
fgrp['fill'][i] gets the ith row from the 'fill' variable. It then immediately tries to index into that row with [0], which errors out because there's nothing in that row. To solve this problem, do the indexing in one step instead:
fgrp['fill'][i, 0] = i

Related

Matlab dot index error in array filled with objects from a class

Hi everyone i'm experiencing some trouble with my code. The goal is to add multiple objects to a class, i've created the following code for that:
array = cell(1,10);
A = matrix %specified somewhere else
for ind = 1:10
[r c] = find(A<3);
random = randi([1, size(r,1)]);
array{ind} = classname(1, r(random), c(random));
end
This block of code does everything I want it to, however, later on I add this array as property to another class and have to work with that specific array to make changes to it. And that is where is goes wrong.
function growing(thisElement)
for i = 1:length(otherClass.array)
range = range((otherClass.array(i).locationX - 2), (otherClass.array(i).locationX +2));
if otherClass.array(i).pos<20
......
end
end
end
Whereby both locationX and pos are properties of classname. I get the error Dot indexing is not supported for variables of this type. For both the range as the last if part. The error is apparently very commen error, however, I don't find any answers relating my case. Has anyone an idea to bypass the error?

How to create an array of shapes in VPython?

I'm trying to create an array of spheres in VPython, each with a manually entered position. Something like:
ball[0] = sphere(pos=vector(-1,4,9))
ball[1] = sphere(pos=vector(-2,6,6))
ball[2] = sphere(pos=vector(0,6,1))
etc. The problem is that I keep getting an error reading "IndexError: list assignment index out of range". How can I resolve this issue?
see How to declare an array in Python?.
You also might want to consider using a compound
Modified code:
# https://stackoverflow.com/questions/56461496/how-to-create-an-array-of-shapes-in-vpython
from vpython import *
# https://stackoverflow.com/questions/1514553/how-to-declare-an-array-in-python
ball=[]
ball.append(sphere(pos=vector(-1,4,9)))
ball.append(sphere(pos=vector(-2,6,6)))
ball.append(sphere(pos=vector(0,6,1)))
Result:

Pytables Value Error (rank of the appended object and "..."EArray differ)

I am trying to use pytables to store my images dataset. I am using Earray to append each image as it is read. The dimensions of my Earray and image are same(except for the first, along which appending is done). I am using the following code:
atom = Atom.from_dtype(np.dtype(np.uint32,(278,278,1)))
i=0
for <read each image from folder using os into img>:
im = cv2.imread(img.path,0)
im = np.expand_dims(im,2) #this is because keras needs 3d images and grayscale images are 2d
if not i:
X = data.create_earray(dataGroup,"X",atom,(0,)+im.shape,chunkshape=(20,20,20,1))
X.append(np.expand_dims(im,0)) #as appending require same dim.
i=1
But still when I run the code, it gives my ValueError saying the my object rank is 1 and X rank is 4. How is that possible when I am assigning X size using im. I even tried printing shape of im, it gives (278,278,1). So, what is the problem? I am using Pytables for first time, so dont know them in depth.
Adding a second answer with a more complicated write method plus an EArray.read example. Frankly, I prefer my simpler method (above) to create the EArray with obj= defined, and let Pytables handle the data structures. However, if you prefer to manage this yourself, see example 2 (below). Key items to note:
Atom definition has 4 dimensions, with 0 axis set to zero (defines
the direction that will be extended).
im = np.expand_dims(im,0) is done until AFTER im.shape is referenced in the
definition of the EArray shape at creation.
[UPDATED CODE BELOW]
import tables as tb, numpy as np
data = tb.open_file("image_data1.h5", mode='w')
dataGroup = data.create_group(data.root, 'MyData')
MyAtom = tb.Atom.from_dtype(np.dtype(np.uint32,(0,278,278,1)))
im = np.arange(278*278).reshape((278,278))
im = np.expand_dims(im,2)
X = data.create_earray(dataGroup,"X", MyAtom, (0,)+im.shape)
im = np.expand_dims(im,0)
X.append( im )
print ('flavor =', X.flavor )
print ('dim=', X.ndim, ', rows = ', X.nrows)
im = np.arange(278*278,278*278+278*278).reshape((278,278))
im = np.expand_dims(im,2)
im = np.expand_dims(im,0)
X.append( im )
print ('dim=', X.ndim, ', rows = ', X.nrows)
data.close()
Here are the lines you need to read the data from EArray X (with a couple of print statements to verify values in the corners). This should work so long as the EArray flavor is Numpy (as it is in my example). You can also use the out= parameter to specify a NumPy array to receive the output data. There are other methods to access EArray data, including .iterrows() to iterate, and .__getitem__() to slice with fancy indexing. Read the Pytables documentation if you want to do any of these.
Y_1 = X.read( 0 )
print (Y_1[0,0,0])
print (Y_1[-1,-1,-1])
Y_2 = X.read( 1 )
print (Y_2[0,0,0])
print (Y_2[-1,-1,-1])
First, note that you don't have to create the EArray before you load the first image dataset. Pytables is smart enough to determine the atom and shape definition from the first object.
It was hard for me to exercise your code without a complete example and your data. So, I created a very simple example that uses np.arange() to create a couple of (278,278) image arrays, then extends them in the 2 and 0 directions. Hopefully this mimics the data you are trying to load to the EArray. The 2 Pytables functions (file.create_earray and earray.append) create 2 rows of data, 1 for each "image". After running, open image_data1.h5 with HDFView and inspect the data.
Maybe this will help you understand how to load your images to HDF5 Earrays:
import tables as tb, numpy as np
data = tb.open_file("image_data1.h5", mode='w')
dataGroup = data.create_group(data.root, 'MyData')
im = np.arange(278*278).reshape((278,278))
im = np.expand_dims(im,2)
im = np.expand_dims(im,0)
X = data.create_earray( dataGroup,"X",obj=im )
print ('dim=', X.ndim, ', rows = ', X.nrows)
im = np.arange(278*278, 278*278+278*278).reshape((278,278))
im = np.expand_dims(im,2)
im = np.expand_dims(im,0)
X.append( im )
print ('dim=', X.ndim, ', rows = ', X.nrows)
data.close()

Move N elements in array from back to front

I have a text file contains 2 columns, I need to select one column of them as an array
which contains 200000 and cut N elements from this array and move them from back to front.
I used the following code:
import numpy as np
import glob
files = glob.glob("input/*.txt")
for file in files:
data_file = np.loadtxt(file)
2nd_columns = data_file [:,1]
2nd_columns_array = np.array(2nd_columns)
cut = 62859 # number of elements to cut
remain_points = 2nd_columns_array[:cut]
cut_points = 2nd_columns_array[cut:]
new_array = cut_points + remain_points
It doesn't work and gave me the following error:
ValueError: operands could not be broadcast together with shapes (137141,) (62859,)
any help, please??
It doesn't work because you are trying to add values stored in both arrays and they have different shapes.
One of the ways is to use numpy.hstack:
new_array = np.hstack((2nd_columns_array[cut:], 2nd_columns_array[:cut]))
Side notes:
with your code you will reorder only 2nd column of the last file since reordering is outside of the for loop
you don't need to store cut_poinsts nor remain_points in separate variables. You can operate directly on the 2nd_columns_array
you shouldn't name variables starting from a number
A simple method for this process is numpy.roll.
new_array = np.roll(2nd_column, cut)

Iterate through an array - returned error "Attempted to access A(:,3); index out of bounds because size(A)=[4,2]"

I am new to MATLAB and I am trying to write some code to iterate through an imported Excel array and do a calculation on each column of the array separately. I am receiving the error "Attempted to access A(:,3); index out of bounds because size(A)=[4,2]" and I am not sure why, as I have tried to give the output variable matrix dimensions. Here is the code:
filename = 'allsets.xlsx';
x1Range = 'B1:C5';
sheet = 1;
A = xlsread(filename,sheet,x1Range);
Mu1 = zeros(size(A));
Mu2 = zeros(size(B));
for i = 1:length(A)
% class mean
Mu1(:,i)=mean(A(:,i))';
end
x2Range = 'B6:C9';
B = xlsread(filename,sheet,x2Range);
for i = 1:length(B)
Mu2(:,i)=mean(B(:,i))';
end
Any suggestions on how to address this error would be very appreciated. Thank you!
use size(A,2) instead of length(A). length returns the size of the largest dimension, which is the number of rows in your case
length gives you the size in the dimension where a matrix is the longest. In your case, A is 3x2, so length returns 3, yet you are looping along the second dimension.

Resources