ValueError: setting an array element with a sequence - passing a list in a dictionary to DataGenerator - arrays

Question

I am working on a keras multilabel problem. In order to work with big amount of data to avoid memory issues, I implemented a custom data generator.
So far I work with a csv file with IDs, Filenames and their corresponding labels (21 in total), which looks like this:
Filename label1 label2 label3 label4 ... ID
abc1.jpg 1 0 0 1 ... id-1
def2.jpg 1 0 0 1 ... id-2
ghi3.jpg 1 0 0 1 ... id-3
...
I put the the ids and the labels in dictionaries which have the following output:
partition: {'train': ['id-1','id-2','id-3',...], 'validation': ['id-7','id-14','id-21',...]}
labels: {'id-0': [1, 0, 0, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1],
'id-1': [1, 0, 0, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1],
'id-2': [1, 0, 0, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1],
...}
All my images are converted to arrays and saved in single npy files. id-1.npy, id-2.npy...
Then I am executing my code:
import numpy as np
import keras
from keras.layers import *
from keras.models import Sequential
class DataGenerator(keras.utils.Sequence):
'Generates data for Keras'
def __init__(self, list_IDs, labels, batch_size=32, dim=(224,224), n_channels=3,
n_classes=21, shuffle=True):
'Initialization'
self.dim = dim
self.batch_size = batch_size
self.labels = labels
self.list_IDs = list_IDs
self.n_channels = n_channels
self.n_classes = n_classes
self.shuffle = shuffle
self.on_epoch_end()
def __len__(self):
'Denotes the number of batches per epoch'
return int(np.floor(len(self.list_IDs) / self.batch_size))
def __getitem__(self, index):
'Generate one batch of data'
# Generate indexes of the batch
indexes = self.indexes[index*self.batch_size:(index+1)*self.batch_size]
# Find list of IDs
list_IDs_temp = [self.list_IDs[k] for k in indexes]
# Generate data
X, y = self.__data_generation(list_IDs_temp)
return X, y
def on_epoch_end(self):
'Updates indexes after each epoch'
self.indexes = np.arange(len(self.list_IDs))
if self.shuffle == True:
np.random.shuffle(self.indexes)
def __data_generation(self, list_IDs_temp):
'Generates data containing batch_size samples' # X : (n_samples, *dim, n_channels)
# Initialization
X = np.empty((self.batch_size, *self.dim, self.n_channels))
y = np.empty((self.batch_size), dtype=int)
# Generate data
for i, ID in enumerate(list_IDs_temp):
# Store sample
X[i,] = np.load('Folder with npy files/' + ID + '.npy')
# Store class
y[i] = self.labels[ID]
return X, keras.utils.to_categorical(y, num_classes=self.n_classes)
# Parameters
params = {'dim': (224, 224),
'batch_size': 32,
'n_classes': 21,
'n_channels': 3,
'shuffle': True}
# Datasets
partition = partition
labels = labels
# Generators
training_generator = DataGenerator(partition['train'], labels, **params)
validation_generator = DataGenerator(partition['validation'], labels, **params)
# Design model
model = Sequential()
model.add(Conv2D(32, (3,3), input_shape=(224, 224, 3)))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2,2)))
...
model.add(Flatten())
model.add(Dense(64))
model.add(Activation('relu'))
model.add(Dense(21))
model.add(Activation('softmax'))
model.compile(loss='categorical_crossentropy', optimizer='rmsprop', metrics=['accuracy'])
# Train model on dataset
model.fit_generator(generator=training_generator,
validation_data=validation_generator)
and the following Error raises:
ValueError: setting an array element with a sequence
the following part of the error seems to be crucial:
<ipython-input-58-fedc63607310> in __getitem__(self, index)
31
32 # Generate data
---> 33 X, y = self.__data_generation(list_IDs_temp)
34
35 return X, y
<ipython-input-58-fedc63607310> in __data_generation(self, list_IDs_temp)
53
54 # Store class
---> 55 y[i] = self.labels[ID]
56
57 return X, keras.utils.to_categorical(y, num_classes=self.n_classes)
as soon as i replace labels from the beginning with the following, the code gets executed:
labels = {'id-0': 0,
'id-1': 2,
'id-2': 1,
...}
I still want to pass multiple labels to the DataGenerator, therefore I chose to put a list in the dictionary, as shown in the beginning, but this gives me the ValueError. How can I anyway pass multiple values for a single ID to the DataGenerator as suggested? What do I have to adjust?
A hint or a snippet of code I appreciate a lot.

Answer 1

If i understand well your code here is the problem :
y = np.empty((self.batch_size), dtype=int)
You are creating an emty 1D array, but here :
y[i] = self.labels[ID]
You are filling it with a sequence :
'id-0': [1, 0, 0, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1]
In order to work you need to create your label array with the shape of your batch_size and the lenght of your sequence :
y = np.empty((self.batch_size, len(sequence)), dtype=int)
EDIT
to_categorical is to encode categorical feature to be arrays like [0, 0, 0, 1], [0, 0, 1, 0], etc But you are feeding sequences, not categorical features.
By feeding sequences to your network, you don't want to one_hot encode it so replace :
return X, keras.utils.to_categorical(y, num_classes=self.n_classes)
by :
return X, y
Recommendation from last comment
The problem is that your Softmax activation will try to give the best score to the correct class, but here you give sequence array that softmax will interpret with multiple "correct class" :
For exemple : if you have 3 labels [1, 2, 3], by one_hot encoding you will have [1, 0, 0], [0, 1, 0], [0, 0, 1], there is only one "1" per encoded label array, one correct class, softmax will try to get this class score bigger as possible.
But in you case your are giving arrays with multiple "1's" :
with that : [1, 0, 1] softmax don't know to which class give the best score.
So i would recommand that, you start with your 21 labels [0,1,2,3, ..] and then you one_hot encode this array and you give it to your network.
If you really need that sequence, you have to find an other solution !
Hope i'm clear !