Related
Imagine you have a bit array (any data type is okay. e.g. list, np.array, bitarray, bitmap, etc of booleans) that is randomly filled. What is the fastest way to “forward fill” (left to right, or 0th index to nth index) that array in Python such that n bits get set to 1 following each bit already set to 1?
For example, take the array below:
[01000100000]
Given n=2 the forward filled array would be:
[01110111000]
edit
Assume that the input is a bit array of 10,000 elements, of which a random 20% are true, and n=25. This can be represented as a python list with 10,000 boolean elements, of which 20% are True. This could also be represented as a set with 2,000 int elements between 0 and 10,000.
edit 2
To get things started, here are some examples using the parameters above:
new = set()
new.update(*[range(i, i+25) for i in existing])
# 2.34 ms ± 56.9 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
new = BitMap() # This is a pyroaring BitMap
for e in existing:
new.add_range(e, e+25)
# 461 µs ± 6.02 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
I have addressed several datatypes below. There are no timings given, you might want to time the statement setting ans or refactor-in functions to time at the granularity that makes sense to you.
# -*- coding: utf-8 -*-
"""
Created on Sun Dec 19 09:08:56 2021
for: https://stackoverflow.com/questions/70397220/most-efficient-way-to-forward-fill-a-bit-array
#author: paddy
"""
from random import sample
n = 2 # bits to the right of set bits to also set
elements = 17
true_percent = 20.0
#%% Using arbitrary precision int
print("\nUsing arbitrary precision int.\n".upper())
from operator import or_
from functools import reduce
# Set some random bits True
bits = sum(1 << r
for r in sample(range(elements), int(true_percent/100 * elements)))
# Set n right-adjacent bits.
ans = reduce(or_, (bits >> x for x in range(n+1)), 0)
# Print
print(f"Random bits = {bits:0{elements}b}")
if 1:
print()
for x in range(n+1):
print(f" {bits >> x:0{elements}b}")
print()
print(f"Answer = {ans:0{elements}b}\n")
#%% Using list.
print("\nUsing list.\n".upper())
from operator import or_
from functools import reduce
bits = [0] * elements
# Set some random bits to 1
for r in sample(range(elements), int(true_percent/100 * elements)):
bits[r] = 1
# Set n right-adjacent bits.
# [0]*x is padding bits on the left.
# zip(*(list1, list2,..)) returns the n'th elements on list1, list2,...
# int(any(...)) or's them.
ans = [int(any(shifts))
for shifts in zip(*([0]*x + bits for x in range(n+1)))]
# Print
print(f"Random bits = {bits}")
if 1:
print()
for x in range(n+1):
print(f" {[0]*x + bits}")
print()
print(f"Answer = {ans}\n")
#%% Using numpy.
# Adapt the list solution to use numpy operators on numpy arrays
#%% Using other ordered collections such as str.
# Convert to and from int solution.
Sample Output:
USING ARBITRARY PRECISION INT.
Random bits = 01000000010000010
01000000010000010
00100000001000001
00010000000100000
Answer = 01110000011100011
USING LIST.
Random bits = [0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0]
[0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0]
[0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0]
[0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0]
Answer = [0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0]
I am reading RGB images in tensorflow.js and need to find out the number of black pixels [0,0,0] in that image? Is there any method to do this operation?
A tensor image is a 3d tensor without the transparency
Given t a tensor image, the following will return the number of black pixels
t = tf.tensor([0, 0, 0, 1, 0, 0, 0, 0, 0, 2, 1, 0], [2, 2, 3])
t.sum(-1).equal(tf.zeros(t.shape.slice(0, -1))).sum().print() // 2
We sum each pixel value (sum over the axis -1) and create a 2d tensor with same width and height as initial tensor. Both tensors are compared and the last tf.sum will count where the first tf.sum tensor has 0 values.
I'm trying to write data to different planes to opengl texture, is it possible?
I tried following code
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, 1, 1, 0, GL_RGBA, GL_UNSIGNED_BYTE, 0);
glTexSubImage2D(GL_TEXTURE_2D, 0, 0, 0, 1, 1, GL_RED, GL_UNSIGNED_BYTE, (void*)&pixel);
glTexSubImage2D(GL_TEXTURE_2D, 0, 0, 0, 1, 1, GL_GREEN, GL_UNSIGNED_BYTE, (void*)&pixel);
It doesn't work as I expected, triangle is green instead of yellow, pixel value is 255.
No it is not. The missing channels are "filled" with 0.0 for red, green and blue, respectively 1.0 for the alpha channel.
See OpenGL 4.6 API Core Profile Specification - 8.4.4.4 Final Expansion to RGBA:
[...] Each group is converted to a group of 4 elements as follows: if a group does not contain an A element, then A is added and set to one for integer components or 1.0 for floating-point components. If any of R, G, or B is missing from the group, each missing element is added and assigned a value of 0 for integer components or 0.0 for floating-point components.
I am working on a keras multilabel problem. In order to work with big amount of data to avoid memory issues, I implemented a custom data generator.
So far I work with a csv file with IDs, Filenames and their corresponding labels (21 in total), which looks like this:
Filename label1 label2 label3 label4 ... ID
abc1.jpg 1 0 0 1 ... id-1
def2.jpg 1 0 0 1 ... id-2
ghi3.jpg 1 0 0 1 ... id-3
...
I put the the ids and the labels in dictionaries which have the following output:
partition: {'train': ['id-1','id-2','id-3',...], 'validation': ['id-7','id-14','id-21',...]}
labels: {'id-0': [1, 0, 0, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1],
'id-1': [1, 0, 0, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1],
'id-2': [1, 0, 0, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1],
...}
All my images are converted to arrays and saved in single npy files. id-1.npy, id-2.npy...
Then I am executing my code:
import numpy as np
import keras
from keras.layers import *
from keras.models import Sequential
class DataGenerator(keras.utils.Sequence):
'Generates data for Keras'
def __init__(self, list_IDs, labels, batch_size=32, dim=(224,224), n_channels=3,
n_classes=21, shuffle=True):
'Initialization'
self.dim = dim
self.batch_size = batch_size
self.labels = labels
self.list_IDs = list_IDs
self.n_channels = n_channels
self.n_classes = n_classes
self.shuffle = shuffle
self.on_epoch_end()
def __len__(self):
'Denotes the number of batches per epoch'
return int(np.floor(len(self.list_IDs) / self.batch_size))
def __getitem__(self, index):
'Generate one batch of data'
# Generate indexes of the batch
indexes = self.indexes[index*self.batch_size:(index+1)*self.batch_size]
# Find list of IDs
list_IDs_temp = [self.list_IDs[k] for k in indexes]
# Generate data
X, y = self.__data_generation(list_IDs_temp)
return X, y
def on_epoch_end(self):
'Updates indexes after each epoch'
self.indexes = np.arange(len(self.list_IDs))
if self.shuffle == True:
np.random.shuffle(self.indexes)
def __data_generation(self, list_IDs_temp):
'Generates data containing batch_size samples' # X : (n_samples, *dim, n_channels)
# Initialization
X = np.empty((self.batch_size, *self.dim, self.n_channels))
y = np.empty((self.batch_size), dtype=int)
# Generate data
for i, ID in enumerate(list_IDs_temp):
# Store sample
X[i,] = np.load('Folder with npy files/' + ID + '.npy')
# Store class
y[i] = self.labels[ID]
return X, keras.utils.to_categorical(y, num_classes=self.n_classes)
# Parameters
params = {'dim': (224, 224),
'batch_size': 32,
'n_classes': 21,
'n_channels': 3,
'shuffle': True}
# Datasets
partition = partition
labels = labels
# Generators
training_generator = DataGenerator(partition['train'], labels, **params)
validation_generator = DataGenerator(partition['validation'], labels, **params)
# Design model
model = Sequential()
model.add(Conv2D(32, (3,3), input_shape=(224, 224, 3)))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2,2)))
...
model.add(Flatten())
model.add(Dense(64))
model.add(Activation('relu'))
model.add(Dense(21))
model.add(Activation('softmax'))
model.compile(loss='categorical_crossentropy', optimizer='rmsprop', metrics=['accuracy'])
# Train model on dataset
model.fit_generator(generator=training_generator,
validation_data=validation_generator)
and the following Error raises:
ValueError: setting an array element with a sequence
the following part of the error seems to be crucial:
<ipython-input-58-fedc63607310> in __getitem__(self, index)
31
32 # Generate data
---> 33 X, y = self.__data_generation(list_IDs_temp)
34
35 return X, y
<ipython-input-58-fedc63607310> in __data_generation(self, list_IDs_temp)
53
54 # Store class
---> 55 y[i] = self.labels[ID]
56
57 return X, keras.utils.to_categorical(y, num_classes=self.n_classes)
as soon as i replace labels from the beginning with the following, the code gets executed:
labels = {'id-0': 0,
'id-1': 2,
'id-2': 1,
...}
I still want to pass multiple labels to the DataGenerator, therefore I chose to put a list in the dictionary, as shown in the beginning, but this gives me the ValueError. How can I anyway pass multiple values for a single ID to the DataGenerator as suggested? What do I have to adjust?
A hint or a snippet of code I appreciate a lot.
If i understand well your code here is the problem :
y = np.empty((self.batch_size), dtype=int)
You are creating an emty 1D array, but here :
y[i] = self.labels[ID]
You are filling it with a sequence :
'id-0': [1, 0, 0, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1]
In order to work you need to create your label array with the shape of your batch_size and the lenght of your sequence :
y = np.empty((self.batch_size, len(sequence)), dtype=int)
EDIT
to_categorical is to encode categorical feature to be arrays like [0, 0, 0, 1], [0, 0, 1, 0], etc But you are feeding sequences, not categorical features.
By feeding sequences to your network, you don't want to one_hot encode it so replace :
return X, keras.utils.to_categorical(y, num_classes=self.n_classes)
by :
return X, y
Recommendation from last comment
The problem is that your Softmax activation will try to give the best score to the correct class, but here you give sequence array that softmax will interpret with multiple "correct class" :
For exemple : if you have 3 labels [1, 2, 3], by one_hot encoding you will have [1, 0, 0], [0, 1, 0], [0, 0, 1], there is only one "1" per encoded label array, one correct class, softmax will try to get this class score bigger as possible.
But in you case your are giving arrays with multiple "1's" :
with that : [1, 0, 1] softmax don't know to which class give the best score.
So i would recommand that, you start with your 21 labels [0,1,2,3, ..] and then you one_hot encode this array and you give it to your network.
If you really need that sequence, you have to find an other solution !
Hope i'm clear !
I have a list of 20-length vectors that I would like to multiply each of those with one of three matrices depending on the length vectors' names. Here is my unsuccessful attempt. Please suggest how I improve my code. Any help is much appreciated!
for (i in 1:length(List)){
.$Value=ifelse(names(List) %in% c("a","b","c"),matrixA%*%.$Value,ifelse(names(List) %in% c("d","e"),matrixB%*%.$Value, matrixC%*%.$Value))
}
Part of my list and the matrices are included below.
list(a = structure(c(3, 0, 0, 5, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 1, 10, 0, 0, 1, 1), .Dim = c(20L, 1L)), b = structure(c(2,
0, 0, 0, 0, 9, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 1, 0, 6, 0), .Dim = c(20L,
1L)))
matrixA <- diag(2,20)
matrixB <- diag(1,20)
matrixC <- diag(4,20)
So... Not sure I understand. But it seems like if the list has name a, b or c you want to multiply it to matrixA, if it's d or e you want to multiply it to matrixB and if neither, the values should be multiplied to matrixC.
Let's use your example.
zz <- list(a = structure(c(3, 0, 0, 5, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 1, 10, 0, 0, 1, 1), .Dim = c(20L, 1L)), b = structure(c(2,
0, 0, 0, 0, 9, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 1, 0, 6, 0), .Dim = c(20L,
1L)))
matrixA <- diag(2,20)
matrixB <- diag(1,20)
matrixC <- diag(4,20)
This is probably not the best solution but it is a simple one. I just made a few tweaks to your ideia so it would work, a matrix needs to be protected by list() (because a list is a vector, and ifelse works with vectors) inside an ifelse() otherwise you only get the first element. This will return you a list of the results.
results <- lapply(seq_along(zz), function(i){
ifelse(names(zz[i]) %in% c("a","b","c"),list(matrixA%*%zz[[i]]),
ifelse(names(zz[i]) %in% c("d","e"), list(matrixB%*%zz[[i]]), list(matrixC%*%zz[[i]])))
})
I used lapply to apply the sequence to (1 to length of zz) to the defined function. For each i the function looks at the name of i element zz (zz[i] returns the element of the list with its name) and if it satisfies the condition we multiply the content of the i element of zz (zz[[i]] just returns the content of the i element of the list without its name) by a predefined matrix.
This also works and you don't need to protect the matrix using list() which is kinda of a bother.
results <- lapply(seq_along(zz), function(i){
if(names(zz[i]) %in% c("a","b","c")) matrixA%*%zz[[i]] else
if(names(zz[i]) %in% c("d","e")) matrixB%*%zz[[i]]
else matrixC%*%zz[[i]]
})
Edit: #akrun answer is way more beautiful and short.
May be this helps
nm1 <- paste0("matrix", toupper(names(lst1)))
Map(crossprod, lst1, mget(nm1))