CNN with RGB input and BW binary output - arrays

Question

I am a beginner to deep learning and I am working with Keras built on top of Tensorflow. I am trying to using RGB images (540 x 360) resolution to predict bounding boxes.
My labels are binary (black/white) 2 dimensional np array of dimensions (540, 360) where all pixels are 0 except for the box edges which are a 1.
Like this:
[[0 0 0 0 0 0 ... 0]
[0 1 1 1 1 0 ... 0]
[0 1 0 0 1 0 ... 0]
[0 1 0 0 1 0 ... 0]
[0 1 1 1 1 0 ... 0]
[0 0 0 0 0 0 ... 0]]
There can be more than one bounding box in every picture. A typical image could look like this:
So, my input has the dimension (None, 540, 360, 3), output has dimensions (None, 540, 360) but if I add an internal array I can change the shape to (None, 540, 360, 1)
How would I define a CNN model such that my model could fit this criteria? How can I design a CNN with these inputs and outputs?

Answer 1

You have do differentiate between object detection and object segmentation. While both can be used for similar problems, the underlying CNN architectures look very different.
Object detection models use a CNN classification/regression architecure, where the output refers to the coordinates of the bounding boxes. It's common practice to use 4 values belonging to vertical center, horizontal center, width and height of each bounding box. Search for Faster R-CNN, SSD or YOLO to find popular object detection models for keras. In your case you would need to define a function that converts the current labels to the 4 coordinates I mentioned.
Object segmentation models commonly use an architecture referred to as encoder-decoder networks, where the original image is scaled down and compressed on the first half and then brought back to it's original resolution to predict a full image. Search for SegNet, U-Net or Tiramisu to find popular object segmentation models for keras. My own implementation of U-Net can be found here. In your case you would need to define a custom function, that fills all the 0s inside your bounding boxes with 1s. Understand that this solution will not predict bounding boxes as such, but segmentation maps showing regions of interest.
What is right for you, depends on what precisely you want to achieve. For getting actual bounding boxes you want to perform an object detection. However, if you're interested in highlighting regions of interest that go beyond rectangle windows a segmentation may be a better fit. In theory, you can use your rectangle labels for a segmentation, where the network will learn to create better masks than the inaccurate segmentation of the ground truth, provided you have enough data.

Answer 2

This is a simple example of how to write intermediate layers to achieve the output. You can use this as a starter code.
def model_360x540(input_shape=(360, 540, 3),num_classes=1):
inputs = Input(shape=input_shape)
# 360x540x3
downblock0 = Conv2D(32, (3, 3), padding='same')(inputs)
# 360x540x32
downblock0 = BatchNormalization()(block0)
downblock0 = Activation('relu')(block0)
downblock0_pool = MaxPooling2D((2, 2), strides=(2, 2))(block0)
# 180x270x32
centerblock0 = Conv2D(1024, (3, 3), padding='same')(downblock0_pool)
#180x270x1024
centerblock0 = BatchNormalization()(center)
centerblock0 = Activation('relu')(center)
upblock0 = UpSampling2D((2, 2))(centerblock0)
# 180x270x32
upblock0 = concatenate([downblock0 , upblock0], axis=3)
upblock0 = Activation('relu')(upblock0)
upblock0 = Conv2D(32, (3, 3), padding='same')(upblock0)
# 360x540x32
upblock0 = BatchNormalization()(upblock0)
upblock0 = Activation('relu')(upblock0)
classify = Conv2D(num_classes, (1, 1), activation='sigmoid')(upblock0)
#360x540x1
model = Model(inputs=inputs, outputs=classify)
model.compile(optimizer=RMSprop(lr=0.001), loss=bce_dice_loss, metrics=[dice_coeff])
return model
The downblock represents the block of layers which perform downsampling(MaxPooling2D).
The centerblock has no sampling layer.
The upblock represents the block of layers which perform up sampling(UpSampling2D).
So here you can see how (360,540,3) is being transformed to (360,540,1)
Basically, you can add such blocks of layers to create your model.
Also check out Holistically-Nested Edge Detection which will help you better with the edge detection task.
Hope this helps!

Answer 3

I have not worked with keras but I will provide a solution approach in more generalized way which can be used on any framework.
Here is full procedure.
Data preparation: I know your labels are edges of boxes which will also work but i will recommend that instead of edges you prepare dataset marking complete box like given in sample (I have marked for two boxes). Now your dataset have three classes (Box,Edges of box and background). Create two lists, Image and label.
Get a pre-trained model (RESNET-51 recommended) solver and train prototxt from here, Remove fc1000 layer and add de-convolution/up-sampling layers to match your input size. use paddding in first layer to make it square and crop in deconvolution layer to match input output dimensions.
Transfer weights from previously trained network (Original) and train your network.
Test your dataset and create bounding boxes using detected blobs.