About This Kernel

  • What is the purpose of the study?

I am working on Deep Learning and Computer Vision in Flying Automobile Project. The project I am working on are Semantic segmentation (Aerial images) during the flight of the vehicle to find suitable areas where the vehicle can land. To make volumetric control of the vehicle to these areas.

With this kernel, I have completed working on the Semantic segmentation

Content

  1. What is semantic segmentation
  2. Implementation of Segnet, FCN, UNet , PSPNet and other models in Keras
  3. I extracted Github codes

1.What is semantic segmentation

Source: https://divamgupta.com/image-segmentation/2019/06/06/deep-learning-semantic-segmentation-keras.html

Semantic image segmentation is the task of classifying each pixel in an image from a predefined set of classes. In the following example, different entities are classified.

Semantic segmentation of a bedroom image

In the above example, the pixels belonging to the bed are classified in the class “bed”, the pixels corresponding to the walls are labeled as “wall”, etc.

In particular, our goal is to take an image of size W x H x 3 and generate a W x H matrix containing the predicted class ID’s corresponding to all the pixels.

Image source: jeremyjordan.me

Usually, in an image with various entities, we want to know which pixel belongs to which entity, For example in an outdoor image, we can segment the sky, ground, trees, people, etc.

Semantic segmentation is different from object detection as it does not predict any bounding boxes around the objects. We do not distinguish between different instances of the same object. For example, there could be multiple cars in the scene and all of them would have the same label.

An example where there are multiple instances of the same object class

In order to perform semantic segmentation, a higher level understanding of the image is required. The algorithm should figure out the objects present and also the pixels which correspond to the object. Semantic segmentation is one of the essential tasks for complete scene understanding.

Dataset

The first step in training our segmentation model is to prepare the dataset. We would need the input RGB images and the corresponding segmentation images. If you want to make your own dataset, a tool like labelme or GIMP can be used to manually generate the ground truth segmentation masks.

Assign each class a unique ID. In the segmentation images, the pixel value should denote the class ID of the corresponding pixel. This is a common format used by most of the datasets and keras_segmentation. For the segmentation maps, do not use the jpg format as jpg is lossy and the pixel values might change. Use bmp or png format instead. And of course, the size of the input image and the segmentation image should be the same.

In the following example, pixel (0,0) is labeled as class 2, pixel (3,4) is labeled as class 1 and rest of the pixels are labeled as class 0.

In [1]:
import cv2
import numpy as np

ann_img = np.zeros((30,30,3)).astype('uint8')
ann_img[ 3 , 4 ] = 1 # this would set the label of pixel 3,4 as 1
ann_img[ 0 , 0 ] = 2 # this would set the label of pixel 0,0 as 2

After generating the segmentation images, place them in the training/testing folder. Make separate folders for input images and the segmentation images. The file name of the input image and the corresponding segmentation image should be the same. For this tutorial we would be using a data-set which is already prepared. You can download it from here (Aerial Semantic Segmentation Drone Dataset).

In [2]:
from PIL import Image
import matplotlib.pyplot as plt
%matplotlib inline

original_image = "/kaggle/input/semantic-drone-dataset/semantic_drone_dataset/original_images/001.jpg"
label_image_semantic = "/kaggle/input/semantic-drone-dataset/semantic_drone_dataset/label_images_semantic/001.png"

fig, axs = plt.subplots(1, 2, figsize=(16, 8), constrained_layout=True)

axs[0].imshow( Image.open(original_image))
axs[0].grid(False)

label_image_semantic = Image.open(label_image_semantic)
label_image_semantic = np.asarray(label_image_semantic)
axs[1].imshow(label_image_semantic)
axs[1].grid(False)

2.Implementation of Segnet, FCN, UNet , PSPNet and other models in Keras

Source Github Link: https://github.com/divamgupta/image-segmentation-keras

Models Following models are supported:

model_name Base Model Segmentation Model
fcn_8 Vanilla CNN FCN8
fcn_32 Vanilla CNN FCN8
fcn_8_vgg VGG 16 FCN8
fcn_32_vgg VGG 16 FCN32
fcn_8_resnet50 Resnet-50 FCN32
fcn_32_resnet50 Resnet-50 FCN32
fcn_8_mobilenet MobileNet FCN32
fcn_32_mobilenet MobileNet FCN32
pspnet Vanilla CNN PSPNet
vgg_pspnet VGG 16 PSPNet
resnet50_pspnet Resnet-50 PSPNet
unet_mini Vanilla Mini CNN U-Net
unet Vanilla CNN U-Net
vgg_unet VGG 16 U-Net
resnet50_unet Resnet-50 U-Net
mobilenet_unet MobileNet U-Net
segnet Vanilla CNN Segnet
vgg_segnet VGG 16 Segnet
resnet50_segnet Resnet-50 Segnet
mobilenet_segnet MobileNet Segnet
In [3]:
!pip install keras-segmentation
Collecting keras-segmentation

  Downloading keras_segmentation-0.3.0.tar.gz (23 kB)

Requirement already satisfied: Keras>=2.0.0 in /opt/conda/lib/python3.6/site-packages (from keras-segmentation) (2.3.1)

Collecting imageio==2.5.0

  Downloading imageio-2.5.0-py3-none-any.whl (3.3 MB)

     |████████████████████████████████| 3.3 MB 7.1 MB/s 

Collecting imgaug==0.2.9

  Downloading imgaug-0.2.9-py2.py3-none-any.whl (753 kB)

     |████████████████████████████████| 753 kB 52.1 MB/s 

Requirement already satisfied: opencv-python in /opt/conda/lib/python3.6/site-packages (from keras-segmentation) (4.2.0.32)

Requirement already satisfied: tqdm in /opt/conda/lib/python3.6/site-packages (from keras-segmentation) (4.42.1)

Requirement already satisfied: six>=1.9.0 in /opt/conda/lib/python3.6/site-packages (from Keras>=2.0.0->keras-segmentation) (1.14.0)

Requirement already satisfied: keras-preprocessing>=1.0.5 in /opt/conda/lib/python3.6/site-packages (from Keras>=2.0.0->keras-segmentation) (1.1.0)

Requirement already satisfied: scipy>=0.14 in /opt/conda/lib/python3.6/site-packages (from Keras>=2.0.0->keras-segmentation) (1.4.1)

Requirement already satisfied: numpy>=1.9.1 in /opt/conda/lib/python3.6/site-packages (from Keras>=2.0.0->keras-segmentation) (1.18.1)

Requirement already satisfied: pyyaml in /opt/conda/lib/python3.6/site-packages (from Keras>=2.0.0->keras-segmentation) (5.3)

Requirement already satisfied: keras-applications>=1.0.6 in /opt/conda/lib/python3.6/site-packages (from Keras>=2.0.0->keras-segmentation) (1.0.8)

Requirement already satisfied: h5py in /opt/conda/lib/python3.6/site-packages (from Keras>=2.0.0->keras-segmentation) (2.10.0)

Requirement already satisfied: pillow in /opt/conda/lib/python3.6/site-packages (from imageio==2.5.0->keras-segmentation) (5.4.1)

Requirement already satisfied: matplotlib in /opt/conda/lib/python3.6/site-packages (from imgaug==0.2.9->keras-segmentation) (3.0.3)

Requirement already satisfied: Shapely in /opt/conda/lib/python3.6/site-packages (from imgaug==0.2.9->keras-segmentation) (1.7.0)

Requirement already satisfied: scikit-image>=0.11.0 in /opt/conda/lib/python3.6/site-packages (from imgaug==0.2.9->keras-segmentation) (0.16.2)

Requirement already satisfied: cycler>=0.10 in /opt/conda/lib/python3.6/site-packages (from matplotlib->imgaug==0.2.9->keras-segmentation) (0.10.0)

Requirement already satisfied: kiwisolver>=1.0.1 in /opt/conda/lib/python3.6/site-packages (from matplotlib->imgaug==0.2.9->keras-segmentation) (1.1.0)

Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /opt/conda/lib/python3.6/site-packages (from matplotlib->imgaug==0.2.9->keras-segmentation) (2.4.6)

Requirement already satisfied: python-dateutil>=2.1 in /opt/conda/lib/python3.6/site-packages (from matplotlib->imgaug==0.2.9->keras-segmentation) (2.8.1)

Requirement already satisfied: PyWavelets>=0.4.0 in /opt/conda/lib/python3.6/site-packages (from scikit-image>=0.11.0->imgaug==0.2.9->keras-segmentation) (1.1.1)

Requirement already satisfied: networkx>=2.0 in /opt/conda/lib/python3.6/site-packages (from scikit-image>=0.11.0->imgaug==0.2.9->keras-segmentation) (2.4)

Requirement already satisfied: setuptools in /opt/conda/lib/python3.6/site-packages (from kiwisolver>=1.0.1->matplotlib->imgaug==0.2.9->keras-segmentation) (45.2.0.post20200210)

Requirement already satisfied: decorator>=4.3.0 in /opt/conda/lib/python3.6/site-packages (from networkx>=2.0->scikit-image>=0.11.0->imgaug==0.2.9->keras-segmentation) (4.4.1)

Building wheels for collected packages: keras-segmentation

  Building wheel for keras-segmentation (setup.py) ... - \ done

  Created wheel for keras-segmentation: filename=keras_segmentation-0.3.0-py3-none-any.whl size=29072 sha256=4100191af3da7bf8ed66cfe8c8493f3177e4a588eb2193e098cb7aa91940389e

  Stored in directory: /root/.cache/pip/wheels/9c/c6/06/1af941e410edfca83a7c568123a136263351cbecc34a3b3653

Successfully built keras-segmentation

ERROR: albumentations 0.4.3 has requirement imgaug<0.2.7,>=0.2.5, but you'll have imgaug 0.2.9 which is incompatible.

Installing collected packages: imageio, imgaug, keras-segmentation

  Attempting uninstall: imageio

    Found existing installation: imageio 2.6.1

    Uninstalling imageio-2.6.1:

      Successfully uninstalled imageio-2.6.1

  Attempting uninstall: imgaug

    Found existing installation: imgaug 0.2.6

    Uninstalling imgaug-0.2.6:

      Successfully uninstalled imgaug-0.2.6

Successfully installed imageio-2.5.0 imgaug-0.2.9 keras-segmentation-0.3.0

Train

In [4]:
kaggle_commit = True

epochs = 20
if kaggle_commit:
    epochs = 5
In [5]:
from keras_segmentation.models.unet import vgg_unet

n_classes = 23 # Aerial Semantic Segmentation Drone Dataset tree, gras, other vegetation, dirt, gravel, rocks, water, paved area, pool, person, dog, car, bicycle, roof, wall, fence, fence-pole, window, door, obstacle
model = vgg_unet(n_classes=n_classes ,  input_height=416, input_width=608  )

model.train( 
    train_images =  "/kaggle/input/semantic-drone-dataset/semantic_drone_dataset/original_images/",
    train_annotations = "/kaggle/input/semantic-drone-dataset/semantic_drone_dataset/label_images_semantic/",
    checkpoints_path = "vgg_unet" , epochs=epochs
)
Using TensorFlow backend.
Downloading data from https://github.com/fchollet/deep-learning-models/releases/download/v0.1/vgg16_weights_tf_dim_ordering_tf_kernels_notop.h5
58892288/58889256 [==============================] - 2s 0us/step
  0%|          | 0/400 [00:00<?, ?it/s]
Verifying training dataset
100%|██████████| 400/400 [04:19<00:00,  1.54it/s]
Dataset verified! 
Epoch 1/5
512/512 [==============================] - 696s 1s/step - loss: 1.4911 - accuracy: 0.5929
saved  vgg_unet.0
Epoch 2/5
512/512 [==============================] - 692s 1s/step - loss: 1.1836 - accuracy: 0.6473
saved  vgg_unet.1
Epoch 3/5
512/512 [==============================] - 692s 1s/step - loss: 1.0570 - accuracy: 0.6818
saved  vgg_unet.2
Epoch 4/5
512/512 [==============================] - 705s 1s/step - loss: 0.9753 - accuracy: 0.7066
saved  vgg_unet.3
Epoch 5/5
512/512 [==============================] - 689s 1s/step - loss: 0.9028 - accuracy: 0.7287
saved  vgg_unet.4

Prediction

In [6]:
import time
from PIL import Image
import matplotlib.pyplot as plt
%matplotlib inline

start = time.time()

input_image = "/kaggle/input/semantic-drone-dataset/semantic_drone_dataset/original_images/001.jpg"
out = model.predict_segmentation(
    inp=input_image,
    out_fname="out.png"
)

fig, axs = plt.subplots(1, 3, figsize=(20, 20), constrained_layout=True)

img_orig = Image.open(input_image)
axs[0].imshow(img_orig)
axs[0].set_title('original image-001.jpg')
axs[0].grid(False)

axs[1].imshow(out)
axs[1].set_title('prediction image-out.png')
axs[1].grid(False)

validation_image = "/kaggle/input/semantic-drone-dataset/semantic_drone_dataset/label_images_semantic/001.png"
axs[2].imshow( Image.open(validation_image))
axs[2].set_title('true label image-001.png')
axs[2].grid(False)

done = time.time()
elapsed = done - start