Today we are going to consider wery interesting, exotic and novel concept: how to adopt any segmentation neural network to find instances. While others use Mask-RCNN (and of course it is good enought for many tasks), today we are going to propose another approach.
The main idea is the following: we will add additional class (object boundary). It will allow us to train model to segment objects with their boundaries, and then we will be able to use these boundaries to split objects instances via DTL query.
Deep Learning has a lot of applications in agriculture. Today we will build system to perform instance segmentation of plant leafs. We will use Aberystwyth Leaf Evaluation Dataset.
Here is the example of image:
You can download the tutorial data and reproduce entire experiment yourself.
We recommend to combine these techniques with other tutorials to obtain better results.
To solve this task we should combine a few approaches: sliding window and adding the additional class (automatically using DTL)
a) Download the original dataset images_and_annotations.zip
b) Choose the aberystwyth
import option
c) Drag images_and_annotations
dir to the upload window
d) Name this project aberystwyth
As you can see, the annotations look unusual. Here is an example of a single object:
Layer #1 ("action": "data"
) takes all data from project aberystwyth
and keeps classes as they are.
Layer #2 ("action": "split_masks"
) splits masks of class leaf
into connected components. As a result from one mask with a few leafs we will get separate masks for each leaf.
Layer #3 ("action": "supervisely"
) saves results to the new project aberystwyth_splitted
.
[{"dst": "$sample","src": ["aberystwyth/*"],"action": "data","settings": {"classes_mapping": "default"}},{"dst": "$sample1","src": ["$sample"],"action": "split_masks","settings": {"classes": ["leaf"]}},{"dst": "aberystwyth_splitted","src": ["$sample1"],"action": "supervisely","settings": {}}]
In this DTL query we apply transformations, filter images and split them into train and validation sets.
Layer #1 ("action": "data"
) takes all data from the project aberystwyth_splitted
and keeps classes as they are.
Layer #2 ("action": "flip"
) flips the data horisontally.
Layer #3 ("action": "multiply"
) creates 10 copies for each image.
Layer #6 ("action": "crop"
) performs random crops (from 20% to 25% in width and from 25% to 30% in height with respect to the image size)
Layer #7 ("action": "if"
) filters the data (sends the data containing at least one object to the first branch, all other data will be sent to null
).
Layer #8 ("action": "if"
) randomly splits the data into two branches: first branch - 95% (will be tagged as train
) and second branch - 5% (will be tagged as val
).
Layer #9 ("action": "tag"
) adds the tag train
to all input images.
Layer #10 ("action": "tag"
) adds the tag val
to all input images.
Layer #11 ("action": "rename"
) makes copies of objects of the class leaf
and renames them as tmp-contour
Layer #12 ("action": "line2bitmap"
) transforms the objects into their contours.
Layer #13 ("action": "crop"
) crops 3px stripes from each side to drop the boundary contours.
Layer #14 ("action": "supervisely"
) saves results to the new project aberystwyth_random_crop_train
.
[{"dst": "$sample","src": ["aberystwyth_splitted/*"],"action": "data","settings": {"classes_mapping": "default"}},{"dst": "$vflip","src": ["$sample"],"action": "flip","settings": {"axis": "vertical"}},{"dst": "$multi","src": ["$sample","$vflip"],"action": "multiply","settings": {"multiply": 10}},{"dst": "$crop","src": ["$multi"],"action": "crop","settings": {"random_part": {"width": {"max_percent": 25,"min_percent": 20},"height": {"max_percent": 30,"min_percent": 25}}}},{"dst": ["$filter","null"],"src": ["$crop"],"action": "if","settings": {"condition": {"min_objects_count": 1}}},{"dst": ["$to_train","$to_val"],"src": ["$filter"],"action": "if","settings": {"condition": {"probability": 0.95}}},{"dst": "$train","src": ["$to_train"],"action": "tag","settings": {"tag": "train","action": "add"}},{"dst": "$val","src": ["$to_val"],"action": "tag","settings": {"tag": "val","action": "add"}},{"dst": "$100","src": ["$train","$val"],"action": "duplicate_objects","settings": {"classes_mapping": {"leaf": "tmp-contour"}}},{"dst": "$101","src": ["$100"],"action": "line2bitmap","settings": {"width": 3,"classes_mapping": {"tmp-contour": "contour"}}},{"dst": "$102","src": ["$101"],"action": "crop","settings": {"sides": {"top": "3px","left": "3px","right": "3px","bottom": "3px"}}},{"dst": "zenodo_random_crop_train","src": ["$102"],"action": "supervisely","settings": {}}]
Detailed description.
Here are a few examples of images after this DTL query.
Basic step by step training guide is here. It is the same for all models inside Supervisely. Detailed information regarding training configs is here.
UNetV2 weights were initialized from the corresponding model from the Models list (UNetV2 with VGG weigths that was pretrained on ImageNet).
Resulting model will be named unet-leafs
. Project zenodo_random_crop_train
is used for training.
{"lr": 0.001,"epochs": 5,"val_every": 1,"batch_size": {"val": 3,"train": 3},"input_size": {"width": 512,"height": 512},"gpu_devices": [0,1,2],"data_workers": {"val": 0,"train": 3},"dataset_tags": {"val": "val","train": "train"},"special_classes": {"neutral": "neutral","background": "bg"},"weights_init_type": "transfer_learning"}
Training takes 15 minutes on three GPU devices.
After training the last model checkpoint is saved to "Neural Networks" list.
!!! hint "Monitor training charts and test various checkpoints" We recommend to carefully monitor the training charts to prevent overfitting or underfitting. Especially it is very important when we use a small training dataset. In this case restoring checkpoints is a key component to successful research.
Basic step by step inference guide is here. It is the same for all models inside Supervisely. Detailed information regarding inference configs is here.
We apply the model unet-leafs
to the project with test images.
Config for sliding window inference:
{"mode": {"save": false,"source": "sliding_window","window": {"width": 512,"height": 512},"min_overlap": {"x": 0,"y": 0}},"gpu_devices": [0],"model_classes": {"add_suffix": "_dl","save_classes": ["contour","leaf"]},"existing_objects": {"add_suffix": "","save_classes": []}}
Afterwards leaf instances can be extracted from the segmentation mask using DTL query like in this tutorial.
As you can see the results are good enough.
You can use any segmentation model with the approach described above.