TinyMS SSD300 Tutorial

In this tutorial, using TinyMS API to train/serve an SSD300 model will be demonstrated.

Prerequisite

• Ubuntu: 18.04

• Python: 3.7.x

• Flask: 1.1.2

• MindSpore: CPU-1.1.1

• TinyMS: 0.1.0

• numpy: 1.17.5

• Pillow: 8.1.0

• pip: 21.0.1

• requests: 2.18.4

Introduction

TinyMS is a high-level API which is designed for amateur of deep learning. It minimizes the number of actions of users required to construct, train, evaluate and serve a model. TinyMS also provides tutorials and documentations for developers.

This tutorial consists of six parts, constructing the model, downloading dataset, training, define servable json, starting server and making predictions in which the server will be run in a sub process.

import os
import json
import time
import tinyms as ts
import xml.etree.ElementTree as et

from PIL import Image
from tinyms import context, layers, primitives as P, Tensor
from tinyms.serving import start_server, predict, list_servables, shutdown, server_started
from tinyms.data import VOCDataset, download_dataset
from tinyms.vision import voc_transform, coco_eval, ImageViewer
from tinyms.model import Model, ssd300_mobilenetv2, ssd300_infer
from tinyms.losses import net_with_loss
from tinyms.optimizers import Momentum
from tinyms.callbacks import ModelCheckpoint, CheckpointConfig, LossMonitor, TimeMonitor
from tinyms.utils.train.lr_generator import mobilenetv2_lr as ssd300_lr
from tinyms.initializers import initializer, TruncatedNormal

[WARNING] ME(14174:140126738016064,MainProcess):2021-03-19-15:33:42.136.028 [mindspore/ops/operations/array_ops.py:2302] WARN_DEPRECATED: The usage of Pack is deprecated. Please use Stack.

WARNING: 'ControlDepend' is deprecated from version 1.1 and will be removed in a future version, use 'Depend' instead.

1. Construct the model

# build network
net = ssd300_mobilenetv2(class_num=21)

2. Download dataset

The VOC dataset will be downloaded if voc folder didn’t exist at the root. If voc folder already exists, this step will not be performed.

# download the dataset
voc_path = '/root/voc'

if not os.path.exists(voc_path):
    download_dataset('voc', '/root')
    print('************Download complete*************')
else:
    print('************Dataset already exists.**************')

************** Downloading the VOC2007 dataset **************
[████████████████████████████████████████████████████████████████████████████████████████████████████] 100.00%
============== /root/voc/VOCtrainval_06-Nov-2007.tar is ready ==============
************Download complete*************

3. Train the model & evaluation

The dataset for both training and evaluation will be defined here, and the parameters for training also set in this block. A trained ckpt file will be saved to /etc/tinyms/serving/ssd300 folder for later use, meanwhile the evaluation will be performed and the Accuracy can be checked

Notice: Since training SSD300 on CPU is time consuming, we recommend skip training and using provided ckpt files to run.

class TrainingWrapper(layers.Layer):
    """
    Encapsulation class of SSD300 network training.

    Append an optimizer to the training network after that the construct
    function can be called to create the backward graph.

    Args:
        network (Layer): The training network. Note that loss function should have been added.
        optimizer (Optimizer): Optimizer for updating the weights.
        sens (float): The adjust parameter. Default: 1.0.
    """

    def __init__(self, network, optimizer, sens=1.0):
        super(TrainingWrapper, self).__init__(auto_prefix=False)
        self.network = network
        self.network.set_grad()
        self.weights = ts.ParameterTuple(network.trainable_params())
        self.optimizer = optimizer
        self.grad = P.GradOperation(get_by_list=True, sens_param=True)
        self.sens = sens
        self.hyper_map = P.HyperMap()

    def construct(self, *args):
        weights = self.weights
        loss = self.network(*args)
        sens = P.Fill()(P.DType()(loss), P.Shape()(loss), self.sens)
        grads = self.grad(self.network, weights)(*args, sens)
        return P.depend(loss, self.optimizer(grads))

def create_voc_label(voc_dir, voc_cls, usage='val'):
    """Get image path and annotation from VOC."""
    if not os.path.isdir(voc_dir):
        raise ValueError(f'Cannot find {voc_dir} dataset path.')
    anno_dir = voc_dir
    if os.path.isdir(os.path.join(voc_dir, 'Annotations')):
        anno_dir = os.path.join(voc_dir, 'Annotations')

    cls_map = {name: i for i, name in enumerate(voc_cls)}
    # Fetch the specific xml files path
    xml_files = []
    with open(os.path.join(voc_dir, 'ImageSets', 'Main', usage+'.txt'), 'r') as f:
        for line in f:
            xml_files.append(line.strip('\n')+'.xml')

    json_dict = {"images": [], "type": "instances", "annotations": [],
                 "categories": []}
    bnd_id = 1
    for xml_file in xml_files:
        img_id = xml_files.index(xml_file)
        tree = et.parse(os.path.join(anno_dir, xml_file))
        root_node = tree.getroot()
        file_name = root_node.find('filename').text

        for obj in root_node.iter('object'):
            cls_name = obj.find('name').text
            if cls_name not in cls_map:
                print(f'Label "{cls_name}" not in "{cls_map}"')
                continue

            bnd_box = obj.find('bndbox')
            x_min = int(float(bnd_box.find('xmin').text)) - 1
            y_min = int(float(bnd_box.find('ymin').text)) - 1
            x_max = int(float(bnd_box.find('xmax').text)) - 1
            y_max = int(float(bnd_box.find('ymax').text)) - 1
            o_width = abs(x_max - x_min)
            o_height = abs(y_max - y_min)
            ann = {'area': o_width * o_height, 'iscrowd': 0,
                   'image_id': img_id,
                   'bbox': [x_min, y_min, o_width, o_height],
                   'category_id': cls_map[cls_name], 'id': bnd_id,
                   'ignore': 0,
                   'segmentation': []}
            json_dict['annotations'].append(ann)
            bnd_id = bnd_id + 1

        size = root_node.find("size")
        width = int(size.find('width').text)
        height = int(size.find('height').text)
        image = {'file_name': file_name, 'height': height, 'width': width,
                 'id': img_id}
        json_dict['images'].append(image)

    for cls_name, cid in cls_map.items():
        cat = {'supercategory': 'none', 'id': cid, 'name': cls_name}
        json_dict['categories'].append(cat)

    anno_file = os.path.join(anno_dir, 'annotation.json')
    with open(anno_file, 'w') as f:
        json.dump(json_dict, f)
    return anno_file


# check ckpt folder exists or not
ckpt_folder = '/etc/tinyms/serving/ssd300'
ckpt_path = '/etc/tinyms/serving/ssd300/ssd300.ckpt'
if not os.path.exists(ckpt_folder):
    !mkdir -p  /etc/tinyms/serving/ssd300
else:
    print('ssd300 ckpt folder already exists')

# set parameters
epoch_size = 800 # default is 800
batch_size = 32
voc_path = '/root/voc/VOCdevkit/VOC2007'

# set environment parameters
context.set_context(mode=context.GRAPH_MODE, device_target="CPU")
dataset_sink_mode = False

# set dataset parameters
train_dataset = VOCDataset(voc_path, task='Detection', usage='trainval', num_parallel_workers=4, shuffle=True, decode=True)
train_dataset = voc_transform.apply_ds(train_dataset, repeat_size=1, batch_size=batch_size, num_parallel_workers=4, is_training=True)
eval_dataset = VOCDataset(voc_path, task='Detection', usage='val', num_parallel_workers=4, shuffle=True, decode=True)
eval_dataset = voc_transform.apply_ds(eval_dataset, repeat_size=1, batch_size=batch_size, num_parallel_workers=4, is_training=False)
dataset_size = train_dataset.get_dataset_size()
total = eval_dataset.get_dataset_size()

# define the loss function
net = net_with_loss(net)
params = net.trainable_params()
for p in params:
        if 'beta' not in p.name and 'gamma' not in p.name and 'bias' not in p.name:
            p.set_data(initializer(TruncatedNormal(0.02), p.data.shape, p.data.dtype))


# define the optimizer
pre_trained_epoch_size = 0
save_checkpoint_epochs = 10
lr = 0.01
lr = ssd300_lr(global_step=pre_trained_epoch_size * dataset_size,
                lr_init=0.001, lr_end=0.001 * lr, lr_max=lr,
                warmup_epochs=2, total_epochs=epoch_size,
                steps_per_epoch=dataset_size)
loss_scale = 1.0
opt = Momentum(filter(lambda x: x.requires_grad, net.get_parameters()), lr,0.9, 1.5e-4, loss_scale)
model = Model(TrainingWrapper(net, opt, loss_scale))
model.compile()
ckpoint_cb = ModelCheckpoint(prefix="ssd300", config=CheckpointConfig(
    save_checkpoint_steps=save_checkpoint_epochs * dataset_size,
    keep_checkpoint_max=10))

print('************************Start training*************************')
model.train(epoch_size, train_dataset, callbacks=[ckpoint_cb, LossMonitor(), TimeMonitor(data_size=dataset_size)],
            dataset_sink_mode=dataset_sink_mode)
model.save_checkpoint(ckpt_path)
print('************************Finished training*************************')

eval_net = ssd300_infer(class_num=21)
model = Model(eval_net)
model.load_checkpoint(ckpt_path)
# perform the model predict operation
print("\n========================================\n")
print("total images num: ", total)
print("Processing, please wait a moment...")
start = time.time()
pred_data = []
id_iter = 0

for data in eval_dataset.create_dict_iterator(output_numpy=True):
    image_np = data['image']
    image_shape = data['image_shape']

    output = model.predict(Tensor(image_np))
    for batch_idx in range(image_np.shape[0]):
        pred_data.append({"boxes": output[0].asnumpy()[batch_idx],
                          "box_scores": output[1].asnumpy()[batch_idx],
                          "img_id": id_iter,
                          "image_shape": image_shape[batch_idx]})
        id_iter += 1
cost_time = int((time.time() - start) * 1000)
print(f'    100% [{total}/{total}] cost {cost_time} ms')

# calculate mAP for the predict data
voc_cls = ['background',
            'aeroplane', 'bicycle', 'bird', 'boat', 'bottle',
            'bus', 'car', 'cat', 'chair', 'cow',
            'diningtable', 'dog', 'horse', 'motorbike', 'person',
            'pottedplant', 'sheep', 'sofa', 'train', 'tvmonitor']
anno_file = create_voc_label(voc_path, voc_cls)
mAP = coco_eval(pred_data, anno_file)
print("\n========================================\n")
print(f"mAP: {mAP}")

Notice: If skipped training process, download the pretrained ckpt file and continue to serving

Click HERE to download the ckpt file and save this file to /etc/tinyms/serving/ssd300/ssd300.ckpt.

Or run the following code to download and store the ckpt file:

ssd300_ckpt_folder = '/etc/tinyms/serving/ssd300'
ssd300_ckpt_path = '/etc/tinyms/serving/ssd300/ssd300.ckpt'

if not os.path.exists(ssd300_ckpt_folder):
    !mkdir -p  /etc/tinyms/serving/ssd300
    !wget -P /etc/tinyms/serving/ssd300 https://ascend-tutorials.obs.cn-north-4.myhuaweicloud.com/ckpt_files/voc/ssd300.ckpt
else:
    print('ssd300 ckpt folder already exists')
    if not os.path.exists(ssd300_ckpt_path):
        !wget -P /etc/tinyms/serving/ssd300 https://ascend-tutorials.obs.cn-north-4.myhuaweicloud.com/ckpt_files/voc/ssd300.ckpt
    else:
        print('ssd300 ckpt file already exists')

ssd300 ckpt folder already exists
--2021-03-19 15:38:53--  https://ascend-tutorials.obs.cn-north-4.myhuaweicloud.com/ckpt_files/voc/ssd300.ckpt
Resolving ascend-tutorials.obs.cn-north-4.myhuaweicloud.com (ascend-tutorials.obs.cn-north-4.myhuaweicloud.com)... 49.4.112.113, 49.4.112.90, 121.36.121.44, ...
Connecting to ascend-tutorials.obs.cn-north-4.myhuaweicloud.com (ascend-tutorials.obs.cn-north-4.myhuaweicloud.com)|49.4.112.113|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 28056511 (27M) [binary/octet-stream]
Saving to: ‘/etc/tinyms/serving/ssd300/ssd300.ckpt’

ssd300.ckpt         100%[===================>]  26.76M  20.7MB/s    in 1.3s

2021-03-19 15:38:55 (20.7 MB/s) - ‘/etc/tinyms/serving/ssd300/ssd300.ckpt’ saved [28056511/28056511]

4. Define servable.json

Run this code to define the servable json file for later use:

servable_json = [{'name': 'ssd300',
                  'description': 'This servable hosts an ssd300 model predicting bounding boxes',
                  'model': {
                      "name": "ssd300",
                      "format": "ckpt",
                      "class_num": 21}}]
os.chdir("/etc/tinyms/serving")
json_data = json.dumps(servable_json, indent=4)

with open('servable.json', 'w') as json_file:
    json_file.write(json_data)

5. Start server

5.1 Introduction

TinyMS Serving is a C/S(client/server) structure. There is a server and client. TinyMS using Flask which is a micro web framework written in python as the C/S communication tool. In order to serve a model, user must start server first. If successfully started, the server will be run in a subprocess and listening to POST requests from 127.0.0.1 port 5000 sent by client and handle the requests using MindSpore backend which will construct the model, run the prediction and send the result back to the client.

5.2 start server

Run the following code block to start the server:

start_server()

Server starts at host 127.0.0.1, port 5000

6. Make predictions

6.1 Upload the pic

A picture is required to be the input. In this tutorial, the SSD300 ckpt requires a picture containing objects of

['background', 'aeroplane', 'bicycle', 'bird', 'boat', 'bottle', 'bus', 'car', 'cat', 'chair', 'cow', 'diningtable', 'dog', 'horse', 'motorbike', 'person', 'pottedplant', 'sheep', 'sofa', 'train', 'tvmonitor']

Click HERE to download the picture which is used in this tutorial. Upload the pic, if using terminal, either scp or wget will do, if running in Jupyter, click Upload button at the top right and select the picture. Save the picture at the root folder, rename to ssd300_test.jpeg(or any name you want).

Or run the following code to download the picture used in this tutorial:

# download the test pic
if not os.path.exists('/root/ssd300_test.jpeg'):
    !wget -P /root/ https://ascend-tutorials.obs.cn-north-4.myhuaweicloud.com/tinyms-test-pics/ssd300_test/ssd300_test.jpeg
else:
    print('ssd300_test.jpeg already exists')

--2021-03-19 15:38:59--  https://ascend-tutorials.obs.cn-north-4.myhuaweicloud.com/tinyms-test-pics/ssd300_test/ssd300_test.jpeg
Resolving ascend-tutorials.obs.cn-north-4.myhuaweicloud.com (ascend-tutorials.obs.cn-north-4.myhuaweicloud.com)... 49.4.112.113, 49.4.112.90, 121.36.121.44, ...
Connecting to ascend-tutorials.obs.cn-north-4.myhuaweicloud.com (ascend-tutorials.obs.cn-north-4.myhuaweicloud.com)|49.4.112.113|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 70412 (69K) [image/jpeg]
Saving to: ‘/root/ssd300_test.jpeg’

ssd300_test.jpeg    100%[===================>]  68.76K   338KB/s    in 0.2s

2021-03-19 15:39:00 (338 KB/s) - ‘/root/ssd300_test.jpeg’ saved [70412/70412]

6.2 List servables

Now, use list_servables function to check what model is servable right now.

list_servables()

[{'description': 'This servable hosts an ssd300 model predicting bounding boxes',
  'model': {'class_num': 21, 'format': 'ckpt', 'name': 'ssd300'},
  'name': 'ssd300'}]

If the output description shows it is an ssd300 model, run the following code to predict the bounding boxes

6.3 Sending request and get the result

Run predict function and get the result, right now only TOP1CLASS strategy is supported for SSD300. Call ImageViewer.draw to draw the bounding boxes

# set image path and output strategy(only TOP1_CLASS)
image_path = "/root/ssd300_test.jpeg"
strategy = "TOP1_CLASS"

labels = ['background',
          'aeroplane', 'bicycle', 'bird', 'boat', 'bottle',
          'bus', 'car', 'cat', 'chair', 'cow',
          'diningtable', 'dog', 'horse', 'motorbike', 'person',
          'pottedplant', 'sheep', 'sofa', 'train', 'tvmonitor']

# predict(image_path, servable_name, dataset_name, strategy)
# ImageViewer(img, title)
# ImageViewer.draw(predict_result, labels)
if server_started() is True:
    res = predict(image_path, 'ssd300', 'voc', strategy)
    img_viewer = ImageViewer(Image.open(image_path))
    img_viewer.draw(res, labels)
else:
    print("Server not started")

Check output

If the input picture is shown with bounding boxes labeled with object classes and score, that means the prediction is successfully performed.

Shutdown server

shutdown()

'Server shutting down...'