tinyms.serving

This module refers to the process of serving pre-trained models so that they can quickly and efficiently process data input by users and obtain results. TinyMS provides a complete set of start server (start_server), check backend (list_servables), check start status (server_started) and shut down the server (shutdown) and other functions based on Flask (https://flask.palletsprojects.com/en/1.1.x/).

Examples

>>> from tinyms.serving import Server, Client
>>>
>>> server = Server()
>>> server.start_server()
Server starts at host 127.0.0.1, port 5000
>>> client = Client()
>>> client.list_servables()
>>> client.predict('example.jpg', 'servable_name', dataset_name='mnist')

class tinyms.serving.Client(host='127.0.0.1', port=5000)

Client is the entrance to connecting to serving server, and also send all requests to server side by HTTP request.

Parameters

• host (str) – Serving server host ip. Default: ‘127.0.0.1’.

• port (int) – Serving server listen port. Default: 5000.

Examples

>>> from tinyms.serving import Client
>>>
>>> client = Client()

list_servables()

List the model that is currently served by the backend server.

A GET request will be sent to the server (127.0.0.1:5000) which will then be routed to 127.0.0.1:5000/servables, and the backend servable information will be returned to the client.

Returns

res_body[‘servables’] (str) will be returned, the backend servable information. Error message will be returned and printed if requests.status_code is not ok. ‘Server not started’ will be returned if server is not started

Examples

>>> # Running the quickstart tutorial, after server started and servable json defined
>>> from tinyms.serving import Client
>>>
>>> client = Client()
>>> client.list_servables()
[{
    'description': 'This servable hosts a lenet5 model predicting numbers',
    'model': {
        'class_num': 10,
        'format': 'ckpt',
        'name': 'lenet5'
    },
    'name': 'lenet5'
}]

predict(img_path, servable_name, dataset_name='mnist', strategy='TOP1_CLASS')

Send the predict request to the backend server, get the return value and do the post process

Predict the input image, and get the result. User must specify the image_path, servable_name, dataset_name and output_strategy to get the predict result.

Parameters

• img_path (str) – path to the image

• servable_name (str) – the name in servable_json, now supports 6 servables: lenet5, resnet50_imagenet2012, resnet50_cifar10, mobilenetv2, ssd300 and cyclegan_cityscape.

• dataset_name (str) – the name of the dataset that is used to train the model, now supports 5 datasets: mnist, imagenet2012, cifar10, voc, cityscape

• strategy (str) – the output strategy, for lenet5, resnet50 and mobilenetv2, select between ‘TOP1_CLASS’ and ‘TOP5_CLASS’, for ssd300, only TOP1_CLASS, for cyclegan_cityscape, select between gray2color and color2gray

Returns

For lenet5, resnet50, mobilenetv2, the output is a string of predict result. For ssd300, the output is a string of bounding boxes coordinates and labels, which can be further processed using ImageViewer function For cyclegan, the output is a numpy of image, which can be transformed to image using Image.fromarray

Examples

>>> # Running the quickstart tutorial, after server started and servable json defined
>>> from tinyms.serving import Client
>>>
>>> client = Client()
>>> print(client.predict('/root/7.png', 'lenet5', 'mnist', 'TOP1_CLASS'))
TOP1: 7, score: 0.99943381547927856445

class tinyms.serving.Server(host='127.0.0.1', port=5000, serving_path='/etc/tinyms/serving/')

Server is the entrance to initialize and shutdown the serving server, and also accept all requests from the client side by calling Flask server.

Parameters

• host (str) – Serving server host ip. Default: ‘127.0.0.1’.

• port (int) – Serving server listen port. Default: 5000.

• serving_path (str, optional) – Set the read path of a service configuration. Default: ‘/etc/tinyms/serving/’.

Examples

>>> from tinyms.serving import Server
>>>
>>> server = Server()

shutdown()

Shutdown the flask server.

Search fot the pid of the process running on port 5000, and kill it. This function will be automatically called when SIGINT, SIGHUP and SIGTERM signals caught.

Returns

A string message of server shutting down or not.

Examples

>>> from tinyms.serving import Server
>>>
>>> server = Server()
>>> server.shutdown()
Server shutting down...

start_server()

Start the flask server in a subprocess.

Catch the signal of CTRL + D to shutdown, otherwise call shutdown() function to shutdown the server, if the ip and port already in use, server won’t start for a second time.

Returns

Start the server in a sub process.

Examples

>>> from tinyms.serving import Server
>>>
>>> server = Server()
>>> server.start_server()
Server starts at host 127.0.0.1, port 5000

tinyms.serving.run_flask(host='127.0.0.1', port=5000)

Start the flask server, only be used to trigger starting the flask server in subprocess.

Directly calling this function is not recommended, please use start_server(). Only Error message will be displayed.

Parameters

• host (str) – the ip address of the flask server. Default: ‘127.0.0.1’.

• port (int) – the port of the server. Default: 5000.

Returns

Server Started

Examples

>>> # In the start_server function
>>> cmd = ['python -c "from tinyms.serving import run_flask; run_flask()"']
>>> server_process = subprocess.Popen(cmd, stdout=subprocess.PIPE, shell=True)