tinyms.layers

Layer module contains pre-defined building blocks or computing units to construct neural networks.

The high-level components (Layers) used to construct the neural network.

class tinyms.layers.Layer(auto_prefix=True, flags=None)

Base class for all neural networks.

A ‘Layer’ could be a single neural network layer, such as conv2d, relu, batch_norm, etc. or a composition of cells to constructing a network.

Note

In general, the autograd algorithm will automatically generate the implementation of the gradient function, but if back-propagation(bprop) method is implemented, the gradient function will be replaced by the bprop. The bprop implementation will receive a Tensor dout containing the gradient of the loss w.r.t. the output, and a Tensor out containing the forward result. The bprop needs to compute the gradient of the loss w.r.t. the inputs, gradient of the loss w.r.t. Parameter variables are not supported currently. The bprop method must contain the self parameter.

Parameters

auto_prefix (bool) – Recursively generate namespaces. Default: True.

Examples

>>> from tinyms import layers, primitives as P
>>>
>>> class MyNet(layers.Layer):
...    def __init__(self):
...        super(MyNet, self).__init__()
...        self.relu = P.ReLU()
...
...    def construct(self, x):
...        return self.relu(x)

add_flags(**flags)

Add customized attributes for cell.

This method is also called when the cell class is instantiated and the class parameter ‘flag’ is set to True.

add_flags_recursive(**flags)

If a cell contains child cells, this method can recursively customize attributes of all cells.

auto_parallel_compile_and_run()

Whether or not to execute compile and run.

Returns

bool, _auto_parallel_compile_and_run value.

property bprop_debug

Get whether cell custom bprop debug is enabled.

cast_inputs(inputs, dst_type)

Cast inputs to specified type.

cast_param(param)

Cast parameter according to auto mix precision level in pynative mode.

This interface is currently used in the case of auto mix precision and usually need not to be used explicitly.

Parameters

param (Parameter) – Parameters, the type of which should be cast.

Returns

Parameter, the input parameter with type automatically cast.

cells()

Returns an iterator over immediate cells.

Returns

Iteration, all the child cells in the cell.

cells_and_names(cells=None, name_prefix='')

Returns an iterator over all cells in the network.

Includes the cell’s name and itself.

Parameters

• cells (str) – Cells to iterate over. Default: None.

• name_prefix (str) – Namespace. Default: ‘’.

Returns

Iteration, all the child cells and corresponding names in the cell.

Examples

>>> n = Net()
>>> names = []
>>> for m in n.cells_and_names():
...     if m[0]:
...         names.append(m[0])

check_names()

Check the names of cell parameters.

compile(*inputs)

Compiles cell.

Parameters

inputs (tuple) – Inputs of the Cell object.

compile_and_run(*inputs)

Compiles and runs cell.

Parameters

inputs (tuple) – Inputs of the Cell object.

Returns

Object, the result of executing.

construct(*inputs, **kwargs)

Defines the computation to be performed. This method must be overridden by all subclasses.

Returns

Tensor, returns the computed result.

exec_checkpoint_graph()

Executes saving checkpoint graph operation.

extend_repr()

Sets the extended representation of the Cell.

To print customized extended information, re-implement this method in your own cells.

generate_scope()

Generate the scope for each cell object in the network.

get_flags()

Get the attributes of cell’s flags.

get_func_graph_proto()

Return graph binary proto.

get_parameters(expand=True)

Returns an iterator over cell parameters.

Yields parameters of this cell. If expand is true, yield parameters of this cell and all subcells.

Parameters

expand (bool) – If true, yields parameters of this cell and all subcells. Otherwise, only yield parameters that are direct members of this cell. Default: True.

Returns

Iteration, all parameters at the cell.

Examples

>>> net = Net()
>>> parameters = []
>>> for item in net.get_parameters():
...     parameters.append(item)

get_scope()

Returns the scope of a cell object in one network.

Returns

String, scope of the cell.

infer_param_pipeline_stage()

Infer pipeline stages of all parameters in the cell.

Note

• If a parameter does not belong to any cell which has been set pipeline_stage, the parameter should use add_pipeline_stage to add it’s pipeline_stage information.

• If a parameter P has been used by two operator in different stages “stageA” and “stageB”, the parameter P should use P.add_pipeline_stage(stageA) and P.add_pipeline_stage(stageB) to add it’s stage information before use infer_param_pipeline_stage.

Returns

The params belong to current stage in pipeline parallel.

Raises

RuntimeError – If there is a parameter does not belong to any stage.

init_parameters_data(auto_parallel_mode=False)

Initialize all parameters and replace the original saved parameters in cell.

Note

trainable_params() and other similar interfaces may return different parameter instance after init_parameters_data, do not save these result.

Parameters

auto_parallel_mode (bool) – If running in auto_parallel_mode.

Returns

Dict[Parameter, Parameter], returns a dict of original parameter and replaced parameter.

insert_child_to_cell(child_name, child_cell)

Adds a child cell to the current cell with a given name.

Parameters

• child_name (str) – Name of the child cell.

• child_cell (Cell) – The child cell to be inserted.

Raises

• KeyError – Child Cell’s name is incorrect or duplicated with the other child name.

• TypeError – Child Cell’s type is incorrect.

insert_param_to_cell(param_name, param, check_name=True)

Adds a parameter to the current cell.

Inserts a parameter with given name to the cell. Please refer to the usage in source code of mindspore.nn.Cell.__setattr__.

Parameters

• param_name (str) – Name of the parameter.

• param (Parameter) – Parameter to be inserted to the cell.

• check_name (bool) – Determines whether the name input is compatible. Default: True.

Raises

• KeyError – If the name of parameter is null or contains dot.

• AttributeError – If user did not call init() first.

• TypeError – If the type of parameter is not Parameter.

load_parameter_slice(params)

Replace parameters with sliced tensors by parallel strategies.

Please refer to the usage in source code of mindspore.common._CellGraphExecutor.compile.

Parameters

params (dict) – The parameters dictionary used for initializing the data graph.

name_cells()

Returns an iterator over all cells in the network.

Include name of the cell and cell itself.

Returns

Dict[String, Cell], all the child cells and corresponding names in the cell.

property param_prefix

Param prefix is the prefix of current cell’s direct child parameter.

property parameter_layout_dict

parameter_layout_dict represents the tensor layout of a parameter, which is inferred by shard strategy and distributed operator information.

parameters_and_names(name_prefix='', expand=True)

Returns an iterator over cell parameters.

Includes the parameter’s name and itself.

Parameters

• name_prefix (str) – Namespace. Default: ‘’.

• expand (bool) – If true, yields parameters of this cell and all subcells. Otherwise, only yield parameters that are direct members of this cell. Default: True.

Returns

Iteration, all the names and corresponding parameters in the cell.

Examples

>>> n = Net()
>>> names = []
>>> for m in n.parameters_and_names():
...     if m[0]:
...         names.append(m[0])

parameters_broadcast_dict(recurse=True)

Gets the parameters broadcast dictionary of this cell.

Parameters

recurse (bool) – Whether contains the parameters of subcells. Default: True.

Returns

OrderedDict, return parameters broadcast dictionary.

parameters_dict(recurse=True)

Gets parameters dictionary.

Gets the parameters dictionary of this cell.

Parameters

recurse (bool) – Whether contains the parameters of subcells. Default: True.

Returns

OrderedDict, return parameters dictionary.

recompute(**kwargs)

Set the cell recomputed. All the primitive in the cell will be set recomputed. If a primitive set recomputed feeds into some backward nodes for computing gradient, rather than storing the intermediate activation computed in forward pass, we will recompute it in backward pass.

Note

• If the computation involves something like randomization or global variable, the equivalence is not guaranteed currently.

• If the recompute api of a primitive in this cell is also called, the recompute mode of this primitive is subject to the recompute api of the primitive.

• The interface can be configured only once. Therefore, when the parent cell is configured, the child cell should not be configured.

• When the memory remains after applying the recompute, configuring ‘mp_comm_recompute=False’ to improve performance if necessary.

• When the memory still not enough after applying the recompute, configuring ‘parallel_optimizer_comm_recompute=True’ to save more memory if necessary. Cells in the same fusion group should has the same parallel_optimizer_comm_recompute configures.

Parameters

• mp_comm_recompute (bool) – Specifies whether the model parallel communication operators in the cell are recomputed in auto parallel or semi auto parallel mode. Default: True.

• parallel_optimizer_comm_recompute (bool) – Specifies whether the communication operator allgathers introduced by optimizer shard are recomputed in auto parallel or semi auto parallel mode. Default: False.

register_backward_hook(fn)

Set the cell backward hook function. Note that this function is only supported in pynative mode.

Note

fn must be defined as the following code. cell_name is the name of registered cell. grad_input is gradient passed to the cell. grad_output is the gradient computed and passed to the next cell or primitive, which may be modified and returned. hook_fn(cell_name, grad_input, grad_output) -> Tensor or None.

Parameters

fn (function) – Specifies the hook function with grad as input.

remove_redundant_parameters()

Remove the redundant parameters.

This interface usually need not to be used explicitly.

set_auto_parallel()

Set the cell to auto parallel mode.

Note

If a cell needs to use the auto parallel or semi auto parallel mode for training, evaluation or prediction, this interface needs to be called by the cell.

set_boost(boost_type)

In order to improve the network performance, configure the network auto enable to accelerate the algorithm in the algorithm library.

If boost_type is not in the algorithm library, Please view the algorithm in the algorithm library through algorithm library.

Note

Some acceleration algorithms may affect the accuracy of the network, please choose carefully.

Parameters

boost_type (str) – accelerate algorithm.

Returns

Cell, the cell itself.

Raises

ValueError – If boost_type is not in the algorithm library.

set_broadcast_flag(mode=True)

Set the cell to data_parallel mode.

The cell can be accessed as an attribute using the given name.

Parameters

mode (bool) – Specifies whether the model is data_parallel. Default: True.

set_comm_fusion(fusion_type, recurse=True)

Set comm_fusion for all the parameters in the Net. Please refer to the description of mindspore.common.parameter.comm_fusion.

Note

The value of attribute will be overwritten when the function is called multiply.

Parameters

• fusion_type (int) – The value of comm_fusion.

• recurse (bool) – Whether sets the trainable parameters of subcells. Default: True.

set_grad(requires_grad=True)

Sets the cell flag for gradient. In pynative mode, this parameter specifies whether the network require gradients. If true, the backward network needed to compute the gradients will be generated when the forward network is executed.

Parameters

requires_grad (bool) – Specifies if the net need to grad, if it is true, the cell will construct backward network in pynative mode. Default: True.

Returns

Cell, the cell itself.

set_parallel_input_with_inputs(*inputs)

Slice inputs tensors by parallel strategies, and set the sliced inputs to _parallel_input_run

Parameters

inputs (tuple) – inputs of construct method.

set_param_fl(push_to_server=False, pull_from_server=False, requires_aggr=True)

Set the way of parameter and server interaction.

Parameters

• push_to_server (bool) – Whether the parameter should be pushed to server. Default: False.

• pull_from_server (bool) – Whether the parameter should be pulled from server. Default: False.

• requires_aggr (bool) – Whether the parameter should be aggregated in the server. Default: True.

set_param_ps(recurse=True, init_in_server=False)

Set whether the trainable parameters are updated by parameter server and whether the trainable parameters are initialized on server.

Note

It only works when a running task is in the parameter server mode.

Parameters

• recurse (bool) – Whether sets the trainable parameters of subcells. Default: True.

• init_in_server (bool) – Whether trainable parameters updated by parameter server are initialized on server. Default: False.

set_train(mode=True)

Sets the cell to training mode.

The cell itself and all children cells will be set to training mode. Layers that have different constructions for training and predicting, such as BatchNorm, will distinguish between the branches by this attribute. If set to true, the training branch will be executed, otherwise another branch.

Parameters

mode (bool) – Specifies whether the model is training. Default: True.

Returns

Cell, the cell itself.

to_float(dst_type)

Add cast on all inputs of cell and child cells to run with certain float type.

If dst_type is mindspore.dtype.float16, all the inputs of Cell including input, Parameter, Tensor as const will be cast to float16. Please refer to the usage in source code of mindspore.train.amp.build_train_network.

Note

Multiple calls will overwrite.

Parameters

dst_type (mindspore.dtype) – Transfer cell to run with dst_type. dst_type can be mindspore.dtype.float16 or mindspore.dtype.float32.

Returns

Cell, the cell itself.

Raises

ValueError – If dst_type is not float32 or float16.

trainable_params(recurse=True)

Returns all trainable parameters.

Returns a list of all trainable parameters.

Parameters

recurse (bool) – Whether contains the trainable parameters of subcells. Default: True.

Returns

List, the list of trainable parameters.

untrainable_params(recurse=True)

Returns all untrainable parameters.

Returns a list of all untrainable parameters.

Parameters

recurse (bool) – Whether contains the untrainable parameters of subcells. Default: True.

Returns

List, the list of untrainable parameters.

update_cell_prefix()

Update the all child cells’ self.param_prefix.

After being invoked, it can get all the cell’s children’s name prefix by ‘_param_prefix’.

update_cell_type(cell_type)

The current cell type is updated when a quantization aware training network is encountered.

After being invoked, it can set the cell type to ‘cell_type’.

update_parameters_name(prefix='', recurse=True)

Updates the names of parameters with given prefix string.

Adds the given prefix to the names of parameters.

Parameters

• prefix (str) – The prefix string. Default: ‘’.

• recurse (bool) – Whether contains the parameters of subcells. Default: True.

Layer module contains pre-defined building blocks or computing units to construct neural networks.

The high-level components (Layers) used to construct the neural network.

class tinyms.layers.SequentialLayer(*args)

Sequential layer container.

A list of Layers will be added to it in the order they are passed in the constructor. Alternatively, an ordered dict of cells can also be passed in.

Parameters

args (Union[list, OrderedDict]) – List of subclass of Layer.

Raises

TypeError – If the type of the argument is not list or OrderedDict.

Inputs:

• input (Tensor) - Tensor with shape according to the first Cell in the sequence.

Outputs:

Tensor, the output Tensor with shape depending on the input and defined sequence of Layers.

Examples

>>> import tinyms as ts
>>> from tinyms.layers import SequentialLayer, Conv2d, ReLU
>>>
>>> seq_layer = SequentialLayer([Conv2d(3, 2, 3, pad_mode='valid', weight_init="ones"), ReLU()])
>>> x = ts.ones([1, 3, 4, 4])
>>> print(seq_layer(x))
[[[[27. 27.]
   [27. 27.]]
  [[27. 27.]
   [27. 27.]]]]

class tinyms.layers.LayerList(*args, **kwargs)

Holds Layers in a list.

LayerList can be used like a regular Python list, support ‘__getitem__’, ‘__setitem__’, ‘__delitem__’, ‘__len__’, ‘__iter__’ and ‘__iadd__’, but layers it contains are properly registered, and will be visible by all Layer methods.

Parameters

args (list, optional) – List of subclass of Layer.

Examples

>>> from tinyms.layers import LayerList, Conv2d, BatchNorm2d, ReLU
>>>
>>> conv = nn.Conv2d(100, 20, 3)
>>> layers = LayerList([BatchNorm2d(20)])
>>> layers.insert(0, Conv2d(100, 20, 3))
>>> layers.append(ReLU())
>>> layers
LayerList<
  (0): Conv2d<input_channels=100, ..., bias_init=None>
  (1): BatchNorm2d<num_features=20, ..., moving_variance=Parameter (name=variance)>
  (2): ReLU<>
  >

class tinyms.layers.TimeDistributed(layer, time_axis, reshape_with_axis=None)

The time distributed layer.

Time distributed is a wrapper which allows to apply a layer to every temporal slice of an input. And the x should be at least 3D. There are two cases in the implementation. When reshape_with_axis provided, the reshape method will be chosen, which is more efficient; otherwise, the method of dividing the inputs along time axis will be used, which is more general. For example, reshape_with_axis could not be provided when deal with Batch Normalization.

Parameters

• layer (Union[Cell, Primitive]) – The Cell or Primitive which will be wrapped.

• time_axis (int) – The axis of time_step.

• reshape_with_axis (int) – The axis which will be reshaped with time_axis. Default: None.

Inputs:

• x (Tensor) - Tensor of shape \((N, T, *)\), where \(*\) means any number of additional dimensions.

Outputs:

Tensor of shape \((N, T, *)\)

Supported Platforms:

Ascend GPU CPU

Raises

TypeError – If layer is not a Cell or Primitive.

Examples

>>> x = Tensor(np.random.random([32, 10, 3]), mindspore.float32)
>>> dense = nn.Dense(3, 6)
>>> net = nn.TimeDistributed(dense, time_axis=1, reshape_with_axis=0)
>>> output = net(x)
>>> print(output.shape)
(32, 10, 6)

class tinyms.layers.ForwardValueAndGrad(network, weights=None, get_all=False, get_by_list=False, sens_param=False)

Network training package class.

Including the network and a gradient function. The resulting Cell is trained with input ‘*inputs’. The backward graph will be created in the gradient function to calculating gradient.

Parameters

• network (Cell) – The training network.

• weights (ParameterTuple) – The parameters of the training network that need to calculate the gradient.

• get_all (bool) – If True, get all the gradients with respect to inputs. Default: False.

• get_by_list (bool) – If True, get all the gradients with respect to Parameter variables. If get_all and get_by_list are both False, get the gradient with respect to first input. If get_all and get_by_list are both True, get the gradients with respect to inputs and Parameter variables at the same time in the form of ((gradients with respect to inputs), (gradients with respect to parameters)). Default: False.

• sens_param (bool) – Whether to append sensitivity (gradient with respect to output) as input. If sens_param is False, a ‘ones_like(outputs)’ sensitivity will be attached automatically. Default: False. If the sens_param is True, a sensitivity (gradient with respect to output) needs to be transferred through the input parameter.

Inputs:

• (*inputs) (Tuple(Tensor…)) - Tuple of inputs with shape \((N, \ldots)\).

• (sens) - A sensitivity (gradient with respect to output) as the input of backpropagation. If network has single output, the sens is a tensor. If network has multiple outputs, the sens is the tuple(tensor).

Outputs:

• forward value - The result of network forward running.

• gradients (tuple(tensor)) - The gradients of network parameters and inputs.

Supported Platforms:

Ascend GPU CPU

Examples

>>> class Net(nn.Cell):
...    def __init__(self):
...        super(Net, self).__init__()
...        self.weight = Parameter(Tensor(np.ones([2, 2]).astype(np.float32)), name="weight")
...        self.matmul = P.MatMul()
...
...    def construct(self, x):
...        out = self.matmul(x, self.weight)
...        return out
...
>>> net = Net()
>>> criterion = nn.SoftmaxCrossEntropyWithLogits()
>>> net_with_criterion = nn.WithLossCell(net, criterion)
>>> weight = ParameterTuple(net.trainable_params())
>>> train_network = nn.ForwardValueAndGrad(net_with_criterion, weights=weight, get_all=True, get_by_list=True)
>>> inputs = Tensor(np.ones([1, 2]).astype(np.float32))
>>> labels = Tensor(np.zeros([1, 2]).astype(np.float32))
>>> result = train_network(inputs, labels)
>>> print(result)
(Tensor(shape=[1], dtype=Float32, value=[0.00000000e+00]), ((Tensor(shape=[1, 2], dtype=Float32, value=
[[1.00000000e+00, 1.00000000e+00]]), Tensor(shape=[1, 2], dtype=Float32, value=
[[0.00000000e+00, 0.00000000e+00]])), (Tensor(shape=[2, 2], dtype=Float32, value=
[[5.00000000e-01, 5.00000000e-01],
 [5.00000000e-01, 5.00000000e-01]]),)))

class tinyms.layers.TrainOneStepCell(network, optimizer, sens=1.0)

Network training package class.

Wraps the network with an optimizer. The resulting Cell is trained with input ‘*inputs’. The backward graph will be created in the construct function to update the parameter. Different parallel modes are available for training.

Parameters

• network (Cell) – The training network. The network only supports single output.

• optimizer (Union[Cell]) – Optimizer for updating the weights.

• sens (numbers.Number) – The scaling number to be filled as the input of backpropagation. Default value is 1.0.

Inputs:

• (*inputs) (Tuple(Tensor)) - Tuple of input tensors with shape \((N, \ldots)\).

Outputs:

Tensor, a tensor means the loss value, the shape of which is usually \(()\).

Raises

TypeError – If sens is not a number.

Supported Platforms:

Ascend GPU CPU

Examples

>>> net = Net()
>>> loss_fn = nn.SoftmaxCrossEntropyWithLogits()
>>> optim = nn.Momentum(net.trainable_params(), learning_rate=0.1, momentum=0.9)
>>> #1) Using the WithLossCell existing provide
>>> loss_net = nn.WithLossCell(net, loss_fn)
>>> train_net = nn.TrainOneStepCell(loss_net, optim)
>>>
>>> #2) Using user-defined WithLossCell
>>> class MyWithLossCell(Cell):
...    def __init__(self, backbone, loss_fn):
...        super(MyWithLossCell, self).__init__(auto_prefix=False)
...        self._backbone = backbone
...        self._loss_fn = loss_fn
...
...    def construct(self, x, y, label):
...        out = self._backbone(x, y)
...        return self._loss_fn(out, label)
...
...    @property
...    def backbone_network(self):
...        return self._backbone
...
>>> loss_net = MyWithLossCell(net, loss_fn)
>>> train_net = nn.TrainOneStepCell(loss_net, optim)

class tinyms.layers.WithLossCell(backbone, loss_fn)

Cell with loss function.

Wraps the network with loss function. This Cell accepts data and label as inputs and the computed loss will be returned.

Parameters

• backbone (Cell) – The target network to wrap.

• loss_fn (Cell) – The loss function used to compute loss.

Inputs:

• data (Tensor) - Tensor of shape \((N, \ldots)\).

• label (Tensor) - Tensor of shape \((N, \ldots)\).

Outputs:

Tensor, a tensor means the loss value, the shape of which is usually \(()\).

Raises

TypeError – If dtype of data or label is neither float16 nor float32.

Supported Platforms:

Ascend GPU CPU

Examples

>>> net = Net()
>>> loss_fn = nn.SoftmaxCrossEntropyWithLogits(sparse=False)
>>> net_with_criterion = nn.WithLossCell(net, loss_fn)
>>>
>>> batch_size = 2
>>> data = Tensor(np.ones([batch_size, 1, 32, 32]).astype(np.float32) * 0.01)
>>> label = Tensor(np.ones([batch_size, 10]).astype(np.float32))
>>>
>>> output_data = net_with_criterion(data, label)

property backbone_network

Get the backbone network.

Returns

Cell, the backbone network.

class tinyms.layers.WithGradCell(network, loss_fn=None, sens=None)

Cell that returns the gradients.

Wraps the network with backward cell to compute gradients. A network with a loss function is necessary as argument. If loss function in None, the network must be a wrapper of network and loss function. This Cell accepts ‘*inputs’ as inputs and returns gradients for each trainable parameter.

Note

Run in PyNative mode.

Parameters

• network (Cell) – The target network to wrap. The network only supports single output.

• loss_fn (Cell) – Primitive loss function used to compute gradients. Default: None.

• sens (Union[None, Tensor, Scalar, Tuple ...]) – The sensitive for backpropagation, the type and shape must be same as the network output. If None, we will fill one to a same type shape of output value. Default: None.

Inputs:

• (*inputs) (Tuple(Tensor)) - Tuple of input tensors with shape \((N, \ldots)\).

Outputs:

list, a list of Tensors with identical shapes as trainable weights.

Raises

TypeError – If sens is not one of None, Tensor, Scalar or Tuple.

Supported Platforms:

Ascend GPU CPU

Examples

>>> # For a defined network Net without loss function
>>> net = Net()
>>> loss_fn = nn.SoftmaxCrossEntropyWithLogits()
>>> grad_net = nn.WithGradCell(net, loss_fn)
>>>
>>> # For a network wrapped with loss function
>>> net = Net()
>>> net_with_criterion = nn.WithLossCell(net, loss_fn)
>>> grad_net = nn.WithGradCell(net_with_criterion)

class tinyms.layers.PipelineCell(network, micro_size)

Wrap the network with Micro Batch.

Note

micro_size must be greater or equal to pipeline stages.

Parameters

• network (Cell) – The target network to wrap.

• micro_size (int) – MicroBatch size.

Examples

>>> net = Net()
>>> net = PipelineCell(net, 4)

class tinyms.layers.WithEvalCell(network, loss_fn, add_cast_fp32=False)

Cell that returns loss, output and label for evaluation.

This Cell accepts a network and loss function as arguments and computes loss for model. It returns loss, output and label to calculate the metrics.

Parameters

• network (Cell) – The network Cell.

• loss_fn (Cell) – The loss Cell.

• add_cast_fp32 (bool) – Adjust the data type to float32.

Inputs:

• data (Tensor) - Tensor of shape \((N, \ldots)\).

• label (Tensor) - Tensor of shape \((N, \ldots)\).

Outputs:

Tuple, containing a scalar loss Tensor, a network output Tensor of shape \((N, \ldots)\) and a label Tensor of shape \((N, \ldots)\).

Raises

TypeError – If add_cast_fp32 is not a bool.

Supported Platforms:

Ascend GPU CPU

Examples

>>> # For a defined network Net without loss function
>>> net = Net()
>>> loss_fn = nn.SoftmaxCrossEntropyWithLogits()
>>> eval_net = nn.WithEvalCell(net, loss_fn)

class tinyms.layers.GetNextSingleOp(dataset_types, dataset_shapes, queue_name)

Cell to run for getting the next operation.

Parameters

• dataset_types (list[mindspore.dtype]) – The types of dataset.

• dataset_shapes (list[tuple[int]]) – The shapes of dataset.

• queue_name (str) – Queue name to fetch the data.

For detailed information, refer to ops.operations.GetNext.

Inputs:

No inputs.

Outputs:

tuple[Tensor], the data get from Dataset.

Supported Platforms:

Ascend GPU

Examples

>>> train_dataset = create_custom_dataset()
>>> dataset_helper = mindspore.DatasetHelper(train_dataset, dataset_sink_mode=True)
>>> dataset = dataset_helper.iter.dataset
>>> dataset_types, dataset_shapes = dataset_helper.types_shapes()
>>> queue_name = dataset.__transfer_dataset__.queue_name
>>> get_next_single_op_net = nn.GetNextSingleOp(dataset_types, dataset_shapes, queue_name)
>>> data, label = get_next_single_op_net()
>>> relu = P.ReLU()
>>> result = relu(data).asnumpy()
>>> print(result.shape)
(32, 1, 32, 32)

class tinyms.layers.TrainOneStepWithLossScaleCell(network, optimizer, scale_sense)

Network training with loss scaling.

This is a training step with loss scaling. It takes a network, an optimizer and possibly a scale update Cell as args. The loss scale value can be updated in both host side or device side. The TrainOneStepWithLossScaleCell will be compiled to be graph which takes *inputs as input data. The Tensor type of scale_sense is acting as loss scaling value. If you want to update it on host side, the value must be provided. If the Tensor type of scale_sense is not given, the loss scale update logic must be provied by Cell type of scale_sense.

Parameters

• network (Cell) – The training network. The network only supports single output.

• optimizer (Cell) – Optimizer for updating the weights.

• scale_sense (Union[Tensor, Cell]) – If this value is Cell type, the loss scaling update logic cell.If this value is Tensor type, Tensor with shape \(()\) or \((1,)\).

Inputs:

• (*inputs) (Tuple(Tensor)) - Tuple of input tensors with shape \((N, \ldots)\).

Outputs:

Tuple of 3 Tensor, the loss, overflow flag and current loss scaling value.

• loss (Tensor) - Tensor with shape \(()\).

• overflow (Tensor) - Tensor with shape \(()\), type is bool.

• loss scaling value (Tensor) - Tensor with shape \(()\)

Raises

• TypeError – If scale_sense is neither Cell nor Tensor.

• ValueError – If shape of scale_sense is neither (1,) nor ().

Supported Platforms:

Ascend GPU

Examples

>>> import numpy as np
>>> from mindspore import Tensor, Parameter, nn, ops
>>> from mindspore import dtype as mstype
>>>
>>> class Net(nn.Cell):
...     def __init__(self, in_features, out_features):
...         super(Net, self).__init__()
...         self.weight = Parameter(Tensor(np.ones([in_features, out_features]).astype(np.float32)),
...                                 name='weight')
...         self.matmul = ops.MatMul()
...
...     def construct(self, x):
...         output = self.matmul(x, self.weight)
...         return output
...
>>> size, in_features, out_features = 16, 16, 10
>>> #1) when the type of scale_sense is Cell:
>>> net = Net(in_features, out_features)
>>> loss = nn.MSELoss()
>>> optimizer = nn.Momentum(net.trainable_params(), learning_rate=0.1, momentum=0.9)
>>> net_with_loss = nn.WithLossCell(net, loss)
>>> manager = nn.DynamicLossScaleUpdateCell(loss_scale_value=2**12, scale_factor=2, scale_window=1000)
>>> train_network = nn.TrainOneStepWithLossScaleCell(net_with_loss, optimizer, scale_sense=manager)
>>> input = Tensor(np.ones([out_features, in_features]), mindspore.float32)
>>> labels = Tensor(np.ones([out_features,]), mindspore.float32)
>>> output = train_network(input, labels)
>>>
>>> #2) when the type of scale_sense is Tensor:
>>> net = Net(in_features, out_features)
>>> loss = nn.MSELoss()
>>> optimizer = nn.Momentum(net.trainable_params(), learning_rate=0.1, momentum=0.9)
>>> net_with_loss = nn.WithLossCell(net, loss)
>>> inputs = Tensor(np.ones([size, in_features]).astype(np.float32))
>>> label = Tensor(np.zeros([size, out_features]).astype(np.float32))
>>> scaling_sens = Tensor(np.full((1), np.finfo(np.float32).max), dtype=mstype.float32)
>>> train_network = nn.TrainOneStepWithLossScaleCell(net_with_loss, optimizer, scale_sense=scaling_sens)
>>> output = train_network(inputs, label)

get_overflow_status(status, compute_output)

Get floating-point overflow status.

Get overflow results after executing the target process for overflow detection.

Inputs:

• status (object) - A status instance used to detect the overflow.

• compute_output - Overflow detection should be performed on a certain computation. Set compute_output as the output of the computation, to ensure overflow status is acquired before executing the computation.

Outputs:

bool, whether the overflow occurs or not.

process_loss_scale(overflow)

Calculate loss scale according to the overflow.

Inputs:

• overflow (bool) - Whether the overflow occurs or not.

Outputs:

bool, overflow value.

set_sense_scale(sens)

If the user has set the sens in the training process and wants to reassign the value, he can call this function again to make modification, and sens needs to be of type Tensor.

Inputs:

• sens (Tensor) - The new sense whose shape and type are the same with original scale_sense.

start_overflow_check(pre_cond, compute_input)

Start floating-point overflow detection. Create and clear the overflow detection state.

Specify the argument ‘pre_cond’ and ‘compute_input’ to make sure overflow status is cleared at the right time. Taking this situation as an example, we need to execute state clearing after loss calculation and then detect overflow in the process of gradient calculation. In this case, pre_cond should be the output of the loss function, and compute_input should be the input of gradients-computing function.

Inputs:

• pre_cond (Tensor) - A precondition for starting overflow detection. It determines the executing order of overflow state clearing and prior processions. It makes sure that the function ‘start_overflow’ clears status after finishing the process of precondition.

• compute_input (object) - The input of subsequent process. Overflow detection should be performed on a certain computation. Set compute_input as the input of the computation, to ensure overflow status is cleared before executing the computation.

Outputs:

Tuple[object, object], the first value is False for GPU backend, while it is a instance of NPUAllocFloatStatus for other backend. The status is used to detect overflow during overflow detection. The second value is the same as the input of compute_input, but contains some information about the execution order.

class tinyms.layers.DistributedGradReducer(parameters, mean=True, degree=None, fusion_type=1, group='hccl_world_group')

A distributed optimizer.

Constructs a gradient reducer Cell, which applies communication and average operations on single-process gradient values.

Parameters

• parameters (list) – the parameters to be updated.

• mean (bool) – When mean is true, the mean coefficient (degree) would apply on gradients. Default: False.

• degree (int) – The mean coefficient. Usually it equals to device number. Default: None.

• fusion_type (int) – The type of all reduce fusion. Default: 1.

Raises

ValueError – If degree is not a int or less than 0.

Supported Platforms:

Ascend GPU

Examples

>>> # This example should be run with multiple processes.
>>> # Please refer to the tutorial > Distributed Training on mindspore.cn.
>>> import numpy as np
>>> from mindspore.communication import init
>>> from mindspore import ops
>>> from mindspore import context
>>> from mindspore.context import ParallelMode
>>> from mindspore import Parameter, Tensor
>>> from mindspore import nn
>>>
>>> context.set_context(mode=context.GRAPH_MODE)
>>> init()
>>> context.reset_auto_parallel_context()
>>> context.set_auto_parallel_context(parallel_mode=ParallelMode.DATA_PARALLEL)
>>>
>>> class TrainingWrapper(nn.Cell):
...     def __init__(self, network, optimizer, sens=1.0):
...         super(TrainingWrapper, self).__init__(auto_prefix=False)
...         self.network = network
...         self.network.add_flags(defer_inline=True)
...         self.weights = optimizer.parameters
...         self.optimizer = optimizer
...         self.grad = ops.GradOperation(get_by_list=True, sens_param=True)
...         self.sens = sens
...         self.reducer_flag = False
...         self.grad_reducer = None
...         self.parallel_mode = context.get_auto_parallel_context("parallel_mode")
...         if self.parallel_mode in [ParallelMode.DATA_PARALLEL, ParallelMode.HYBRID_PARALLEL]:
...             self.reducer_flag = True
...         if self.reducer_flag:
...             mean = context.get_auto_parallel_context("gradients_mean")
...             degree = context.get_auto_parallel_context("device_num")
...             self.grad_reducer = nn.DistributedGradReducer(optimizer.parameters, mean, degree)
...
...     def construct(self, *args):
...         weights = self.weights
...         loss = self.network(*args)
...         sens = ops.Fill()(ops.DType()(loss), ops.Shape()(loss), self.sens)
...         grads = self.grad(self.network, weights)(*args, sens)
...         if self.reducer_flag:
...             # apply grad reducer on grads
...             grads = self.grad_reducer(grads)
...         return ops.Depend(loss, self.optimizer(grads))
>>>
>>> class Net(nn.Cell):
...     def __init__(self, in_features, out_features):
...         super(Net, self).__init__()
...         self.weight = Parameter(Tensor(np.ones([in_features, out_features]).astype(np.float32)),
...                                 name='weight')
...         self.matmul = ops.MatMul()
...
...     def construct(self, x):
...         output = self.matmul(x, self.weight)
...         return output
>>>
>>> size, in_features, out_features = 16, 16, 10
>>> network = Net(in_features, out_features)
>>> loss = nn.MSELoss()
>>> net_with_loss = nn.WithLossCell(network, loss)
>>> optimizer = nn.Momentum(net_with_loss.trainable_params(), learning_rate=0.1, momentum=0.9)
>>> train_cell = TrainingWrapper(net_with_loss, optimizer)
>>> inputs = Tensor(np.ones([size, in_features]).astype(np.float32))
>>> label = Tensor(np.zeros([size, out_features]).astype(np.float32))
>>> grads = train_cell(inputs, label)
>>> print(grads)
256.0

construct(grads)

Under certain circumstances, the data precision of grads could be mixed with float16 and float32. Thus, the result of AllReduce is unreliable. To solve the problem, grads must be cast to float32 before AllReduce, and cast back after the operation.

Parameters

grads (Union[Tensor, tuple[Tensor]]) – The gradient tensor or tuple before operation.

Returns

new_grads (Union[Tensor, tuple[Tensor]]), the gradient tensor or tuple after operation.

class tinyms.layers.ParameterUpdate(param)

Cell that updates parameter.

With this Cell, one can manually update param with the input Tensor.

Parameters

param (Parameter) – The parameter to be updated manually.

Inputs:

• x (Tensor) - A tensor whose shape and type are the same with param.

Outputs:

Tensor, the input x.

Raises

KeyError – If parameter with the specified name does not exist.

Supported Platforms:

Ascend GPU CPU

Examples

>>> network = nn.Dense(3, 4)
>>> param = network.parameters_dict()['weight']
>>> update = nn.ParameterUpdate(param)
>>> update.phase = "update_param"
>>> weight = Tensor(np.arange(12).reshape((4, 3)), mindspore.float32)
>>> output = update(weight)

class tinyms.layers.DynamicLossScaleUpdateCell(loss_scale_value, scale_factor, scale_window)

Dynamic Loss scale update cell.

For loss scaling training, the initial loss scaling value will be set to be loss_scale_value. In each training step, the loss scaling value will be updated by loss scaling value/scale_factor when there is an overflow. And it will be increased by loss scaling value * scale_factor if there is no overflow for a continuous scale_window steps. This cell is used for Graph mode training in which all logic will be executed on device side(Another training mode is normal(non-sink) mode in which some logic will be executed on host).

Parameters

• loss_scale_value (float) – Initializes loss scale.

• scale_factor (int) – Coefficient of increase and decrease.

• scale_window (int) – Maximum continuous training steps that do not have overflow.

Inputs:

• loss_scale (Tensor) - The loss scale value during training with shape \(()\).

• overflow (bool) - Whether the overflow occurs or not.

Outputs:

bool, the input overflow.

Raises

TypeError – If dtype of inputs or label is neither float16 nor float32.

Supported Platforms:

Ascend GPU

Examples

>>> import numpy as np
>>> from mindspore import Tensor, Parameter, nn
>>> import mindspore.ops as ops
>>>
>>> class Net(nn.Cell):
...     def __init__(self, in_features, out_features):
...         super(Net, self).__init__()
...         self.weight = Parameter(Tensor(np.ones([in_features, out_features]).astype(np.float32)),
...                                 name='weight')
...         self.matmul = ops.MatMul()
...
...     def construct(self, x):
...         output = self.matmul(x, self.weight)
...         return output
...
>>> in_features, out_features = 16, 10
>>> net = Net(in_features, out_features)
>>> loss = nn.MSELoss()
>>> optimizer = nn.Momentum(net.trainable_params(), learning_rate=0.1, momentum=0.9)
>>> net_with_loss = nn.WithLossCell(net, loss)
>>> manager = nn.DynamicLossScaleUpdateCell(loss_scale_value=2**12, scale_factor=2, scale_window=1000)
>>> train_network = nn.TrainOneStepWithLossScaleCell(net_with_loss, optimizer, scale_sense=manager)
>>> input = Tensor(np.ones([out_features, in_features]), mindspore.float32)
>>> labels = Tensor(np.ones([out_features,]), mindspore.float32)
>>> output = train_network(input, labels)

get_loss_scale()

Get Loss Scale value.

class tinyms.layers.FixedLossScaleUpdateCell(loss_scale_value)

Static scale update cell, the loss scaling value will not be updated.

For usage, refer to DynamicLossScaleUpdateCell.

Parameters

loss_scale_value (float) – Initializes loss scale.

Inputs:

• loss_scale (Tensor) - The loss scale value during training with shape \(()\), that will be ignored.

• overflow (bool) - Whether the overflow occurs or not.

Outputs:

bool, the input overflow.

Supported Platforms:

Ascend GPU

Examples

>>> import numpy as np
>>> from mindspore import Tensor, Parameter, nn, ops
>>>
>>> class Net(nn.Cell):
...     def __init__(self, in_features, out_features):
...         super(Net, self).__init__()
...         self.weight = Parameter(Tensor(np.ones([in_features, out_features]).astype(np.float32)),
...                                 name='weight')
...         self.matmul = ops.MatMul()
...
...     def construct(self, x):
...         output = self.matmul(x, self.weight)
...         return output
...
>>> in_features, out_features = 16, 10
>>> net = Net(in_features, out_features)
>>> loss = nn.MSELoss()
>>> optimizer = nn.Momentum(net.trainable_params(), learning_rate=0.1, momentum=0.9)
>>> net_with_loss = nn.WithLossCell(net, loss)
>>> manager = nn.FixedLossScaleUpdateCell(loss_scale_value=2**12)
>>> train_network = nn.TrainOneStepWithLossScaleCell(net_with_loss, optimizer, scale_sense=manager)
>>> input = Tensor(np.ones([out_features, in_features]), mindspore.float32)
>>> labels = Tensor(np.ones([out_features,]), mindspore.float32)
>>> output = train_network(input, labels)

get_loss_scale()

Get Loss Scale value.

class tinyms.layers.VirtualDatasetCellTriple(backbone)

Wrap the network with virtual dataset to convert data parallel layout to model parallel layout.

VirtualDatasetCellTriple is a virtual Primitive, it does not exist in the final executing graph. Inputs and outputs of VirtualDatasetCellTriple are distributed in data parallel pattern, tensor redistribution Primitives is inserted dynamically during the graph compile process.

Note

Only used in semi auto parallel and auto parallel mode. There are three inputs, as contrary to two inputs in _VirtualDatasetCell.

Parameters

backbone (Cell) – The target network to wrap.

Examples

>>> net = Net()
>>> net = VirtualDatasetCellTriple(net)

class tinyms.layers.Softmax(axis=-1)

Softmax activation function.

Applies the Softmax function to an n-dimensional input Tensor.

The input is a Tensor of logits transformed with exponential function and then normalized to lie in range [0, 1] and sum up to 1.

Softmax is defined as:

\[\text{softmax}(x_{i}) = \frac{\exp(x_i)}{\sum_{j=0}^{n-1}\exp(x_j)},\]

where \(x_{i}\) is the \(i\)-th slice in the given dimension of the input Tensor.

Parameters

axis (Union[int, tuple[int]]) – The axis to apply Softmax operation, -1 means the last dimension. Default: -1.

Inputs:

• x (Tensor) - The input of Softmax with data type of float16 or float32.

Outputs:

Tensor, which has the same type and shape as x with values in the range[0,1].

Raises

• TypeError – If axis is neither an int nor a tuple.

• TypeError – If dtype of x is neither float16 nor float32.

• ValueError – If axis is a tuple whose length is less than 1.

• ValueError – If axis is a tuple whose elements are not all in range [-len(x), len(x)).

Supported Platforms:

Ascend GPU CPU

Examples

>>> x = Tensor(np.array([-1, -2, 0, 2, 1]), mindspore.float16)
>>> softmax = nn.Softmax()
>>> output = softmax(x)
>>> print(output)
[0.03168 0.01166 0.0861  0.636   0.2341 ]

class tinyms.layers.LogSoftmax(axis=-1)

LogSoftmax activation function.

Applies the LogSoftmax function to n-dimensional input tensor.

The input is transformed by the Softmax function and then by the log function to lie in range[-inf,0).

Logsoftmax is defined as:

\[\text{logsoftmax}(x_i) = \log \left(\frac{\exp(x_i)}{\sum_{j=0}^{n-1} \exp(x_j)}\right),\]

where \(x_{i}\) is the \(i\)-th slice in the given dimension of the input Tensor.

Parameters

axis (int) – The axis to apply LogSoftmax operation, -1 means the last dimension. Default: -1.

Inputs:

• x (Tensor) - The input of LogSoftmax, with float16 or float32 data type.

Outputs:

Tensor, which has the same type and shape as the input as x with values in the range[-inf,0).

Raises

• TypeError – If axis is not an int.

• TypeError – If dtype of x is neither float16 nor float32.

• ValueError – If axis is not in range [-len(x), len(x)).

Supported Platforms:

Ascend GPU CPU

Examples

>>> x = Tensor(np.array([[-1.0, 4.0, -8.0], [2.0, -5.0, 9.0]]), mindspore.float32)
>>> log_softmax = nn.LogSoftmax()
>>> output = log_softmax(x)
>>> print(output)
[[-5.00672150e+00 -6.72150636e-03 -1.20067215e+01]
 [-7.00091219e+00 -1.40009127e+01 -9.12250078e-04]]

class tinyms.layers.ReLU

Rectified Linear Unit activation function.

Applies the rectified linear unit function element-wise.

\[\text{ReLU}(x) = (x)^+ = \max(0, x),\]

It returns element-wise \(\max(0, x)\), specially, the neurons with the negative output will be suppressed and the active neurons will stay the same.

The picture about ReLU looks like this ReLU.

Inputs:

• x (Tensor) - The input of ReLU. The data type is Number. The shape is \((N,*)\) where \(*\) means, any number of additional dimensions.

Outputs:

Tensor, with the same type and shape as the x.

Raises

TypeError – If dtype of x is not a number.

Supported Platforms:

Ascend GPU CPU

Examples

>>> x = Tensor(np.array([-1, 2, -3, 2, -1]), mindspore.float16)
>>> relu = nn.ReLU()
>>> output = relu(x)
>>> print(output)
[0. 2. 0. 2. 0.]

class tinyms.layers.ReLU6

Compute ReLU6 activation function.

ReLU6 is similar to ReLU with a upper limit of 6, which if the inputs are greater than 6, the outputs will be suppressed to 6. It computes element-wise as

\[\min(\max(0, x), 6).\]

The input is a Tensor of any valid shape.

Inputs:

• x (Tensor) - The input of ReLU6 with data type of float16 or float32. The shape is \((N,*)\) where \(*\) means, any number of additional dimensions.

Outputs:

Tensor, which has the same type as x.

Raises

TypeError – If dtype of x is neither float16 nor float32.

Supported Platforms:

Ascend GPU CPU

Examples

>>> x = Tensor(np.array([-1, -2, 0, 2, 1]), mindspore.float16)
>>> relu6 = nn.ReLU6()
>>> output = relu6(x)
>>> print(output)
[0. 0. 0. 2. 1.]

class tinyms.layers.Tanh

Tanh activation function.

Applies the Tanh function element-wise, returns a new tensor with the hyperbolic tangent of the elements of input, The input is a Tensor with any valid shape.

Tanh function is defined as:

\[tanh(x_i) = \frac{\exp(x_i) - \exp(-x_i)}{\exp(x_i) + \exp(-x_i)} = \frac{\exp(2x_i) - 1}{\exp(2x_i) + 1},\]

where \(x_i\) is an element of the input Tensor.

Inputs:

• x (Tensor) - The input of Tanh with data type of float16 or float32. The shape is \((N,*)\) where \(*\) means, any number of additional dimensions.

Outputs:

Tensor, with the same type and shape as the x.

Raises

TypeError – If dtype of x is neither float16 nor float32.

Supported Platforms:

Ascend GPU CPU

Examples

>>> x = Tensor(np.array([1, 2, 3, 2, 1]), mindspore.float16)
>>> tanh = nn.Tanh()
>>> output = tanh(x)
>>> print(output)
[0.7617 0.964  0.995  0.964  0.7617]

class tinyms.layers.GELU

Gaussian error linear unit activation function.

Applies GELU function to each element of the input. The input is a Tensor with any valid shape.

GELU is defined as:

\[GELU(x_i) = x_i*P(X < x_i),\]

where \(P\) is the cumulative distribution function of standard Gaussian distribution and \(x_i\) is the element of the input.

The picture about GELU looks like this GELU.

Inputs:

• x (Tensor) - The input of GELU with data type of float16 or float32. The shape is \((N,*)\) where \(*\) means, any number of additional dimensions.

Outputs:

Tensor, with the same type and shape as the x.

Raises

TypeError – If dtype of x is neither float16 nor float32.

Supported Platforms:

Ascend GPU CPU

Examples

>>> x = Tensor(np.array([[-1.0, 4.0, -8.0], [2.0, -5.0, 9.0]]), mindspore.float32)
>>> gelu = nn.GELU()
>>> output = gelu(x)
>>> print(output)
[[-1.5880802e-01  3.9999299e+00 -3.1077917e-21]
 [ 1.9545976e+00 -2.2918017e-07  9.0000000e+00]]

class tinyms.layers.FastGelu

Fast Gaussian error linear unit activation function.

Applies FastGelu function to each element of the input. The input is a Tensor with any valid shape.

FastGelu is defined as:

\[FastGelu(x_i) = \frac {x_i} {1 + \exp(-1.702 * \left| x_i \right|)} * \exp(0.851 * (x_i - \left| x_i \right|))\]

where \(x_i\) is the element of the input.

Inputs:

• x (Tensor) - The input of FastGelu with data type of float16 or float32. The shape is \((N,*)\) where \(*\) means, any number of additional dimensions.

Outputs:

Tensor, with the same type and shape as the x.

Raises

TypeError – If dtype of x is neither float16 nor float32.

Supported Platforms:

Ascend

Examples

>>> x = Tensor(np.array([[-1.0, 4.0, -8.0], [2.0, -5.0, 9.0]]), mindspore.float32)
>>> fast_gelu = nn.FastGelu()
>>> output = fast_gelu(x)
>>> print(output)
[[-1.5418735e-01  3.9921875e+00 -9.7473649e-06]
 [ 1.9375000e+00 -1.0052517e-03  8.9824219e+00]]

class tinyms.layers.Sigmoid

Sigmoid activation function.

Applies sigmoid-type activation element-wise.

Sigmoid function is defined as:

\[\text{sigmoid}(x_i) = \frac{1}{1 + \exp(-x_i)},\]

where \(x_i\) is the element of the input.

The picture about Sigmoid looks like this Sigmoid.

Inputs:

• x (Tensor) - The input of Sigmoid with data type of float16 or float32. The shape is \((N,*)\) where \(*\) means, any number of additional dimensions.

Outputs:

Tensor, with the same type and shape as the x.

Raises

TypeError – If dtype of x is neither float16 nor float32.

Supported Platforms:

Ascend GPU CPU

Examples

>>> x = Tensor(np.array([-1, -2, 0, 2, 1]), mindspore.float16)
>>> sigmoid = nn.Sigmoid()
>>> output = sigmoid(x)
>>> print(output)
[0.2688  0.11914 0.5     0.881   0.7305 ]

class tinyms.layers.PReLU(channel=1, w=0.25)

PReLU activation function.

Applies the PReLU function element-wise.

PReLU is defined as:

\[prelu(x_i)= \max(0, x_i) + w * \min(0, x_i),\]

where \(x_i\) is an element of an channel of the input.

Here \(w\) is a learnable parameter with a default initial value 0.25. Parameter \(w\) has dimensionality of the argument channel. If called without argument channel, a single parameter \(w\) will be shared across all channels.

The picture about PReLU looks like this PReLU.

Parameters

• channel (int) – The elements number of parameter. It could be an int, and the value is 1 or the channels number of input tensor x. Default: 1.

• w (Union[float, list, Tensor]) – The initial value of parameter. It could be a float, a float list or a tensor has the same dtype as the input tensor x. Default: 0.25.

Inputs:

• x (Tensor) - The input of PReLU with data type of float16 or float32. The shape is \((N,*)\) where \(*\) means, any number of additional dimensions.

Outputs:

Tensor, with the same dtype and shape as the x.

Raises

• TypeError – If channel is not an int.

• TypeError – If w is not one of a float, a float list, a float Tensor.

• TypeError – If dtype of x is neither float16 nor float32.

• ValueError – If the x is a 0-D or 1-D Tensor on Ascend.

• ValueError – If channel is less than 1.

Supported Platforms:

Ascend GPU

Examples

>>> x = Tensor(np.array([[[[0.1, 0.6], [0.9, 0.9]]]]), mindspore.float32)
>>> prelu = nn.PReLU()
>>> output = prelu(x)
>>> print(output)
[[[[0.1 0.6]
   [0.9 0.9]]]]

tinyms.layers.get_activation(name, prim_name=None)

Gets the activation function.

Parameters

name (str) – The name of the activation function.

Returns

Function, the activation function.

Supported Platforms:

Ascend GPU CPU

Examples

>>> sigmoid = nn.get_activation('sigmoid')
>>> print(sigmoid)
Sigmoid<>

class tinyms.layers.LeakyReLU(alpha=0.2)

Leaky ReLU activation function.

LeakyReLU is similar to ReLU, but LeakyReLU has a slope that makes it not equal to 0 at x < 0. The activation function is defined as:

\[\text{leaky_relu}(x) = \begin{cases}x, &\text{if } x \geq 0; \cr \text{alpha} * x, &\text{otherwise.}\end{cases}\]

See https://ai.stanford.edu/~amaas/papers/relu_hybrid_icml2013_final.pdf

Parameters

alpha (Union[int, float]) – Slope of the activation function at x < 0. Default: 0.2.

Inputs:

• x (Tensor) - The input of LeakyReLU. The shape is \((N,*)\) where \(*\) means, any number of additional dimensions.

Outputs:

Tensor, has the same type and shape as the x.

Raises

TypeError – If alpha is not a float or an int.

Supported Platforms:

Ascend GPU CPU

Examples

>>> x = Tensor(np.array([[-1.0, 4.0, -8.0], [2.0, -5.0, 9.0]]), mindspore.float32)
>>> leaky_relu = nn.LeakyReLU()
>>> output = leaky_relu(x)
>>> print(output)
[[-0.2  4.  -1.6]
 [ 2.  -1.   9. ]]

class tinyms.layers.HSigmoid

Hard sigmoid activation function.

Applies hard sigmoid activation element-wise. The input is a Tensor with any valid shape.

Hard sigmoid is defined as:

\[\text{hsigmoid}(x_{i}) = max(0, min(1, \frac{x_{i} + 3}{6})),\]

where \(x_{i}\) is the \(i\)-th slice in the given dimension of the input Tensor.

Inputs:

• input_x (Tensor) - The input of HSigmoid. The shape is \((N,*)\) where \(*\) means, any number of additional dimensions.

Outputs:

Tensor, with the same type and shape as the input_x.

Raises

TypeError – If input_x is not a Tensor.

Supported Platforms:

Ascend GPU CPU

Examples

>>> x = Tensor(np.array([-1, -2, 0, 2, 1]), mindspore.float16)
>>> hsigmoid = nn.HSigmoid()
>>> result = hsigmoid(x)
>>> print(result)
[0.3333 0.1666 0.5    0.8335 0.6665]

class tinyms.layers.HSwish

Hard swish activation function.

Applies hswish-type activation element-wise. The input is a Tensor with any valid shape.

Hard swish is defined as:

\[\text{hswish}(x_{i}) = x_{i} * \frac{ReLU6(x_{i} + 3)}{6},\]

where \(x_{i}\) is the \(i\)-th slice in the given dimension of the input Tensor.

Inputs:

• x (Tensor) - The input of HSwish, data type must be float16 or float32. The shape is \((N,*)\) where \(*\) means, any number of additional dimensions.

Outputs:

Tensor, with the same type and shape as the x.

Raises

TypeError – If dtype of x is neither float16 nor float32.

Supported Platforms:

GPU CPU

Examples

>>> x = Tensor(np.array([-1, -2, 0, 2, 1]), mindspore.float16)
>>> hswish = nn.HSwish()
>>> result = hswish(x)
>>> print(result)
[-0.3333  -0.3333  0  1.666  0.6665]

class tinyms.layers.ELU(alpha=1.0)

Exponential Linear Uint activation function.

Applies the exponential linear unit function element-wise. The activation function is defined as:

\[E_{i} = \begin{cases} x, &\text{if } x \geq 0; \cr \text{alpha} * (\exp(x_i) - 1), &\text{otherwise.} \end{cases}\]

The picture about ELU looks like this ELU.

Parameters

alpha (float) – The coefficient of negative factor whose type is float. Default: 1.0.

Inputs:

• x (Tensor) - The input of ELU with data type of float16 or float32. The shape is \((N,*)\) where \(*\) means,any number of additional dimensions.

Outputs:

Tensor, with the same type and shape as the x.

Raises

• TypeError – If alpha is not a float.

• TypeError – If dtype of x is neither float16 nor float32.

• ValueError – If alpha is not equal to 1.0.

Supported Platforms:

Ascend GPU CPU

Examples

>>> x = Tensor(np.array([-1, -2, 0, 2, 1]), mindspore.float32)
>>> elu = nn.ELU()
>>> result = elu(x)
>>> print(result)
[-0.63212055  -0.86466473  0.  2.  1.]

class tinyms.layers.LogSigmoid

Logsigmoid activation function.

Applies logsigmoid activation element-wise. The input is a Tensor with any valid shape.

Logsigmoid is defined as:

\[\text{logsigmoid}(x_{i}) = log(\frac{1}{1 + \exp(-x_i)}),\]

where \(x_{i}\) is the element of the input.

Inputs:

• x (Tensor) - The input of LogSigmoid with data type of float16 or float32. The shape is \((N,*)\) where \(*\) means, any number of additional dimensions.

Outputs:

Tensor, with the same type and shape as the x.

Raises

TypeError – If dtype of x is neither float16 nor float32.

Supported Platforms:

Ascend GPU CPU

Examples

>>> net = nn.LogSigmoid()
>>> x = Tensor(np.array([1.0, 2.0, 3.0]), mindspore.float32)
>>> output = net(x)
>>> print(output)
[-0.31326166 -0.12692806 -0.04858734]

class tinyms.layers.SoftShrink(lambd=0.5)

Applies the soft shrinkage function elementwise.

\[\begin{split}\text{SoftShrink}(x) = \begin{cases} x - \lambda, & \text{ if } x > \lambda \\ x + \lambda, & \text{ if } x < -\lambda \\ 0, & \text{ otherwise } \end{cases}\end{split}\]

Parameters

lambd – the \(\lambda\) must be no less than zero value for the Softshrink formulation. Default: 0.5.

Inputs:

• input_x (Tensor) - The input of SoftShrink with data type of float16 or float32. Any number of additional dimensions.

Outputs:

Tensor, has the same shape and data type as input_x.

Raises

• TypeError – If lambd is not a float.

• TypeError – If input_x is not a Tensor.

• TypeError – If dtype of input_x is neither float16 nor float32.

• ValueError – If lambd is less than 0.

Supported Platforms:

Ascend

Examples

>>> input_x = Tensor(np.array([[ 0.5297,  0.7871,  1.1754], [ 0.7836,  0.6218, -1.1542]]), mstype.float16)
>>> softshrink = nn.SoftShrink()
>>> output = softshrink(input_x)
>>> print(output)
[[ 0.02979  0.287    0.676  ]
 [ 0.2837   0.1216  -0.6543 ]]

class tinyms.layers.HShrink(lambd=0.5)

Applies the hard shrinkage function element-wise, each element complies the follow function:

\[\begin{split}\text{HardShrink}(x) = \begin{cases} x, & \text{ if } x > \lambda \\ x, & \text{ if } x < -\lambda \\ 0, & \text{ otherwise } \end{cases}\end{split}\]

Parameters

lambd (float) – The value for the HardShrink formulation. Default: 0.5

Inputs:

• input_x (Tensor) - The input of HardShrink with data type of float16 or float32.

Outputs:

Tensor, the same shape and data type as the input.

Supported Platforms:

Ascend

Raises

• TypeError – If lambd is not a float.

• TypeError – If dtype of input_x is neither float16 nor float32.

Examples

>>> input_x = Tensor(np.array([[ 0.5,  1,  2.0],[0.0533,0.0776,-2.1233]]),mstype.float32)
>>> hshrink = nn.HShrink()
>>> output = hshrink(input_x)
>>> print(output)
[[ 0.      1.      2.    ]
[ 0.      0.     -2.1233]]

class tinyms.layers.BatchNorm1d(num_features, eps=1e-05, momentum=0.9, affine=True, gamma_init='ones', beta_init='zeros', moving_mean_init='zeros', moving_var_init='ones', use_batch_statistics=None)

Batch Normalization layer over a 2D input.

Batch Normalization is widely used in convolutional networks. This layer applies Batch Normalization over a 2D input (a mini-batch of 1D inputs) to reduce internal covariate shift as described in the paper Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. It rescales and recenters the feature using a mini-batch of data and the learned parameters which can be described in the following formula.

\[y = \frac{x - \mathrm{E}[x]}{\sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta\]

Note

The implementation of BatchNorm is different in graph mode and pynative mode, therefore the mode is not recommended to be changed after net was initialized.

Parameters

• num_features (int) – C from an expected input of size (N, C).

• eps (float) – A value added to the denominator for numerical stability. Default: 1e-5.

• momentum (float) – A floating hyperparameter of the momentum for the running_mean and running_var computation. Default: 0.9.

• affine (bool) – A bool value. When set to True, gamma and beta can be learned. Default: True.

• gamma_init (Union[Tensor, str, Initializer, numbers.Number]) – Initializer for the gamma weight. The values of str refer to the function initializer including ‘zeros’, ‘ones’, etc. Default: ‘ones’.

• beta_init (Union[Tensor, str, Initializer, numbers.Number]) – Initializer for the beta weight. The values of str refer to the function initializer including ‘zeros’, ‘ones’, etc. Default: ‘zeros’.

• moving_mean_init (Union[Tensor, str, Initializer, numbers.Number]) – Initializer for the moving mean. The values of str refer to the function initializer including ‘zeros’, ‘ones’, etc. Default: ‘zeros’.

• moving_var_init (Union[Tensor, str, Initializer, numbers.Number]) – Initializer for the moving variance. The values of str refer to the function initializer including ‘zeros’, ‘ones’, etc. Default: ‘ones’.

• use_batch_statistics (bool) – If true, use the mean value and variance value of current batch data. If false, use the mean value and variance value of specified value. If None, the training process will use the mean and variance of current batch data and track the running mean and variance, the evaluation process will use the running mean and variance. Default: None.

Inputs:

• x (Tensor) - Tensor of shape \((N, C_{in})\).

Outputs:

Tensor, the normalized, scaled, offset tensor, of shape \((N, C_{out})\).

Supported Platforms:

Ascend GPU CPU

Raises

• TypeError – If num_features is not an int.

• TypeError – If eps is not a float.

• ValueError – If num_features is less than 1.

• ValueError – If momentum is not in range [0, 1].

Examples

>>> import numpy as np
>>> import mindspore.nn as nn
>>> from mindspore import Tensor
>>> net = nn.BatchNorm1d(num_features=4)
>>> x = Tensor(np.array([[0.7, 0.5, 0.5, 0.6],
...                      [0.5, 0.4, 0.6, 0.9]]).astype(np.float32))
>>> output = net(x)
>>> print(output)
[[ 0.6999965   0.4999975  0.4999975  0.59999704 ]
 [ 0.4999975   0.399998   0.59999704 0.89999545 ]]

class tinyms.layers.BatchNorm2d(num_features, eps=1e-05, momentum=0.9, affine=True, gamma_init='ones', beta_init='zeros', moving_mean_init='zeros', moving_var_init='ones', use_batch_statistics=None, data_format='NCHW')

Batch Normalization layer over a 4D input.

Batch Normalization is widely used in convolutional networks. This layer applies Batch Normalization over a 4D input (a mini-batch of 2D inputs with additional channel dimension) to avoid internal covariate shift as described in the paper Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. It rescales and recenters the feature using a mini-batch of data and the learned parameters which can be described in the following formula.

\[y = \frac{x - \mathrm{E}[x]}{\sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta\]

Note

The implementation of BatchNorm is different in graph mode and pynative mode, therefore that mode can not be changed after net was initialized. Note that the formula for updating the running_mean and running_var is \(\hat{x}_\text{new} = (1 - \text{momentum}) \times x_t + \text{momentum} \times \hat{x}\), where \(\hat{x}\) is the estimated statistic and \(x_t\) is the new observed value.

Parameters

• num_features (int) – C from an expected input of size (N, C, H, W).

• eps (float) – A value added to the denominator for numerical stability. Default: 1e-5.

• momentum (float) – A floating hyperparameter of the momentum for the running_mean and running_var computation. Default: 0.9.

• affine (bool) – A bool value. When set to True, gamma and beta can be learned. Default: True.

• gamma_init (Union[Tensor, str, Initializer, numbers.Number]) – Initializer for the gamma weight. The values of str refer to the function initializer including ‘zeros’, ‘ones’, etc. Default: ‘ones’.

• beta_init (Union[Tensor, str, Initializer, numbers.Number]) – Initializer for the beta weight. The values of str refer to the function initializer including ‘zeros’, ‘ones’, etc. Default: ‘zeros’.

• moving_mean_init (Union[Tensor, str, Initializer, numbers.Number]) – Initializer for the moving mean. The values of str refer to the function initializer including ‘zeros’, ‘ones’, etc. Default: ‘zeros’.

• moving_var_init (Union[Tensor, str, Initializer, numbers.Number]) – Initializer for the moving variance. The values of str refer to the function initializer including ‘zeros’, ‘ones’, etc. Default: ‘ones’.

• use_batch_statistics (bool) –

  • If true, use the mean value and variance value of current batch data and track running mean and running varance.

  • If false, use the mean value and variance value of specified value, and not track statistical value.

  • If None, The use_batch_statistics is automatically assigned process according to the training and eval mode. During training, batchnorm2d process will be the same with use_batch_statistics=True. Contrarily, in eval, batchnorm2d process will be the same with use_batch_statistics=False. Default: None.

• data_format (str) – The optional value for data format, is ‘NHWC’ or ‘NCHW’. Default: ‘NCHW’.

Inputs:

• x (Tensor) - Tensor of shape \((N, C_{in}, H_{in}, W_{in})\).

Outputs:

Tensor, the normalized, scaled, offset tensor, of shape \((N, C_{out}, H_{out}, W_{out})\).

Raises

• TypeError – If num_features is not an int.

• TypeError – If eps is not a float.

• ValueError – If num_features is less than 1.

• ValueError – If momentum is not in range [0, 1].

• ValueError – If data_format is neither ‘NHWC’ not ‘NCHW’.

Supported Platforms:

Ascend GPU CPU

Examples

>>> import numpy as np
>>> import mindspore.nn as nn
>>> from mindspore import Tensor
>>> net = nn.BatchNorm2d(num_features=3)
>>> x = Tensor(np.ones([1, 3, 2, 2]).astype(np.float32))
>>> output = net(x)
>>> print(output)
[[[[ 0.999995 0.999995 ]
   [ 0.999995 0.999995 ]]
  [[ 0.999995 0.999995 ]
   [ 0.999995 0.999995 ]]
  [[ 0.999995 0.999995 ]
   [ 0.999995 0.999995 ]]]]

class tinyms.layers.BatchNorm3d(num_features, eps=1e-05, momentum=0.9, affine=True, gamma_init='ones', beta_init='zeros', moving_mean_init='zeros', moving_var_init='ones', use_batch_statistics=None, data_format='NCDHW')

Batch Normalization layer over a 5D input.

Batch Normalization is widely used in convolutional networks. This layer applies Batch Normalization over a 5D input (a mini-batch of 3D inputs with additional channel dimension) to avoid internal covariate shift.

\[y = \frac{x - \mathrm{E}[x]}{\sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta\]

Note

The implementation of BatchNorm is different in graph mode and pynative mode, therefore that mode can not be changed after net was initialized. Note that the formula for updating the running_mean and running_var is \(\hat{x}_\text{new} = (1 - \text{momentum}) \times x_t + \text{momentum} \times \hat{x}\), where \(\hat{x}\) is the estimated statistic and \(x_t\) is the new observed value.

Parameters

• num_features (int) – C from an expected input of size (N, C, D, H, W).

• eps (float) – A value added to the denominator for numerical stability. Default: 1e-5.

• momentum (float) – A floating hyperparameter of the momentum for the running_mean and running_var computation. Default: 0.9.

• affine (bool) – A bool value. When set to True, gamma and beta can be learned. Default: True.

• gamma_init (Union[Tensor, str, Initializer, numbers.Number]) – Initializer for the gamma weight. The values of str refer to the function initializer including ‘zeros’, ‘ones’, etc. Default: ‘ones’.

• beta_init (Union[Tensor, str, Initializer, numbers.Number]) – Initializer for the beta weight. The values of str refer to the function initializer including ‘zeros’, ‘ones’, etc. Default: ‘zeros’.

• moving_mean_init (Union[Tensor, str, Initializer, numbers.Number]) – Initializer for the moving mean. The values of str refer to the function initializer including ‘zeros’, ‘ones’, etc. Default: ‘zeros’.

• moving_var_init (Union[Tensor, str, Initializer, numbers.Number]) – Initializer for the moving variance. The values of str refer to the function initializer including ‘zeros’, ‘ones’, etc. Default: ‘ones’.

• use_batch_statistics (bool) – If true, use the mean value and variance value of current batch data. If false, use the mean value and variance value of specified value. If None, the training process will use the mean and variance of current batch data and track the running mean and variance, the evaluation process will use the running mean and variance. Default: None.

• data_format (str) – The optional value for data format is ‘NCDHW’. Default: ‘NCDHW’.

Inputs:

• x (Tensor) - Tensor of shape \((N, C_{in}, D_{in}, H_{in}, W_{in})\).

Outputs:

Tensor, the normalized, scaled, offset tensor, of shape \((N, C_{out}, D_{out},H_{out}, W_{out})\).

Raises

• TypeError – If num_features is not an int.

• TypeError – If eps is not a float.

• ValueError – If num_features is less than 1.

• ValueError – If momentum is not in range [0, 1].

• ValueError – If data_format is not ‘NCDHW’.

Supported Platforms:

Ascend GPU CPU

Examples

>>> import numpy as np
>>> import mindspore.nn as nn
>>> from mindspore import Tensor
>>> net = nn.BatchNorm3d(num_features=3)
>>> x = Tensor(np.ones([16, 3, 10, 32, 32]).astype(np.float32))
>>> output = net(x)
>>> print(output.shape)
(16, 3, 10, 32, 32)

class tinyms.layers.LayerNorm(normalized_shape, begin_norm_axis=-1, begin_params_axis=-1, gamma_init='ones', beta_init='zeros', epsilon=1e-07)

Applies Layer Normalization over a mini-batch of inputs.

Layer Normalization is widely used in recurrent neural networks. It applies normalization on a mini-batch of inputs for each single training case as described in the paper Layer Normalization. Unlike Batch Normalization, Layer Normalization performs exactly the same computation at training and testing time. It can be described using the following formula. It is applied across all channels and pixel but only one batch size.

\[y = \frac{x - \mathrm{E}[x]}{\sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta\]

Parameters

• normalized_shape (Union(tuple[int], list[int]) – The normalization is performed over axis begin_norm_axis … R - 1.

• begin_norm_axis (int) – The first normalization dimension: normalization will be performed along dimensions begin_norm_axis: rank(inputs), the value should be in [-1, rank(input)). Default: -1.

• begin_params_axis (int) – The first parameter(beta, gamma)dimension: scale and centering parameters will have dimensions begin_params_axis: rank(inputs) and will be broadcast with the normalized inputs accordingly, the value should be in [-1, rank(input)). Default: -1.

• gamma_init (Union[Tensor, str, Initializer, numbers.Number]) – Initializer for the gamma weight. The values of str refer to the function initializer including ‘zeros’, ‘ones’, ‘xavier_uniform’, ‘he_uniform’, etc. Default: ‘ones’.

• beta_init (Union[Tensor, str, Initializer, numbers.Number]) – Initializer for the beta weight. The values of str refer to the function initializer including ‘zeros’, ‘ones’, ‘xavier_uniform’, ‘he_uniform’, etc. Default: ‘zeros’.

• epsilon (float) – A value added to the denominator for numerical stability. Default: 1e-7.

Inputs:

• x (Tensor) - The shape of ‘x’ is \((x_1, x_2, ..., x_R)\), and input_shape[begin_norm_axis:] is equal to normalized_shape.

Outputs:

Tensor, the normalized and scaled offset tensor, has the same shape and data type as the x.

Raises

• TypeError – If normalized_shape is neither a list nor tuple.

• TypeError – If begin_norm_axis or begin_params_axis is not an int.

• TypeError – If epsilon is not a float.

Supported Platforms:

Ascend GPU CPU

Examples

>>> x = Tensor(np.ones([20, 5, 10, 10]), mindspore.float32)
>>> shape1 = x.shape[1:]
>>> m = nn.LayerNorm(shape1,  begin_norm_axis=1, begin_params_axis=1)
>>> output = m(x).shape
>>> print(output)
(20, 5, 10, 10)

extend_repr()

Display instance object as string.

class tinyms.layers.GroupNorm(num_groups, num_channels, eps=1e-05, affine=True, gamma_init='ones', beta_init='zeros')

Group Normalization over a mini-batch of inputs.

Group Normalization is widely used in recurrent neural networks. It applies normalization on a mini-batch of inputs for each single training case as described in the paper Group Normalization. Group Normalization divides the channels into groups and computes within each group the mean and variance for normalization, and it performs very stable over a wide range of batch size. It can be described using the following formula.

\[y = \frac{x - \mathrm{E}[x]}{\sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta\]

Parameters

• num_groups (int) – The number of groups to be divided along the channel dimension.

• num_channels (int) – The number of channels per group.

• eps (float) – A value added to the denominator for numerical stability. Default: 1e-5.

• affine (bool) – A bool value, this layer will have learnable affine parameters when set to true. Default: True.

• gamma_init (Union[Tensor, str, Initializer, numbers.Number]) – Initializer for the gamma weight. The values of str refer to the function initializer including ‘zeros’, ‘ones’, ‘xavier_uniform’, ‘he_uniform’, etc. Default: ‘ones’. If gamma_init is a Tensor, the shape must be [num_channels].

• beta_init (Union[Tensor, str, Initializer, numbers.Number]) – Initializer for the beta weight. The values of str refer to the function initializer including ‘zeros’, ‘ones’, ‘xavier_uniform’, ‘he_uniform’, etc. Default: ‘zeros’. If beta_init is a Tensor, the shape must be [num_channels].

Inputs:

• x (Tensor) - The input feature with shape [N, C, H, W].

Outputs:

Tensor, the normalized and scaled offset tensor, has the same shape and data type as the x.

Raises

• TypeError – If num_groups or num_channels is not an int.

• TypeError – If eps is not a float.

• TypeError – If affine is not a bool.

• ValueError – If num_groups or num_channels is less than 1.

• ValueError – If num_channels is not divided by num_groups.

Supported Platforms:

Ascend GPU CPU

Examples

>>> group_norm_op = nn.GroupNorm(2, 2)
>>> x = Tensor(np.ones([1, 2, 4, 4], np.float32))
>>> output = group_norm_op(x)
>>> print(output)
[[[[0. 0. 0. 0.]
   [0. 0. 0. 0.]
   [0. 0. 0. 0.]
   [0. 0. 0. 0.]]
  [[0. 0. 0. 0.]
   [0. 0. 0. 0.]
   [0. 0. 0. 0.]
   [0. 0. 0. 0.]]]]

extend_repr()

Display instance object as string.

class tinyms.layers.GlobalBatchNorm(**kwargs)

Global Batch Normalization layer over a N-dimension input.

Global Batch Normalization is cross device synchronized Batch Normalization. The implementation of Batch Normalization only normalizes the data within each device. Global Normalization will normalize the input within the group.It has been described in the paper Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. It rescales and recenters the feature using a mini-batch of data and the learned parameters which can be described in the following formula.

\[y = \frac{x - \mathrm{E}[x]}{\sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta\]

Note

Currently, GlobalBatchNorm only supports 2D and 4D inputs.

Parameters

• num_features (int) – C from an expected input of size (N, C, H, W).

• device_num_each_group (int) – The number of devices in each group. Default: 2.

• eps (float) – A value added to the denominator for numerical stability. Default: 1e-5.

• momentum (float) – A floating hyperparameter of the momentum for the running_mean and running_var computation. Default: 0.9.

• affine (bool) – A bool value. When set to True, gamma and beta can be learned. Default: True.

• gamma_init (Union[Tensor, str, Initializer, numbers.Number]) – Initializer for the gamma weight. The values of str refer to the function initializer including ‘zeros’, ‘ones’, ‘xavier_uniform’, ‘he_uniform’, etc. Default: ‘ones’.

• beta_init (Union[Tensor, str, Initializer, numbers.Number]) – Initializer for the beta weight. The values of str refer to the function initializer including ‘zeros’, ‘ones’, ‘xavier_uniform’, ‘he_uniform’, etc. Default: ‘zeros’.

• moving_mean_init (Union[Tensor, str, Initializer, numbers.Number]) – Initializer for the moving mean. The values of str refer to the function initializer including ‘zeros’, ‘ones’, ‘xavier_uniform’, ‘he_uniform’, etc. Default: ‘zeros’.

• moving_var_init (Union[Tensor, str, Initializer, numbers.Number]) – Initializer for the moving variance. The values of str refer to the function initializer including ‘zeros’, ‘ones’, ‘xavier_uniform’, ‘he_uniform’, etc. Default: ‘ones’.

• use_batch_statistics (bool) – If true, use the mean value and variance value of current batch data. If false, use the mean value and variance value of specified value. If None, training process will use the mean and variance of current batch data and track the running mean and variance, eval process will use the running mean and variance. Default: None.

Inputs:

• x (Tensor) - Tensor of shape \((N, C_{in}, H_{in}, W_{in})\).

Outputs:

Tensor, the normalized, scaled, offset tensor, of shape \((N, C_{out}, H_{out}, W_{out})\).

Raises

• TypeError – If num_features or device_num_each_group is not an int.

• TypeError – If eps is not a float.

• ValueError – If num_features is less than 1.

• ValueError – If momentum is not in range [0, 1].

• ValueError – If device_num_each_group is less than 2.

Supported Platforms:

Ascend

Examples

>>> # This example should be run with multiple processes.
>>> # Please refer to the tutorial > Distributed Training on mindspore.cn.
>>> import numpy as np
>>> from mindspore.communication import init
>>> from mindspore import context
>>> from mindspore.context import ParallelMode
>>> from mindspore import nn
>>> from mindspore import Tensor
>>>
>>> context.set_context(mode=context.GRAPH_MODE)
>>> init()
>>> context.reset_auto_parallel_context()
>>> context.set_auto_parallel_context(parallel_mode=ParallelMode.DATA_PARALLEL)
>>> global_bn_op = nn.GlobalBatchNorm(num_features=3, device_num_each_group=2)
>>> x = Tensor(np.ones([1, 3, 2, 2]).astype(np.float32))
>>> output = global_bn_op(x)
>>> print(output)
[[[[ 0.999995 0.999995 ]
   [ 0.999995 0.999995 ]]
  [[ 0.999995 0.999995 ]
   [ 0.999995 0.999995 ]]
  [[ 0.999995 0.999995 ]
   [ 0.999995 0.999995 ]]]]

class tinyms.layers.SyncBatchNorm(num_features, eps=1e-05, momentum=0.9, affine=True, gamma_init='ones', beta_init='zeros', moving_mean_init='zeros', moving_var_init='ones', use_batch_statistics=None, process_groups=None)

Sync Batch Normalization layer over a N-dimension input.

Sync Batch Normalization is cross device synchronized Batch Normalization. The implementation of Batch Normalization only normalizes the data within each device. Sync Batch Normalization will normalize the input within the group. It has been described in the paper Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. It rescales and recenters the feature using a mini-batch of data and the learned parameters which can be described in the following formula.

\[y = \frac{x - \mathrm{E}[x]}{\sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta\]

Note

Currently, SyncBatchNorm only supports 2D and 4D inputs.

Parameters

• num_features (int) – C from an expected input of size (N, C, H, W).

• eps (float) – A value added to the denominator for numerical stability. Default: 1e-5.

• momentum (float) – A floating hyperparameter of the momentum for the running_mean and running_var computation. Default: 0.9.

• affine (bool) – A bool value. When set to True, gamma and beta can be learned. Default: True.

• gamma_init (Union[Tensor, str, Initializer, numbers.Number]) – Initializer for the gamma weight. The values of str refer to the function initializer including ‘zeros’, ‘ones’, ‘xavier_uniform’, ‘he_uniform’, etc. Default: ‘ones’.

• beta_init (Union[Tensor, str, Initializer, numbers.Number]) – Initializer for the beta weight. The values of str refer to the function initializer including ‘zeros’, ‘ones’, ‘xavier_uniform’, ‘he_uniform’, etc. Default: ‘zeros’.

• moving_mean_init (Union[Tensor, str, Initializer, numbers.Number]) – Initializer for the moving mean. The values of str refer to the function initializer including ‘zeros’, ‘ones’, ‘xavier_uniform’, ‘he_uniform’, etc. Default: ‘zeros’.

• moving_var_init (Union[Tensor, str, Initializer, numbers.Number]) – Initializer for the moving variance. The values of str refer to the function initializer including ‘zeros’, ‘ones’, ‘xavier_uniform’, ‘he_uniform’, etc. Default: ‘ones’.

• use_batch_statistics (bool) – If true, use the mean value and variance value of current batch data. If false, use the mean value and variance value of specified value. If None, training process will use the mean and variance of current batch data and track the running mean and variance, eval process will use the running mean and variance. Default: None.

• process_groups (list) – A list to divide devices into different sync groups, containing N subtraction lists. Each subtraction list contains int numbers identifying rank ids which need to be synchronized in the same group. All int values must be in [0, rank_size) and different from each other. Default: None, indicating synchronization across all devices.

Inputs:

• x (Tensor) - Tensor of shape \((N, C_{in}, H_{in}, W_{in})\).

Outputs:

Tensor, the normalized, scaled, offset tensor, of shape \((N, C_{out}, H_{out}, W_{out})\).

Raises

• TypeError – If num_features is not an int.

• TypeError – If eps is not a float.

• TypeError – If process_groups is not a list.

• ValueError – If num_features is less than 1.

• ValueError – If momentum is not in range [0, 1].

• ValueError – If rank_id in process_groups is not in range [0, rank_size).

Supported Platforms:

Ascend

Examples

>>> # This example should be run with multiple processes.
>>> # Please refer to the tutorial > Distributed Training on mindspore.cn.
>>> import numpy as np
>>> from mindspore.communication import init
>>> from mindspore import context
>>> from mindspore.context import ParallelMode
>>> from mindspore import Tensor
>>> from mindspore import nn
>>> from mindspore import dtype as mstype
>>>
>>> context.set_context(mode=context.GRAPH_MODE)
>>> init()
>>> context.reset_auto_parallel_context()
>>> context.set_auto_parallel_context(parallel_mode=ParallelMode.DATA_PARALLEL)
>>> sync_bn_op = nn.SyncBatchNorm(num_features=3, process_groups=[[0, 1], [2, 3]])
>>> x = Tensor(np.ones([1, 3, 2, 2]), mstype.float32)
>>> output = sync_bn_op(x)
>>> print(output)
[[[[ 0.999995 0.999995 ]
   [ 0.999995 0.999995 ]]
  [[ 0.999995 0.999995 ]
   [ 0.999995 0.999995 ]]
  [[ 0.999995 0.999995 ]
   [ 0.999995 0.999995 ]]]]

class tinyms.layers.InstanceNorm2d(num_features, eps=1e-05, momentum=0.1, affine=True, gamma_init='ones', beta_init='zeros')

Instance Normalization layer over a 4D input.

This layer applies Instance Normalization over a 4D input (a mini-batch of 2D inputs with additional channel dimension) as described in the paper Instance Normalization: The Missing Ingredient for Fast Stylization. It rescales and recenters the feature using a mini-batch of data and the learned parameters which can be described in the following formula.

\[y = \frac{x - \mathrm{E}[x]}{\sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta\]

gamma and beta are learnable parameter vectors of size num_features if affine is True. The standard-deviation is calculated via the biased estimator.

This layer uses instance statistics computed from input data in both training and evaluation modes.

InstanceNorm2d and BatchNorm2d are very similar, but have some differences. InstanceNorm2d is applied on each channel of channeled data like RGB images, but BatchNorm2d is usually applied on each batch of batched data.

Note

Note that the formula for updating the running_mean and running_var is \(\hat{x}_\text{new} = (1 - \text{momentum}) \times x_t + \text{momentum} \times \hat{x}\), where \(\hat{x}\) is the estimated statistic and \(x_t\) is the new observed value.

Parameters

• num_features (int) – C from an expected input of size (N, C, H, W).

• eps (float) – A value added to the denominator for numerical stability. Default: 1e-5.

• momentum (float) – A floating hyperparameter of the momentum for the running_mean and running_var computation. Default: 0.1.

• affine (bool) – A bool value. When set to True, gamma and beta can be learned. Default: True.

• gamma_init (Union[Tensor, str, Initializer, numbers.Number]) – Initializer for the gamma weight. The values of str refer to the function initializer including ‘zeros’, ‘ones’, etc. Default: ‘ones’.

• beta_init (Union[Tensor, str, Initializer, numbers.Number]) – Initializer for the beta weight. The values of str refer to the function initializer including ‘zeros’, ‘ones’, etc. Default: ‘zeros’.

Inputs:

• x (Tensor) - Tensor of shape \((N, C, H, W)\). Data type: float16 or float32.

Outputs:

Tensor, the normalized, scaled, offset tensor, of shape \((N, C, H, W)\). Same type and shape as the x.

Supported Platforms:

GPU

Raises

• TypeError – If num_features is not an int.

• TypeError – If eps is not a float.

• TypeError – If momentum is not a float.

• TypeError – If affine is not a bool.

• TypeError – If the type of gamma_init/beta_init is not same, or if the initialized element type is not float32.

• ValueError – If num_features is less than 1.

• ValueError – If momentum is not in range [0, 1].

• KeyError – If any of gamma_init/beta_init is str and the homonymous class inheriting from Initializer not exists.

Examples

>>> import mindspore
>>> import numpy as np
>>> import mindspore.nn as nn
>>> from mindspore import Tensor
>>> net = nn.InstanceNorm2d(3)
>>> x = Tensor(np.ones([2, 3, 2, 2]), mindspore.float32)
>>> output = net(x)
>>> print(output.shape)
(2, 3, 2, 2)

class tinyms.layers.SequentialCell(*args)

Sequential cell container.

A list of Cells will be added to it in the order they are passed in the constructor. Alternatively, an ordered dict of cells can also be passed in.

Parameters

args (list, OrderedDict) – List of subclass of Cell.

Inputs:

• x (Tensor) - Tensor with shape according to the first Cell in the sequence.

Outputs:

Tensor, the output Tensor with shape depending on the input x and defined sequence of Cells.

Raises

TypeError – If the type of the args is not list or OrderedDict.

Supported Platforms:

Ascend GPU CPU

Examples

>>> conv = nn.Conv2d(3, 2, 3, pad_mode='valid', weight_init="ones")
>>> relu = nn.ReLU()
>>> seq = nn.SequentialCell([conv, relu])
>>> x = Tensor(np.ones([1, 3, 4, 4]), dtype=mindspore.float32)
>>> output = seq(x)
>>> print(output)
[[[[27. 27.]
   [27. 27.]]
  [[27. 27.]
   [27. 27.]]]]

append(cell)

Appends a given cell to the end of the list.

Examples

>>> conv = nn.Conv2d(3, 2, 3, pad_mode='valid', weight_init="ones")
>>> bn = nn.BatchNorm2d(2)
>>> relu = nn.ReLU()
>>> seq = nn.SequentialCell([conv, bn])
>>> seq.append(relu)
>>> x = Tensor(np.ones([1, 3, 4, 4]), dtype=mindspore.float32)
>>> output = seq(x)
>>> print(output)
[[[[26.999863 26.999863]
   [26.999863 26.999863]]
  [[26.999863 26.999863]
   [26.999863 26.999863]]]]

class tinyms.layers.CellList(*args, **kwargs)

Holds Cells in a list.

CellList can be used like a regular Python list, support ‘__getitem__’, ‘__setitem__’, ‘__delitem__’, ‘__len__’, ‘__iter__’ and ‘__iadd__’, but cells it contains are properly registered, and will be visible by all Cell methods.

Parameters

args (list, optional) – List of subclass of Cell.

Supported Platforms:

Ascend GPU CPU

Examples

>>> conv = nn.Conv2d(100, 20, 3)
>>> bn = nn.BatchNorm2d(20)
>>> relu = nn.ReLU()
>>> cell_ls = nn.CellList([bn])
>>> cell_ls.insert(0, conv)
>>> cell_ls.append(relu)
>>> print(cell_ls)
CellList<
  (0): Conv2d<input_channels=100, output_channels=20, kernel_size=(3, 3),stride=(1, 1),  pad_mode=same,
  padding=0, dilation=(1, 1), group=1, has_bias=False, weight_init=normal, bias_init=zeros, format=NCHW>
  (1): BatchNorm2d<num_features=20, eps=1e-05, momentum=0.09999999999999998, gamma=Parameter (name=1.gamma,
  shape=(20,), dtype=Float32, requires_grad=True), beta=Parameter (name=1.beta, shape=(20,), dtype=Float32,
  requires_grad=True), moving_mean=Parameter (name=1.moving_mean, shape=(20,), dtype=Float32,
  requires_grad=False), moving_variance=Parameter (name=1.moving_variance, shape=(20,), dtype=Float32,
  requires_grad=False)>
  (2): ReLU<>
  >

append(cell)

Appends a given cell to the end of the list.

extend(cells)

Appends cells from a Python iterable to the end of the list.

Raises

TypeError – If the cells are not a list of subcells.

insert(index, cell)

Inserts a given cell before a given index in the list.

class tinyms.layers.Conv2d(in_channels, out_channels, kernel_size, stride=1, pad_mode='same', padding=0, dilation=1, group=1, has_bias=False, weight_init='normal', bias_init='zeros', data_format='NCHW')

2D convolution layer.

Applies a 2D convolution over an input tensor which is typically of shape \((N, C_{in}, H_{in}, W_{in})\), where \(N\) is batch size, \(C_{in}\) is channel number, and \(H_{in}, W_{in}\) are height and width. For each batch of shape \((C_{in}, H_{in}, W_{in})\), the formula is defined as:

\[out_j = \sum_{i=0}^{C_{in} - 1} ccor(W_{ij}, X_i) + b_j,\]

where \(ccor\) is the cross-correlation operator, \(C_{in}\) is the input channel number, \(j\) ranges from \(0\) to \(C_{out} - 1\), \(W_{ij}\) corresponds to the \(i\)-th channel of the \(j\)-th filter and \(out_{j}\) corresponds to the \(j\)-th channel of the output. \(W_{ij}\) is a slice of kernel and it has shape \((\text{kernel_size[0]}, \text{kernel_size[1]})\), where \(\text{kernel_size[0]}\) and \(\text{kernel_size[1]}\) are the height and width of the convolution kernel. The full kernel has shape \((C_{out}, C_{in} // \text{group}, \text{kernel_size[0]}, \text{kernel_size[1]})\), where group is the group number to split the input x in the channel dimension.

If the ‘pad_mode’ is set to be “valid”, the output height and width will be \(\left \lfloor{1 + \frac{H_{in} + \text{padding[0]} + \text{padding[1]} - \text{kernel_size[0]} - (\text{kernel_size[0]} - 1) \times (\text{dilation[0]} - 1) }{\text{stride[0]}}} \right \rfloor\) and \(\left \lfloor{1 + \frac{W_{in} + \text{padding[2]} + \text{padding[3]} - \text{kernel_size[1]} - (\text{kernel_size[1]} - 1) \times (\text{dilation[1]} - 1) }{\text{stride[1]}}} \right \rfloor\) respectively.

The first introduction can be found in paper Gradient Based Learning Applied to Document Recognition.

Parameters

• in_channels (int) – The number of input channel \(C_{in}\).

• out_channels (int) – The number of output channel \(C_{out}\).

• kernel_size (Union[int, tuple[int]]) – The data type is int or a tuple of 2 integers. Specifies the height and width of the 2D convolution window. Single int means the value is for both the height and the width of the kernel. A tuple of 2 ints means the first value is for the height and the other is for the width of the kernel.

• stride (Union[int, tuple[int]]) – The distance of kernel moving, an int number that represents the height and width of movement are both strides, or a tuple of two int numbers that represent height and width of movement respectively. Default: 1.

• pad_mode (str) –

  Specifies padding mode. The optional values are “same”, “valid”, “pad”. Default: “same”.

  • same: Adopts the way of completion. The height and width of the output will be the same as the input x. The total number of padding will be calculated in horizontal and vertical directions and evenly distributed to top and bottom, left and right if possible. Otherwise, the last extra padding will be done from the bottom and the right side. If this mode is set, padding must be 0.

  • valid: Adopts the way of discarding. The possible largest height and width of output will be returned without padding. Extra pixels will be discarded. If this mode is set, padding must be 0.

  • pad: Implicit paddings on both sides of the input x. The number of padding will be padded to the input Tensor borders. padding must be greater than or equal to 0.

• padding (Union[int, tuple[int]]) – Implicit paddings on both sides of the input x. If padding is one integer, the paddings of top, bottom, left and right are the same, equal to padding. If padding is a tuple with four integers, the paddings of top, bottom, left and right will be equal to padding[0], padding[1], padding[2], and padding[3] accordingly. Default: 0.

• dilation (Union[int, tuple[int]]) – The data type is int or a tuple of 2 integers. Specifies the dilation rate to use for dilated convolution. If set to be \(k > 1\), there will be \(k - 1\) pixels skipped for each sampling location. Its value must be greater or equal to 1 and bounded by the height and width of the input x. Default: 1.

• group (int) – Splits filter into groups, in_ channels and out_channels must be divisible by the number of groups. If the group is equal to in_channels and out_channels, this 2D convolution layer also can be called 2D depthwise convolution layer. Default: 1.

• has_bias (bool) – Specifies whether the layer uses a bias vector. Default: False.

• weight_init (Union[Tensor, str, Initializer, numbers.Number]) – Initializer for the convolution kernel. It can be a Tensor, a string, an Initializer or a number. When a string is specified, values from ‘TruncatedNormal’, ‘Normal’, ‘Uniform’, ‘HeUniform’ and ‘XavierUniform’ distributions as well as constant ‘One’ and ‘Zero’ distributions are possible. Alias ‘xavier_uniform’, ‘he_uniform’, ‘ones’ and ‘zeros’ are acceptable. Uppercase and lowercase are both acceptable. Refer to the values of Initializer for more details. Default: ‘normal’.

• bias_init (Union[Tensor, str, Initializer, numbers.Number]) – Initializer for the bias vector. Possible Initializer and string are the same as ‘weight_init’. Refer to the values of Initializer for more details. Default: ‘zeros’.

• data_format (str) – The optional value for data format, is ‘NHWC’ or ‘NCHW’. Default: ‘NCHW’.

Inputs:

• x (Tensor) - Tensor of shape \((N, C_{in}, H_{in}, W_{in})\) or \((N, H_{in}, W_{in}, C_{in})\).

Outputs:

Tensor of shape \((N, C_{out}, H_{out}, W_{out})\) or \((N, H_{out}, W_{out}, C_{out})\).

Raises

• TypeError – If in_channels, out_channels or group is not an int.

• TypeError – If kernel_size, stride, padding or dilation is neither an int not a tuple.

• ValueError – If in_channels, out_channels, kernel_size, stride or dilation is less than 1.

• ValueError – If padding is less than 0.

• ValueError – If pad_mode is not one of ‘same’, ‘valid’, ‘pad’.

• ValueError – If padding is a tuple whose length is not equal to 4.

• ValueError – If pad_mode is not equal to ‘pad’ and padding is not equal to (0, 0, 0, 0).

• ValueError – If data_format is neither ‘NCHW’ not ‘NHWC’.

Supported Platforms:

Ascend GPU CPU

Examples

>>> net = nn.Conv2d(120, 240, 4, has_bias=False, weight_init='normal')
>>> x = Tensor(np.ones([1, 120, 1024, 640]), mindspore.float32)
>>> output = net(x).shape
>>> print(output)
(1, 240, 1024, 640)

class tinyms.layers.Conv2dTranspose(in_channels, out_channels, kernel_size, stride=1, pad_mode='same', padding=0, dilation=1, group=1, has_bias=False, weight_init='normal', bias_init='zeros')

2D transposed convolution layer.

Compute a 2D transposed convolution, which is also known as a deconvolution (although it is not an actual deconvolution). This module can be seen as the gradient of Conv2d with respect to its input.

x is typically of shape \((N, C, H, W)\), where \(N\) is batch size, \(C\) is channel number, \(H\) is the height of the characteristic layer and \(W\) is the width of the characteristic layer.

The pad_mode argument effectively adds \(dilation * (kernel\_size - 1) - padding\) amount of zero padding to both sizes of the input. So that when a Conv2d and a ConvTranspose2d are initialized with same parameters, they are inverses of each other in regard to the input and output shapes. However, when stride > 1, Conv2d maps multiple input shapes to the same output shape. ConvTranspose2d provide padding argument to increase the calculated output shape on one or more side.

The height and width of output are defined as:

if the ‘pad_mode’ is set to be “pad”,

\[ \begin{align}\begin{aligned}H_{out} = (H_{in} - 1) \times \text{stride[0]} - \left (\text{padding[0]} + \text{padding[1]}\right ) + \text{dilation[0]} \times (\text{kernel_size[0]} - 1) + 1\\W_{out} = (W_{in} - 1) \times \text{stride[1]} - \left (\text{padding[2]} + \text{padding[3]}\right ) + \text{dilation[1]} \times (\text{kernel_size[1]} - 1) + 1\end{aligned}\end{align} \]

if the ‘pad_mode’ is set to be “same”,

\[\begin{split}H_{out} = (H_{in} + \text{stride[0]} - 1)/\text{stride[0]} \\ W_{out} = (W_{in} + \text{stride[1]} - 1)/\text{stride[1]}\end{split}\]

if the ‘pad_mode’ is set to be “valid”,

\[\begin{split}H_{out} = (H_{in} - 1) \times \text{stride[0]} + \text{dilation[0]} \times (\text{ks_w[0]} - 1) + 1 \\ W_{out} = (W_{in} - 1) \times \text{stride[1]} + \text{dilation[1]} \times (\text{ks_w[1]} - 1) + 1\end{split}\]

where \(\text{kernel_size[0]}\) is the height of the convolution kernel and \(\text{kernel_size[1]}\) is the width of the convolution kernel.

Parameters

• in_channels (int) – The number of channels in the input space.

• out_channels (int) – The number of channels in the output space.

• kernel_size (Union[int, tuple]) – int or a tuple of 2 integers, which specifies the height and width of the 2D convolution window. Single int means the value is for both the height and the width of the kernel. A tuple of 2 ints means the first value is for the height and the other is for the width of the kernel.

• stride (Union[int, tuple[int]]) – The distance of kernel moving, an int number that represents the height and width of movement are both strides, or a tuple of two int numbers that represent height and width of movement respectively. Its value must be equal to or greater than 1. Default: 1.

• pad_mode (str) –

  Select the mode of the pad. The optional values are “pad”, “same”, “valid”. Default: “same”.

  • pad: Implicit paddings on both sides of the input x.

  • same: Adopted the way of completion.

  • valid: Adopted the way of discarding.

• padding (Union[int, tuple[int]]) – Implicit paddings on both sides of the input x. If padding is one integer, the paddings of top, bottom, left and right are the same, equal to padding. If padding is a tuple with four integers, the paddings of top, bottom, left and right will be equal to padding[0], padding[1], padding[2], and padding[3] accordingly. Default: 0.

• dilation (Union[int, tuple[int]]) – The data type is int or a tuple of 2 integers. Specifies the dilation rate to use for dilated convolution. If set to be \(k > 1\), there will be \(k - 1\) pixels skipped for each sampling location. Its value must be greater than or equal to 1 and bounded by the height and width of the input x. Default: 1.

• group (int) – Splits filter into groups, in_channels and out_channels must be divisible by the number of groups. This does not support for Davinci devices when group > 1. Default: 1.

• has_bias (bool) – Specifies whether the layer uses a bias vector. Default: False.

• weight_init (Union[Tensor, str, Initializer, numbers.Number]) – Initializer for the convolution kernel. It can be a Tensor, a string, an Initializer or a number. When a string is specified, values from ‘TruncatedNormal’, ‘Normal’, ‘Uniform’, ‘HeUniform’ and ‘XavierUniform’ distributions as well as constant ‘One’ and ‘Zero’ distributions are possible. Alias ‘xavier_uniform’, ‘he_uniform’, ‘ones’ and ‘zeros’ are acceptable. Uppercase and lowercase are both acceptable. Refer to the values of Initializer for more details. Default: ‘normal’.

• bias_init (Union[Tensor, str, Initializer, numbers.Number]) – Initializer for the bias vector. Possible Initializer and string are the same as ‘weight_init’. Refer to the values of Initializer for more details. Default: ‘zeros’.

Inputs:

• x (Tensor) - Tensor of shape \((N, C_{in}, H_{in}, W_{in})\).

Outputs:

Tensor of shape \((N, C_{out}, H_{out}, W_{out})\).

Raises

• TypeError – If in_channels, out_channels or group is not an int.

• TypeError – If kernel_size, stride, padding or dilation is neither an int not a tuple.

• ValueError – If in_channels, out_channels, kernel_size, stride or dilation is less than 1.

• ValueError – If padding is less than 0.

• ValueError – If pad_mode is not one of ‘same’, ‘valid’, ‘pad’.

• ValueError – If padding is a tuple whose length is not equal to 4.

• ValueError – If pad_mode is not equal to ‘pad’ and padding is not equal to (0, 0, 0, 0).

Supported Platforms:

Ascend GPU CPU

Examples

>>> net = nn.Conv2dTranspose(3, 64, 4, has_bias=False, weight_init='normal', pad_mode='pad')
>>> x = Tensor(np.ones([1, 3, 16, 50]), mindspore.float32)
>>> output = net(x).shape
>>> print(output)
(1, 64, 19, 53)

class tinyms.layers.Conv1d(in_channels, out_channels, kernel_size, stride=1, pad_mode='same', padding=0, dilation=1, group=1, has_bias=False, weight_init='normal', bias_init='zeros')

1D convolution layer.

Applies a 1D convolution over an input tensor which is typically of shape \((N, C_{in}, W_{in})\), where \(N\) is batch size and \(C_{in}\) is channel number. For each batch of shape \((C_{in}, W_{in})\), the formula is defined as:

\[out_j = \sum_{i=0}^{C_{in} - 1} ccor(W_{ij}, X_i) + b_j,\]

where \(ccor\) is the cross correlation operator, \(C_{in}\) is the input channel number, \(j\) ranges from \(0\) to \(C_{out} - 1\), \(W_{ij}\) corresponds to the \(i\)-th channel of the \(j\)-th filter and \(out_{j}\) corresponds to the \(j\)-th channel of the output. \(W_{ij}\) is a slice of kernel and it has shape \((\text{ks_w})\), where \(\text{ks_w}\) is the width of the convolution kernel. The full kernel has shape \((C_{out}, C_{in} // \text{group}, \text{ks_w})\), where group is the group number to split the input x in the channel dimension.

If the ‘pad_mode’ is set to be “valid”, the output width will be \(\left \lfloor{1 + \frac{W_{in} + 2 \times \text{padding} - \text{ks_w} - (\text{ks_w} - 1) \times (\text{dilation} - 1) }{\text{stride}}} \right \rfloor\) respectively.

The first introduction of convolution layer can be found in paper Gradient Based Learning Applied to Document Recognition.

Parameters

• in_channels (int) – The number of input channel \(C_{in}\).

• out_channels (int) – The number of output channel \(C_{out}\).

• kernel_size (int) – The data type is int. Specifies the width of the 1D convolution window.

• stride (int) – The distance of kernel moving, an int number that represents the width of movement. Default: 1.

• pad_mode (str) –

  Specifies padding mode. The optional values are “same”, “valid”, “pad”. Default: “same”.

  • same: Adopts the way of completion. The output width will be the same as the input x. The total number of padding will be calculated in the horizontal direction and evenly distributed to left and right if possible. Otherwise, the last extra padding will be done from the bottom and the right side. If this mode is set, padding must be 0.

  • valid: Adopts the way of discarding. The possible largest width of the output will be returned without padding. Extra pixels will be discarded. If this mode is set, padding must be 0.

  • pad: Implicit paddings on both sides of the input x. The number of padding will be padded to the input Tensor borders. padding must be greater than or equal to 0.

• padding (int) – Implicit paddings on both sides of the input x. Default: 0.

• dilation (int) – The data type is int. Specifies the dilation rate to use for dilated convolution. If set to be \(k > 1\), there will be \(k - 1\) pixels skipped for each sampling location. Its value must be greater or equal to 1 and bounded by the height and width of the input x. Default: 1.

• group (int) – Splits filter into groups, in_ channels and out_channels must be divisible by the number of groups. Default: 1.

• has_bias (bool) – Specifies whether the layer uses a bias vector. Default: False.

• weight_init (Union[Tensor, str, Initializer, numbers.Number]) – An initializer for the convolution kernel. It can be a Tensor, a string, an Initializer or a number. When a string is specified, values from ‘TruncatedNormal’, ‘Normal’, ‘Uniform’, ‘HeUniform’ and ‘XavierUniform’ distributions as well as constant ‘One’ and ‘Zero’ distributions are possible. Alias ‘xavier_uniform’, ‘he_uniform’, ‘ones’ and ‘zeros’ are acceptable. Uppercase and lowercase are both acceptable. Refer to the values of Initializer for more details. Default: ‘normal’.

• bias_init (Union[Tensor, str, Initializer, numbers.Number]) – Initializer for the bias vector. Possible Initializer and string are the same as ‘weight_init’. Refer to the values of Initializer for more details. Default: ‘zeros’.

Inputs:

• x (Tensor) - Tensor of shape \((N, C_{in}, W_{in})\).

Outputs:

Tensor of shape \((N, C_{out}, W_{out})\).

Raises

• TypeError – If in_channels, out_channels, kernel_size, stride, padding or dilation is not an int.

• ValueError – If in_channels, out_channels, kernel_size, stride or dilation is less than 1.

• ValueError – If padding is less than 0.

• ValueError – If pad_mode is not one of ‘same’, ‘valid’, ‘pad’.

Supported Platforms:

Ascend GPU CPU

Examples

>>> net = nn.Conv1d(120, 240, 4, has_bias=False, weight_init='normal')
>>> x = Tensor(np.ones([1, 120, 640]), mindspore.float32)
>>> output = net(x).shape
>>> print(output)
(1, 240, 640)

class tinyms.layers.Conv1dTranspose(in_channels, out_channels, kernel_size, stride=1, pad_mode='same', padding=0, dilation=1, group=1, has_bias=False, weight_init='normal', bias_init='zeros')

1D transposed convolution layer.

Compute a 1D transposed convolution, which is also known as a deconvolution (although it is not an actual deconvolution). This module can be seen as the gradient of Conv1d with respect to its input.

x is typically of shape \((N, C, W)\), where \(N\) is batch size, \(C\) is channel number and \(W\) is the characteristic length.

The padding argument effectively adds \(dilation * (kernel\_size - 1) - padding\) amount of zero padding to both sizes of the input. So that when a Conv1d and a ConvTranspose1d are initialized with same parameters, they are inverses of each other in regard to the input and output shapes. However, when stride > 1, Conv1d maps multiple input shapes to the same output shape.

The width of output is defined as:

\[\begin{split}W_{out} = \begin{cases} (W_{in} - 1) \times \text{stride} - 2 \times \text{padding} + \text{dilation} \times (\text{ks_w} - 1) + 1, & \text{if pad_mode='pad'}\\ (W_{in} + \text{stride} - 1)/\text{stride}, & \text{if pad_mode='same'}\\ (W_{in} - 1) \times \text{stride} + \text{dilation} \times (\text{ks_w} - 1) + 1, & \text{if pad_mode='valid'} \end{cases}\end{split}\]

where \(\text{ks_w}\) is the width of the convolution kernel.

Parameters

• in_channels (int) – The number of channels in the input space.

• out_channels (int) – The number of channels in the output space.

• kernel_size (int) – int, which specifies the width of the 1D convolution window.

• stride (int) – The distance of kernel moving, an int number that represents the width of movement. Default: 1.

• pad_mode (str) –

  Select the mode of the pad. The optional values are “pad”, “same”, “valid”. Default: “same”.

  • pad: Implicit paddings on both sides of the input x.

  • same: Adopted the way of completion.

  • valid: Adopted the way of discarding.

• padding (int) – Implicit paddings on both sides of the input x. Default: 0.

• dilation (int) – The data type is int. Specifies the dilation rate to use for dilated convolution. If set to be \(k > 1\), there will be \(k - 1\) pixels skipped for each sampling location. Its value must be greater or equal to 1 and bounded by the width of the input x. Default: 1.

• group (int) – Splits filter into groups, in_channels and out_channels must be divisible by the number of groups. This is not support for Davinci devices when group > 1. Default: 1.

• has_bias (bool) – Specifies whether the layer uses a bias vector. Default: False.

• weight_init (Union[Tensor, str, Initializer, numbers.Number]) – Initializer for the convolution kernel. It can be a Tensor, a string, an Initializer or a numbers.Number. When a string is specified, values from ‘TruncatedNormal’, ‘Normal’, ‘Uniform’, ‘HeUniform’ and ‘XavierUniform’ distributions as well as constant ‘One’ and ‘Zero’ distributions are possible. Alias ‘xavier_uniform’, ‘he_uniform’, ‘ones’ and ‘zeros’ are acceptable. Uppercase and lowercase are both acceptable. Refer to the values of Initializer for more details. Default: ‘normal’.

• bias_init (Union[Tensor, str, Initializer, numbers.Number]) – Initializer for the bias vector. Possible Initializer and string are the same as ‘weight_init’. Refer to the values of Initializer for more details. Default: ‘zeros’.

Inputs:

• x (Tensor) - Tensor of shape \((N, C_{in}, W_{in})\).

Outputs:

Tensor of shape \((N, C_{out}, W_{out})\).

Raises

• TypeError – If in_channels, out_channels, kernel_size, stride, padding or dilation is not an int.

• ValueError – If in_channels, out_channels, kernel_size, stride or dilation is less than 1.

• ValueError – If padding is less than 0.

• ValueError – If pad_mode is not one of ‘same’, ‘valid’, ‘pad’.

Supported Platforms:

Ascend GPU CPU

Examples

>>> net = nn.Conv1dTranspose(3, 64, 4, has_bias=False, weight_init='normal', pad_mode='pad')
>>> x = Tensor(np.ones([1, 3, 50]), mindspore.float32)
>>> output = net(x).shape
>>> print(output)
(1, 64, 53)

class tinyms.layers.Conv3d(in_channels, out_channels, kernel_size, stride=1, pad_mode='same', padding=0, dilation=1, group=1, has_bias=False, weight_init='normal', bias_init='zeros', data_format='NCDHW')

3D convolution layer.

Applies a 3D convolution over an input tensor which is typically of shape \((N, C_{in}, D_{in}, H_{in}, W_{in})\) and output shape \((N, C_{out}, D_{out}, H_{out}, W_{out})\). where \(N\) is batch size. \(C\) is channel number. the formula is defined as:

\[\operatorname{out}\left(N_{i}, C_{\text {out}_j}\right)=\operatorname{bias}\left(C_{\text {out}_j}\right)+ \sum_{k=0}^{C_{in}-1} ccor(\text {weight}\left(C_{\text {out}_j}, k\right), \operatorname{input}\left(N_{i}, k\right))\]

where \(ccor\) is the cross-correlation operator.

If the ‘pad_mode’ is set to be “valid”, the output depth, height and width will be \(\left \lfloor{1 + \frac{D_{in} + \text{padding[0]} + \text{padding[1]} - \text{kernel_size[0]} - (\text{kernel_size[0]} - 1) \times (\text{dilation[0]} - 1) }{\text{stride[0]}}} \right \rfloor\) and \(\left \lfloor{1 + \frac{H_{in} + \text{padding[2]} + \text{padding[3]} - \text{kernel_size[1]} - (\text{kernel_size[1]} - 1) \times (\text{dilation[1]} - 1) }{\text{stride[1]}}} \right \rfloor\) and \(\left \lfloor{1 + \frac{W_{in} + \text{padding[4]} + \text{padding[5]} - \text{kernel_size[2]} - (\text{kernel_size[2]} - 1) \times (\text{dilation[2]} - 1) }{\text{stride[2]}}} \right \rfloor\) respectively.

Parameters

• in_channels (int) – The number of input channel \(C_{in}\).

• out_channels (int) – The number of output channel \(C_{out}\).

• kernel_size (Union[int, tuple[int]]) – The data type is int or a tuple of 3 integers. Specifies the depth, height and width of the 3D convolution window. Single int means the value is for the depth, height and the width of the kernel. A tuple of 3 ints means the first value is for the depth, second value is for height and the other is for the width of the kernel.

• stride (Union[int, tuple[int]]) – The distance of kernel moving, an int number that represents the depth, height and width of movement are both strides, or a tuple of three int numbers that represent depth, height and width of movement respectively. Default: 1.

• pad_mode (str) –

  Specifies padding mode. The optional values are “same”, “valid”, “pad”. Default: “same”.

  • same: Adopts the way of completion. The depth, height and width of the output will be the same as the input x. The total number of padding will be calculated in depth, horizontal and vertical directions and evenly distributed to head and tail, top and bottom, left and right if possible. Otherwise, the last extra padding will be done from the tail, bottom and the right side. If this mode is set, padding must be 0.

  • valid: Adopts the way of discarding. The possible largest depth, height and width of output will be returned without padding. Extra pixels will be discarded. If this mode is set, padding must be 0.

  • pad: Implicit paddings on both sides of the input x in depth, height, width. The number of padding will be padded to the input Tensor borders. padding must be greater than or equal to 0.

• padding (Union(int, tuple[int])) – Implicit paddings on both sides of the input x. The data type is int or a tuple of 6 integers. Default: 0. If padding is an integer, the paddings of head, tail, top, bottom, left and right are the same, equal to padding. If paddings is a tuple of six integers, the padding of head, tail, top, bottom, left and right equal to padding[0], padding[1], padding[2], padding[3], padding[4] and padding[5] correspondingly.

• dilation (Union[int, tuple[int]]) – The data type is int or a tuple of 3 integers : math:(dilation_d, dilation_h, dilation_w). Currently, dilation on depth only supports the case of 1. Specifies the dilation rate to use for dilated convolution. If set to be \(k > 1\), there will be \(k - 1\) pixels skipped for each sampling location. Its value must be greater or equal to 1 and bounded by the height and width of the input x. Default: 1.

• group (int) – Splits filter into groups, in_ channels and out_channels must be divisible by the number of groups. Default: 1. Only 1 is currently supported.

• has_bias (bool) – Specifies whether the layer uses a bias vector. Default: False.

• weight_init (Union[Tensor, str, Initializer, numbers.Number]) – Initializer for the convolution kernel. It can be a Tensor, a string, an Initializer or a number. When a string is specified, values from ‘TruncatedNormal’, ‘Normal’, ‘Uniform’, ‘HeUniform’ and ‘XavierUniform’ distributions as well as constant ‘One’ and ‘Zero’ distributions are possible. Alias ‘xavier_uniform’, ‘he_uniform’, ‘ones’ and ‘zeros’ are acceptable. Uppercase and lowercase are both acceptable. Refer to the values of Initializer for more details. Default: ‘normal’.

• bias_init (Union[Tensor, str, Initializer, numbers.Number]) – Initializer for the bias vector. Possible Initializer and string are the same as ‘weight_init’. Refer to the values of Initializer for more details. Default: ‘zeros’.

• data_format (str) – The optional value for data format. Currently only support “NCDHW”.

Inputs:

• x (Tensor) - Tensor of shape \((N, C_{in}, D_{in}, H_{in}, W_{in})\). Currently input data type only support float16 and float32.

Outputs:

Tensor, the value that applied 3D convolution. The shape is \((N, C_{out}, D_{out}, H_{out}, W_{out})\).

Raises

• TypeError – If in_channels, out_channels or group is not an int.

• TypeError – If kernel_size, stride, padding or dilation is neither an int nor a tuple.

• ValueError – If out_channels, kernel_size, stride or dilation is less than 1.

• ValueError – If padding is less than 0.

• ValueError – If pad_mode is not one of ‘same’, ‘valid’, ‘pad’.

• ValueError – If padding is a tuple whose length is not equal to 6.

• ValueError – If pad_mode is not equal to ‘pad’ and padding is not equal to (0, 0, 0, 0, 0, 0).

• ValueError – If data_format is not ‘NCDHW’.

Supported Platforms:

Ascend GPU

Examples

>>> x = Tensor(np.ones([16, 3, 10, 32, 32]), mindspore.float32)
>>> conv3d = nn.Conv3d(in_channels=3, out_channels=32, kernel_size=(4, 3, 3))
>>> output = conv3d(x)
>>> print(output.shape)
(16, 32, 10, 32, 32)

class tinyms.layers.Conv3dTranspose(in_channels, out_channels, kernel_size, stride=1, pad_mode='same', padding=0, dilation=1, group=1, output_padding=0, has_bias=False, weight_init='normal', bias_init='zeros', data_format='NCDHW')

Compute a 3D transposed convolution, which is also known as a deconvolution (although it is not an actual deconvolution). The transposed convolution operator multiplies each input value element-wise by a learnable kernel, and sums over the outputs from all input feature planes. This module can be seen as the gradient of Conv3d with respect to its input.

x is typically of shape \((N, C, D, H, W)\), where \(N\) is batch size, \(C\) is channel number, \(D\) is the characteristic depth, \(H\) is the height of the characteristic layer, and \(W\) is the width of the characteristic layer. The calculation process of transposed convolution is equivalent to the reverse calculation of convolution.

The pad_mode argument effectively adds \(dilation * (kernel\_size - 1) - padding\) amount of zero padding to both sizes of the input. So that when a Conv3d and a ConvTranspose3d are initialized with same parameters, they are inverses of each other in regard to the input and output shapes. However, when stride > 1, Conv3d maps multiple input shapes to the same output shape. ConvTranspose3d provide padding argument to increase the calculated output shape on one or more side.

The height and width of output are defined as:

if the ‘pad_mode’ is set to be “pad”,

\[ \begin{align}\begin{aligned}D_{out} = (D_{in} - 1) \times \text{stride_d} - 2 \times \text{padding_d} + \text{dilation_d} \times (\text{kernel_size_d} - 1) + \text{output_padding_d} + 1\\H_{out} = (H_{in} - 1) \times \text{stride_h} - 2 \times \text{padding_h} + \text{dilation_h} \times (\text{kernel_size_h} - 1) + \text{output_padding_h} + 1\\W_{out} = (W_{in} - 1) \times \text{stride_w} - 2 \times \text{padding_w} + \text{dilation_w} \times (\text{kernel_size_w} - 1) + \text{output_padding_w} + 1\end{aligned}\end{align} \]

if the ‘pad_mode’ is set to be “same”,

\[\begin{split}D_{out} = (D_{in} + \text{stride_d} - 1)/\text{stride_d} \\ H_{out} = (H_{in} + \text{stride_h} - 1)/\text{stride_h} \\ W_{out} = (W_{in} + \text{stride_w} - 1)/\text{stride_w}\end{split}\]

if the ‘pad_mode’ is set to be “valid”,

\[\begin{split}D_{out} = (D_{in} - 1) \times \text{stride_d} + \text{dilation_d} \times (\text{kernel_size_d} - 1) + 1 \\ H_{out} = (H_{in} - 1) \times \text{stride_h} + \text{dilation_h} \times (\text{kernel_size_h} - 1) + 1 \\ W_{out} = (W_{in} - 1) \times \text{stride_w} + \text{dilation_w} \times (\text{kernel_size_w} - 1) + 1\end{split}\]

Parameters

• in_channels (int) – The number of input channel \(C_{in}\).

• out_channels (int) – The number of output channel \(C_{out}\).

• kernel_size (Union[int, tuple[int]]) – The kernel size of the 3D convolution.

• stride (Union[int, tuple[int]]) – The distance of kernel moving, an int number that represents the depth, height and width of movement are both strides, or a tuple of three int numbers that represent depth, height and width of movement respectively. Its value must be equal to or greater than 1. Default: 1.

• pad_mode (str) –

  Select the mode of the pad. The optional values are “pad”, “same”, “valid”. Default: “same”.

  • same: Adopts the way of completion. The depth, height and width of the output will be the same as the input x. The total number of padding will be calculated in depth, horizontal and vertical directions and evenly distributed to head and tail, top and bottom, left and right if possible. Otherwise, the last extra padding will be done from the tail, bottom and the right side. If this mode is set, padding and output_padding must be 0.

  • valid: Adopts the way of discarding. The possible largest depth, height and width of output will be returned without padding. Extra pixels will be discarded. If this mode is set, padding and output_padding must be 0.

  • pad: Implicit paddings on both sides of the input x in depth, height, width. The number of pad will be padded to the input Tensor borders. padding must be greater than or equal to 0.

• padding (Union(int, tuple[int])) – The pad value to be filled. Default: 0. If padding is an integer, the paddings of head, tail, top, bottom, left and right are the same, equal to padding. If padding is a tuple of six integers, the padding of head, tail, top, bottom, left and right equal to padding[0], padding[1], padding[2], padding[3], padding[4] and padding[5] correspondingly.

• dilation (Union(int, tuple[int])) – The data type is int or a tuple of 3 integers : math:(dilation_d, dilation_h, dilation_w). Currently, dilation on depth only supports the case of 1. Specifies the dilation rate to use for dilated convolution. If set to be \(k > 1\), there will be \(k - 1\) pixels skipped for each sampling location. Its value must be greater or equal to 1 and bounded by the height and width of the input x. Default: 1.

• group (int) – Splits filter into groups, in_ channels and out_channels must be divisible by the number of groups. Default: 1. Only 1 is currently supported.

• output_padding (Union(int, tuple[int])) – Add extra size to each dimension of the output. Default: 0. Must be greater than or equal to 0.

• has_bias (bool) – Specifies whether the layer uses a bias vector. Default: False.

• weight_init (Union[Tensor, str, Initializer, numbers.Number]) – Initializer for the convolution kernel. It can be a Tensor, a string, an Initializer or a number. When a string is specified, values from ‘TruncatedNormal’, ‘Normal’, ‘Uniform’, ‘HeUniform’ and ‘XavierUniform’ distributions as well as constant ‘One’ and ‘Zero’ distributions are possible. Alias ‘xavier_uniform’, ‘he_uniform’, ‘ones’ and ‘zeros’ are acceptable. Uppercase and lowercase are both acceptable. Refer to the values of Initializer for more details. Default: ‘normal’.

• bias_init (Union[Tensor, str, Initializer, numbers.Number]) – Initializer for the bias vector. Possible Initializer and string are the same as ‘weight_init’. Refer to the values of Initializer for more details. Default: ‘zeros’.

• data_format (str) – The optional value for data format. Currently only support ‘NCDHW’.

Inputs:

• x (Tensor) - Tensor of shape \((N, C_{in}, D_{in}, H_{in}, W_{in})\). Currently input data type only support float16 and float32.

Outputs:

Tensor, the shape is \((N, C_{out}, D_{out}, H_{out}, W_{out})\).

Supported Platforms:

Ascend GPU

Raises

• TypeError – If in_channels, out_channels or group is not an int.

• TypeError – If kernel_size, stride, padding , dilation or output_padding is neither an int not a tuple of three.

• TypeError – If input data type is not float16 or float32.

• ValueError – If in_channels, out_channels, kernel_size, stride or dilation is less than 1.

• ValueError – If padding is less than 0.

• ValueError – If pad_mode is not one of ‘same’, ‘valid’, ‘pad’.

• ValueError – If padding is a tuple whose length is not equal to 6.

• ValueError – If pad_mode is not equal to ‘pad’ and padding is not equal to (0, 0, 0, 0, 0, 0).

• ValueError – If data_format is not ‘NCDHW’.

Examples

>>> x = Tensor(np.ones([32, 16, 10, 32, 32]), mindspore.float32)
>>> conv3d_transpose = nn.Conv3dTranspose(in_channels=16, out_channels=3, kernel_size=(4, 6, 2),
...                                       pad_mode='pad')
>>> output = conv3d_transpose(x)
>>> print(output.shape)
(32, 3, 13, 37, 33)

class tinyms.layers.LSTM(input_size, hidden_size, num_layers=1, has_bias=True, batch_first=False, dropout=0, bidirectional=False)

Stacked LSTM (Long Short-Term Memory) layers.

Apply LSTM layer to the input.

There are two pipelines connecting two consecutive cells in a LSTM model; one is cell state pipeline and the other is hidden state pipeline. Denote two consecutive time nodes as \(t-1\) and \(t\). Given an input \(x_t\) at time \(t\), an hidden state \(h_{t-1}\) and an cell state \(c_{t-1}\) of the layer at time \({t-1}\), the cell state and hidden state at time \(t\) is computed using an gating mechanism. Input gate \(i_t\) is designed to protect the cell from perturbation by irrelevant inputs. Forget gate \(f_t\) affords protection of the cell by forgetting some information in the past, which is stored in \(h_{t-1}\). Output gate \(o_t\) protects other units from perturbation by currently irrelevant memory contents. Candidate cell state \(\tilde{c}_t\) is calculated with the current input, on which the input gate will be applied. Finally, current cell state \(c_{t}\) and hidden state \(h_{t}\) are computed with the calculated gates and cell states. The complete formulation is as follows.

\[\begin{split}\begin{array}{ll} \\ i_t = \sigma(W_{ix} x_t + b_{ix} + W_{ih} h_{(t-1)} + b_{ih}) \\ f_t = \sigma(W_{fx} x_t + b_{fx} + W_{fh} h_{(t-1)} + b_{fh}) \\ \tilde{c}_t = \tanh(W_{cx} x_t + b_{cx} + W_{ch} h_{(t-1)} + b_{ch}) \\ o_t = \sigma(W_{ox} x_t + b_{ox} + W_{oh} h_{(t-1)} + b_{oh}) \\ c_t = f_t * c_{(t-1)} + i_t * \tilde{c}_t \\ h_t = o_t * \tanh(c_t) \\ \end{array}\end{split}\]

Here \(\sigma\) is the sigmoid function, and \(*\) is the Hadamard product. \(W, b\) are learnable weights between the output and the input in the formula. For instance, \(W_{ix}, b_{ix}\) are the weight and bias used to transform from input \(x\) to \(i\). Details can be found in paper LONG SHORT-TERM MEMORY and Long Short-Term Memory Recurrent Neural Network Architectures for Large Scale Acoustic Modeling.

Parameters

• input_size (int) – Number of features of input.

• hidden_size (int) – Number of features of hidden layer.

• num_layers (int) – Number of layers of stacked LSTM . Default: 1.

• has_bias (bool) – Whether the cell has bias b_ih and b_hh. Default: True.

• batch_first (bool) – Specifies whether the first dimension of input x is batch_size. Default: False.

• dropout (float, int) – If not 0, append Dropout layer on the outputs of each LSTM layer except the last layer. Default 0. The range of dropout is [0.0, 1.0].

• bidirectional (bool) – Specifies whether it is a bidirectional LSTM. Default: False.

Inputs:

• x (Tensor) - Tensor of shape (seq_len, batch_size, input_size) or (batch_size, seq_len, input_size).

• hx (tuple) - A tuple of two Tensors (h_0, c_0) both of data type mindspore.float32 or mindspore.float16 and shape (num_directions * num_layers, batch_size, hidden_size). Data type of hx must be the same as x.

Outputs:

Tuple, a tuple contains (output, (h_n, c_n)).

• output (Tensor) - Tensor of shape (seq_len, batch_size, num_directions * hidden_size).

• hx_n (tuple) - A tuple of two Tensor (h_n, c_n) both of shape (num_directions * num_layers, batch_size, hidden_size).

Raises

• TypeError – If input_size, hidden_size or num_layers is not an int.

• TypeError – If has_bias, batch_first or bidirectional is not a bool.

• TypeError – If dropout is neither a float nor an int.

• ValueError – If dropout is not in range [0.0, 1.0].

Supported Platforms:

Ascend GPU

Examples

>>> net = nn.LSTM(10, 16, 2, has_bias=True, batch_first=True, bidirectional=False)
>>> x = Tensor(np.ones([3, 5, 10]).astype(np.float32))
>>> h0 = Tensor(np.ones([1 * 2, 3, 16]).astype(np.float32))
>>> c0 = Tensor(np.ones([1 * 2, 3, 16]).astype(np.float32))
>>> output, (hn, cn) = net(x, (h0, c0))
>>> print(output.shape)
(3, 5, 16)

class tinyms.layers.LSTMCell(input_size, hidden_size, has_bias=True, batch_first=False, dropout=0, bidirectional=False)

LSTM (Long Short-Term Memory) layer.

Apply LSTM layer to the input.

There are two pipelines connecting two consecutive cells in a LSTM model; one is cell state pipeline and the other is hidden state pipeline. Denote two consecutive time nodes as \(t-1\) and \(t\). Given an input \(x_t\) at time \(t\), an hidden state \(h_{t-1}\) and an cell state \(c_{t-1}\) of the layer at time \({t-1}\), the cell state and hidden state at time \(t\) is computed using an gating mechanism. Input gate \(i_t\) is designed to protect the cell from perturbation by irrelevant inputs. Forget gate \(f_t\) affords protection of the cell by forgetting some information in the past, which is stored in \(h_{t-1}\). Output gate \(o_t\) protects other units from perturbation by currently irrelevant memory contents. Candidate cell state \(\tilde{c}_t\) is calculated with the current input, on which the input gate will be applied. Finally, current cell state \(c_{t}\) and hidden state \(h_{t}\) are computed with the calculated gates and cell states. The complete formulation is as follows.

\[\begin{split}\begin{array}{ll} \\ i_t = \sigma(W_{ix} x_t + b_{ix} + W_{ih} h_{(t-1)} + b_{ih}) \\ f_t = \sigma(W_{fx} x_t + b_{fx} + W_{fh} h_{(t-1)} + b_{fh}) \\ \tilde{c}_t = \tanh(W_{cx} x_t + b_{cx} + W_{ch} h_{(t-1)} + b_{ch}) \\ o_t = \sigma(W_{ox} x_t + b_{ox} + W_{oh} h_{(t-1)} + b_{oh}) \\ c_t = f_t * c_{(t-1)} + i_t * \tilde{c}_t \\ h_t = o_t * \tanh(c_t) \\ \end{array}\end{split}\]

Here \(\sigma\) is the sigmoid function, and \(*\) is the Hadamard product. \(W, b\) are learnable weights between the output and the input in the formula. For instance, \(W_{ix}, b_{ix}\) are the weight and bias used to transform from input \(x\) to \(i\). Details can be found in paper LONG SHORT-TERM MEMORY and Long Short-Term Memory Recurrent Neural Network Architectures for Large Scale Acoustic Modeling.

Note

LSTMCell is a single-layer RNN, you can achieve multi-layer RNN by stacking LSTMCell.

Parameters

• input_size (int) – Number of features of input.

• hidden_size (int) – Number of features of hidden layer.

• has_bias (bool) – Whether the cell has bias b_ih and b_hh. Default: True.

• batch_first (bool) – Specifies whether the first dimension of input x is batch_size. Default: False.

• dropout (float, int) – If not 0, append Dropout layer on the outputs of each LSTM layer except the last layer. Default 0. The range of dropout is [0.0, 1.0].

• bidirectional (bool) – Specifies whether this is a bidirectional LSTM. If set True, number of directions will be 2 otherwise number of directions is 1. Default: False.

Inputs:

• x (Tensor) - Tensor of shape (seq_len, batch_size, input_size).

• h - data type mindspore.float32 or mindspore.float16 and shape (num_directions, batch_size, hidden_size).

• c - data type mindspore.float32 or mindspore.float16 and shape (num_directions, batch_size, hidden_size). Data type of h’ and ‘c’ must be the same of `x.

• w - data type mindspore.float32 or mindspore.float16 and shape (weight_size, 1, 1). The value of weight_size depends on input_size, hidden_size and bidirectional

Outputs:

output, h_n, c_n, ‘reserve’, ‘state’.

• output (Tensor) - Tensor of shape (seq_len, batch_size, num_directions * hidden_size).

• h - A Tensor with shape (num_directions, batch_size, hidden_size).

• c - A Tensor with shape (num_directions, batch_size, hidden_size).

• reserve - reserved

• state - reserved

Raises

• TypeError – If input_size or hidden_size or num_layers is not an int.

• TypeError – If has_bias or batch_first or bidirectional is not a bool.

• TypeError – If dropout is neither a float nor an int.

• ValueError – If dropout is not in range [0.0, 1.0].

Supported Platforms:

GPU CPU

Examples

>>> net = nn.LSTMCell(10, 12, has_bias=True, batch_first=True, bidirectional=False)
>>> x = Tensor(np.ones([3, 5, 10]).astype(np.float32))
>>> h = Tensor(np.ones([1, 3, 12]).astype(np.float32))
>>> c = Tensor(np.ones([1, 3, 12]).astype(np.float32))
>>> w = Tensor(np.ones([1152, 1, 1]).astype(np.float32))
>>> output, h, c, _, _ = net(x, h, c, w)
>>> print(output.shape)
(3, 5, 12)

class tinyms.layers.GRU(*args, **kwargs)

Stacked GRU (Gated Recurrent Unit) layers.

Apply GRU layer to the input.

There are two gates in a GRU model; one is update gate and the other is reset gate. Denote two consecutive time nodes as \(t-1\) and \(t\). Given an input \(x_t\) at time \(t\), an hidden state \(h_{t-1}\), the update and reset gate at time \(t\) is computed using an gating mechanism. Update gate \(z_t\) is designed to protect the cell from perturbation by irrelevant inputs and past hidden state. Reset gate \(r_t\) determines how much information should be reset from old hidden state. New memory state \({n}_t\) is calculated with the current input, on which the reset gate will be applied. Finally, current hidden state \(h_{t}\) is computed with the calculated update grate and new memory state. The complete formulation is as follows.

\[\begin{split}\begin{array}{ll} r_t = \sigma(W_{ir} x_t + b_{ir} + W_{hr} h_{(t-1)} + b_{hr}) \\ z_t = \sigma(W_{iz} x_t + b_{iz} + W_{hz} h_{(t-1)} + b_{hz}) \\ n_t = \tanh(W_{in} x_t + b_{in} + r_t * (W_{hn} h_{(t-1)}+ b_{hn})) \\ h_t = (1 - z_t) * n_t + z_t * h_{(t-1)} \end{array}\end{split}\]

Here \(\sigma\) is the sigmoid function, and \(*\) is the Hadamard product. \(W, b\) are learnable weights between the output and the input in the formula. For instance, \(W_{ir}, b_{ir}\) are the weight and bias used to transform from input \(x\) to \(r\). Details can be found in paper Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation.

Parameters

• input_size (int) – Number of features of input.

• hidden_size (int) – Number of features of hidden layer.

• num_layers (int) – Number of layers of stacked GRU. Default: 1.

• has_bias (bool) – Whether the cell has bias b_ih and b_hh. Default: True.

• batch_first (bool) – Specifies whether the first dimension of input x is batch_size. Default: False.

• dropout (float) – If not 0.0, append Dropout layer on the outputs of each GRU layer except the last layer. Default 0.0. The range of dropout is [0.0, 1.0).

• bidirectional (bool) – Specifies whether it is a bidirectional GRU, num_directions=2 if bidirectional=True otherwise 1. Default: False.

Inputs:

• x (Tensor) - Tensor of data type mindspore.float32 and shape (seq_len, batch_size, input_size) or (batch_size, seq_len, input_size).

• hx (Tensor) - Tensor of data type mindspore.float32 and shape (num_directions * num_layers, batch_size, hidden_size). Data type of hx must be the same as x.

• seq_length (Tensor) - The length of each sequence in a input batch. Tensor of shape \((\text{batch_size})\). Default: None. This input indicates the real sequence length before padding to avoid padded elements have been used to compute hidden state and affect the final output. It is recommend to use this input when x has padding elements.

Outputs:

Tuple, a tuple contains (output, h_n).

• output (Tensor) - Tensor of shape (seq_len, batch_size, num_directions * hidden_size) or (batch_size, seq_len, num_directions * hidden_size).

• hx_n (Tensor) - Tensor of shape (num_directions * num_layers, batch_size, hidden_size).

Raises

• TypeError – If input_size, hidden_size or num_layers is not an int.

• TypeError – If has_bias, batch_first or bidirectional is not a bool.

• TypeError – If dropout is neither a float nor an int.

• ValueError – If dropout is not in range [0.0, 1.0).

Supported Platforms:

Ascend GPU

Examples

>>> net = nn.GRU(10, 16, 2, has_bias=True, batch_first=True, bidirectional=False)
>>> x = Tensor(np.ones([3, 5, 10]).astype(np.float32))
>>> h0 = Tensor(np.ones([1 * 2, 3, 16]).astype(np.float32))
>>> output, hn = net(x, h0)
>>> print(output.shape)
(3, 5, 16)

class tinyms.layers.RNN(*args, **kwargs)

Stacked Elman RNN layers.

Apply RNN layer with \(\tanh\) or \(\text{ReLU}\) non-linearity to the input.

For each element in the input sequence, each layer computes the following function:

\[h_t = \tanh(W_{ih} x_t + b_{ih} + W_{hh} h_{(t-1)} + b_{hh})\]

Here \(h_t\) is the hidden state at time t, \(x_t\) is the input at time t, and \(h_{(t-1)}\) is the hidden state of the previous layer at time t-1 or the initial hidden state at time 0. If nonlinearity is 'relu', then \(\text{ReLU}\) is used instead of \(\tanh\).

Parameters

• input_size (int) – Number of features of input.

• hidden_size (int) – Number of features of hidden layer.

• num_layers (int) – Number of layers of stacked RNN. Default: 1.

• nonlinearity (str) – The non-linearity to use. Can be either 'tanh' or 'relu'. Default: 'tanh'

• has_bias (bool) – Whether the cell has bias b_ih and b_hh. Default: True.

• batch_first (bool) – Specifies whether the first dimension of input x is batch_size. Default: False.

• dropout (float) – If not 0.0, append Dropout layer on the outputs of each RNN layer except the last layer. Default 0.0. The range of dropout is [0.0, 1.0).

• bidirectional (bool) – Specifies whether it is a bidirectional RNN, num_directions=2 if bidirectional=True otherwise 1. Default: False.

Inputs:

• x (Tensor) - Tensor of data type mindspore.float32 and shape (seq_len, batch_size, input_size) or (batch_size, seq_len, input_size).

• hx (Tensor) - Tensor of data type mindspore.float32 and shape (num_directions * num_layers, batch_size, hidden_size). Data type of hx must be the same as x.

• seq_length (Tensor) - The length of each sequence in a input batch. Tensor of shape \((\text{batch_size})\). Default: None. This input indicates the real sequence length before padding to avoid padded elements have been used to compute hidden state and affect the final output. It is recommend to use this input when x has padding elements.

Outputs:

Tuple, a tuple contains (output, h_n).

• output (Tensor) - Tensor of shape (seq_len, batch_size, num_directions * hidden_size) or (batch_size, seq_len, num_directions * hidden_size).

• hx_n (Tensor) - Tensor of shape (num_directions * num_layers, batch_size, hidden_size).

Raises

• TypeError – If input_size, hidden_size or num_layers is not an int.

• TypeError – If has_bias, batch_first or bidirectional is not a bool.

• TypeError – If dropout is neither a float nor an int.

• ValueError – If dropout is not in range [0.0, 1.0).

• ValueError – If nonlinearity is not in [‘tanh’, ‘relu’].

Supported Platforms:

Ascend GPU

Examples

>>> net = nn.RNN(10, 16, 2, has_bias=True, batch_first=True, bidirectional=False)
>>> x = Tensor(np.ones([3, 5, 10]).astype(np.float32))
>>> h0 = Tensor(np.ones([1 * 2, 3, 16]).astype(np.float32))
>>> output, hn = net(x, h0)
>>> print(output.shape)
(3, 5, 16)

class tinyms.layers.GRUCell(input_size: int, hidden_size: int, has_bias: bool = True)

A GRU(Gated Recurrent Unit) cell.

\[\begin{split}\begin{array}{ll} r = \sigma(W_{ir} x + b_{ir} + W_{hr} h + b_{hr}) \\ z = \sigma(W_{iz} x + b_{iz} + W_{hz} h + b_{hz}) \\ n = \tanh(W_{in} x + b_{in} + r * (W_{hn} h + b_{hn})) \\ h' = (1 - z) * n + z * h \end{array}\end{split}\]

Here \(\sigma\) is the sigmoid function, and \(*\) is the Hadamard product. \(W, b\) are learnable weights between the output and the input in the formula. For instance, \(W_{ir}, b_{ir}\) are the weight and bias used to transform from input \(x\) to \(r\). Details can be found in paper Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation.

Parameters

• input_size (int) – Number of features of input.

• hidden_size (int) – Number of features of hidden layer.

• has_bias (bool) – Whether the cell has bias b_ih and b_hh. Default: True.

Inputs:

• x (Tensor) - Tensor of shape (batch_size, input_size).

• hx (Tensor) - Tensor of data type mindspore.float32 and shape (batch_size, hidden_size). Data type of hx must be the same as x.

Outputs:

• h’ (Tensor) - Tensor of shape (batch_size, hidden_size).

Raises

• TypeError – If input_size, hidden_size is not an int.

• TypeError – If has_bias is not a bool.

Supported Platforms:

Ascend GPU

Examples

>>> net = nn.GRUCell(10, 16)
>>> x = Tensor(np.ones([5, 3, 10]).astype(np.float32))
>>> hx = Tensor(np.ones([3, 16]).astype(np.float32))
>>> output = []
>>> for i in range(5):
>>>     hx = net(x[i], hx)
>>>     output.append(hx)
>>> print(output[0].shape)
(3, 16)

class tinyms.layers.RNNCell(input_size: int, hidden_size: int, has_bias: bool = True, nonlinearity: str = 'tanh')

An Elman RNN cell with tanh or ReLU non-linearity.

\[h_t = \tanh(W_{ih} x_t + b_{ih} + W_{hh} h_{(t-1)} + b_{hh})\]

Here \(h_t\) is the hidden state at time t, \(x_t\) is the input at time t, and \(h_{(t-1)}\) is the hidden state of the previous layer at time t-1 or the initial hidden state at time 0. If nonlinearity is relu, then relu is used instead of tanh.

Parameters

• input_size (int) – Number of features of input.

• hidden_size (int) – Number of features of hidden layer.

• has_bias (bool) – Whether the cell has bias b_ih and b_hh. Default: True.

• nonlinearity (str) – The non-linearity to use. Can be either tanh or relu. Default: tanh.

Inputs:

• x (Tensor) - Tensor of shape (batch_size, input_size).

• hx (Tensor) - Tensor of data type mindspore.float32 and shape (batch_size, hidden_size). Data type of hx must be the same as x.

Outputs:

• h’ (Tensor) - Tensor of shape (batch_size, hidden_size).

Raises

• TypeError – If input_size or hidden_size is not an int or not greater than 0.

• TypeError – If has_bias is not a bool.

• ValueError – If nonlinearity is not in [‘tanh’, ‘relu’].

Supported Platforms:

Ascend GPU

Examples

>>> net = nn.RNNCell(10, 16)
>>> x = Tensor(np.ones([5, 3, 10]).astype(np.float32))
>>> hx = Tensor(np.ones([3, 16]).astype(np.float32))
>>> output = []
>>> for i in range(5):
>>>     hx = net(x[i], hx)
>>>     output.append(hx)
>>> print(output[0].shape)
(3, 16)

class tinyms.layers.Dropout(keep_prob=0.5, dtype=mindspore.float32)

Dropout layer for the input.

Randomly set some elements of the input tensor to zero with probability \(1 - keep\_prob\) during training using samples from a Bernoulli distribution.

The outputs are scaled by a factor of \(\frac{1}{keep\_prob}\) during training so that the output layer remains at a similar scale. During inference, this layer returns the same tensor as the x.

This technique is proposed in paper Dropout: A Simple Way to Prevent Neural Networks from Overfitting and proved to be effective to reduce over-fitting and prevents neurons from co-adaptation. See more details in Improving neural networks by preventing co-adaptation of feature detectors.

Note

Each channel will be zeroed out independently on every construct call.

Parameters

• keep_prob (float) – The keep rate, greater than 0 and less equal than 1. E.g. rate=0.9, dropping out 10% of input units. Default: 0.5.

• dtype (mindspore.dtype) – Data type of x. Default: mindspore.float32.

Inputs:

• x (Tensor) - The input of Dropout with data type of float16 or float32. The shape is \((N,*)\) where \(*\) means, any number of additional dimensions.

Outputs:

Tensor, output tensor with the same shape as the x.

Raises

• TypeError – If keep_prob is not a float.

• TypeError – If dtype of x is not neither float16 nor float32.

• ValueError – If keep_prob is not in range (0, 1].

• ValueError – If length of shape of x is less than 1.

Supported Platforms:

Ascend GPU CPU

Examples

>>> x = Tensor(np.ones([2, 2, 3]), mindspore.float32)
>>> net = nn.Dropout(keep_prob=0.8)
>>> net.set_train()
Dropout<keep_prob=0.8>
>>> output = net(x)
>>> print(output.shape)
(2, 2, 3)

class tinyms.layers.Flatten

Flatten layer for the input.

Flattens a tensor without changing dimension of batch size on the 0-th axis.

Inputs:

• x (Tensor) - Tensor of shape \((N, \ldots)\) to be flattened. The data type is Number. The shape is \((N,*)\) where \(*\) means, any number of additional dimensions and the shape can’t be ().

Outputs:

Tensor, the shape of the output tensor is \((N, X)\), where \(X\) is the product of the remaining dimensions.

Raises

TypeError – If x is not a subclass of Tensor.

Supported Platforms:

Ascend GPU CPU

Examples

>>> x = Tensor(np.array([[[1.2, 1.2], [2.1, 2.1]], [[2.2, 2.2], [3.2, 3.2]]]), mindspore.float32)
>>> net = nn.Flatten()
>>> output = net(x)
>>> print(output)
[[1.2 1.2 2.1 2.1]
 [2.2 2.2 3.2 3.2]]
>>> print(f"before flatten the x shape is {x.shape}")
before flatten the x shape is  (2, 2, 2)
>>> print(f"after flatten the output shape is {output.shape}")
after flatten the output shape is (2, 4)

class tinyms.layers.Dense(in_channels, out_channels, weight_init='normal', bias_init='zeros', has_bias=True, activation=None)

The dense connected layer.

Applies dense connected layer for the input. This layer implements the operation as:

\[\text{outputs} = \text{activation}(\text{X} * \text{kernel} + \text{bias}),\]

where \(X\) is the input tensors, \(\text{activation}\) is the activation function passed as the activation argument (if passed in), \(\text{kernel}\) is a weight matrix with the same data type as the \(X\) created by the layer, and \(\text{bias}\) is a bias vector with the same data type as the \(X\) created by the layer (only if has_bias is True).

Parameters

• in_channels (int) – The number of channels in the input space.

• out_channels (int) – The number of channels in the output space.

• weight_init (Union[Tensor, str, Initializer, numbers.Number]) – The trainable weight_init parameter. The dtype is same as x. The values of str refer to the function initializer. Default: ‘normal’.

• bias_init (Union[Tensor, str, Initializer, numbers.Number]) – The trainable bias_init parameter. The dtype is same as x. The values of str refer to the function initializer. Default: ‘zeros’.

• has_bias (bool) – Specifies whether the layer uses a bias vector. Default: True.

• activation (Union[str, Cell, Primitive]) – activate function applied to the output of the fully connected layer, eg. ‘ReLU’.Default: None.

Inputs:

• x (Tensor) - Tensor of shape \((*, in\_channels)\). The in_channels in Args should be equal to \(in\_channels\) in Inputs.

Outputs:

Tensor of shape \((*, out\_channels)\).

Raises

• TypeError – If in_channels or out_channels is not an int.

• TypeError – If has_bias is not a bool.

• TypeError – If activation is not one of str, Cell, Primitive, None.

• ValueError – If length of shape of weight_init is not equal to 2 or shape[0] of weight_init is not equal to out_channels or shape[1] of weight_init is not equal to in_channels.

• ValueError – If length of shape of bias_init is not equal to 1 or shape[0] of bias_init is not equal to out_channels.

Supported Platforms:

Ascend GPU CPU

Examples

>>> x = Tensor(np.array([[180, 234, 154], [244, 48, 247]]), mindspore.float32)
>>> net = nn.Dense(3, 4)
>>> output = net(x)
>>> print(output.shape)
(2, 4)

class tinyms.layers.ClipByNorm(axis=None)

Clips tensor values to a maximum \(L_2\)-norm.

The output of this layer remains the same if the \(L_2\)-norm of the input tensor is not greater than the argument clip_norm. Otherwise the tensor will be normalized as:

\[\text{output}(X) = \frac{\text{clip_norm} * X}{L_2(X)},\]

where \(L_2(X)\) is the \(L_2\)-norm of \(X\).

Parameters

axis (Union[None, int, tuple(int)]) – Compute the L2-norm along the Specific dimension. Default: None, all dimensions to calculate.

Inputs:

• x (Tensor) - Tensor of shape N-D. The type must be float32 or float16.

• clip_norm (Tensor) - A scalar Tensor of shape \(()\) or \((1)\). Or a tensor shape can be broadcast to input shape.

Outputs:

Tensor, clipped tensor with the same shape as the x, whose type is float32.

Raises

• TypeError – If axis is not one of None, int, tuple.

• TypeError – If dtype of x is neither float32 nor float16.

Supported Platforms:

Ascend GPU CPU

Examples

>>> net = nn.ClipByNorm()
>>> x = Tensor(np.random.randint(0, 10, [4, 16]), mindspore.float32)
>>> clip_norm = Tensor(np.array([100]).astype(np.float32))
>>> output = net(x, clip_norm)
>>> print(output.shape)
(4, 16)

class tinyms.layers.Norm(axis=(), keep_dims=False)

Computes the norm of vectors, currently including Euclidean norm, i.e., \(L_2\)-norm.

\[norm(x) = \sqrt{\sum_{i=1}^{n} (x_i^2)}\]

Parameters

• axis (Union[tuple, int]) – The axis over which to compute vector norms. Default: ().

• keep_dims (bool) – If true, the axis indicated in axis are kept with size 1. Otherwise, the dimensions in axis are removed from the output shape. Default: False.

Inputs:

• x (Tensor) - Tensor which is not empty. The data type should be float16 or float32. \((N,*)\) where \(*\) means, any number of additional dimensions.

Outputs:

Tensor, output tensor with dimensions in ‘axis’ reduced to 1 will be returned if ‘keep_dims’ is True; otherwise a Tensor with dimensions in ‘axis’ removed is returned. The data type is the same with x.

Raises

• TypeError – If axis is neither an int nor a tuple.

• TypeError – If keep_dims is not a bool.

Supported Platforms:

Ascend GPU CPU

Examples

>>> net = nn.Norm(axis=0)
>>> x = Tensor(np.array([[4, 4, 9, 1], [2, 1, 3, 6]]), mindspore.float32)
>>> print(x.shape)
(2, 4)
>>> output = net(x)
>>> print(output)
[4.472136 4.1231055 9.486833 6.0827627]
>>> print(output.shape)
(4,)
>>> net = nn.Norm(axis=0, keep_dims=True)
>>> x = Tensor(np.array([[4, 4, 9, 1], [2, 1, 3, 6]]), mindspore.float32)
>>> print(x.shape)
(2, 4)
>>> output = net(x)
>>> print(output)
[4.472136 4.1231055 9.486833 6.0827627]
>>> print(output.shape)
(1, 4)
>>> net = nn.Norm(axis=1)
>>> x = Tensor(np.array([[4, 4, 9, 1], [2, 1, 3, 6]]), mindspore.float32)
>>> print(x.shape)
(2, 4)
>>> output = net(x)
>>> print(output)
[10.677078 7.071068]
>>> print(output.shape)
(2,)

class tinyms.layers.OneHot(axis=-1, depth=1, on_value=1.0, off_value=0.0, dtype=mindspore.float32)

Returns a one-hot tensor.

The locations represented by indices in argument indices take value on_value, while all other locations take value off_value.

Note

If the input indices is rank \(N\), the output will have rank \(N+1\). The new axis is created at dimension axis.

If indices is a scalar, the output shape will be a vector of length depth.

If indices is a vector of length features, the output shape will be:

features * depth if axis == -1

depth * features if axis == 0

If indices is a matrix with shape [batch, features], the output shape will be:

batch * features * depth if axis == -1

batch * depth * features if axis == 1

depth * batch * features if axis == 0

Parameters

• axis (int) – Features x depth if axis is -1, depth x features if axis is 0. Default: -1.

• depth (int) – A scalar defining the depth of the one hot dimension. Default: 1.

• on_value (float) – A scalar defining the value to fill in output[i][j] when indices[j] = i. Default: 1.0.

• off_value (float) – A scalar defining the value to fill in output[i][j] when indices[j] != i. Default: 0.0.

• dtype (mindspore.dtype) – Data type of ‘on_value’ and ‘off_value’, not the data type of indices. Default: mindspore.float32.

Inputs:

• indices (Tensor) - A tensor of indices with data type of int32 or int64. The shape is \((N,*)\) where \(*\) means, any number of additional dimensions.

Outputs:

Tensor, the one-hot tensor of data type dtype with dimension at axis expanded to depth and filled with on_value and off_value. The dimension of the Outputs is equal to the dimension of the indices plus one.

Raises

• TypeError – If axis or depth is not an int.

• TypeError – If dtype of indices is neither int32 nor int64.

• ValueError – If axis is not in range [-1, len(indices_shape)].

• ValueError – If depth is less than 0.

Supported Platforms:

Ascend GPU CPU

Examples

>>> # 1st sample: add new coordinates at axis 1
>>> net = nn.OneHot(depth=4, axis=1)
>>> indices = Tensor([[1, 3], [0, 2]], dtype=mindspore.int32)
>>> output = net(indices)
>>> print(output)
[[[0. 0.]
  [1. 0.]
  [0. 0.]
  [0. 1.]]
 [[1. 0.]
  [0. 0.]
  [0. 1.]
  [0. 0.]]]
>>> # The results are shown below:
>>> print(output.shape)
(2, 4, 2)
>>> # 2nd sample: add new coordinates at axis 0
>>> net = nn.OneHot(depth=4, axis=0)
>>> indices = Tensor([[1, 3], [0, 2]], dtype=mindspore.int32)
>>> output = net(indices)
>>> print(output)
[[[0. 0.]
  [1. 0.]]
 [[1. 0.]
  [0. 0.]]
 [[0. 0.]
  [0. 1.]]
 [[0. 1.]
  [0. 0.]]]
>>> # The results are shown below:
>>> print(output.shape)
(4, 2, 2)
>>> # 3rd sample: add new coordinates at the last dimension.
>>> net = nn.OneHot(depth=4, axis=-1)
>>> indices = Tensor([[1, 3], [0, 2]], dtype=mindspore.int32)
>>> output = net(indices)
>>> # The results are shown below:
>>> print(output)
[[[0. 1. 0. 0.]
  [0. 0. 0. 1.]]
 [[1. 0. 0. 0.]
  [0. 0. 1. 0.]]]
>>> print(output.shape)
(2, 2, 4)
>>> indices = Tensor([1, 3, 0, 2], dtype=mindspore.int32)
>>> output = net(indices)
>>> print(output)
[[0. 1. 0. 0.]
 [0. 0. 0. 1.]
 [1. 0. 0. 0.]
 [0. 0. 1. 0.]]
>>> print(output.shape)
(4, 4)

class tinyms.layers.Pad(paddings, mode='CONSTANT')

Pads the input tensor according to the paddings and mode.

Parameters

• paddings (tuple) –

  The shape of parameter paddings is (N, 2). N is the rank of input data. All elements of paddings are int type. For D th dimension of the x, paddings[D, 0] indicates how many sizes to be extended ahead of the D th dimension of the input tensor, and paddings[D, 1] indicates how many sizes to be extended behind of the D th dimension of the input tensor. The padded size of each dimension D of the output is: \(paddings[D, 0] + input\_x.dim\_size(D) + paddings[D, 1]\), e.g.:

  mode = "CONSTANT".
  paddings = [[1,1], [2,2]].
  x = [[1,2,3], [4,5,6], [7,8,9]].
  # The above can be seen: 1st dimension of `x` is 3, 2nd dimension of `x` is 3.
  # Substitute into the formula to get:
  # 1st dimension of output is paddings[0][0] + 3 + paddings[0][1] = 1 + 3 + 1 = 5.
  # 2nd dimension of output is paddings[1][0] + 3 + paddings[1][1] = 2 + 3 + 2 = 7.
  # So the shape of output is (5, 7).
  

• mode (str) – Specifies padding mode. The optional values are “CONSTANT”, “REFLECT”, “SYMMETRIC”. Default: “CONSTANT”.

Inputs:

• x (Tensor) - The input tensor.

Outputs:

Tensor, the tensor after padding.

• If mode is “CONSTANT”, it fills the edge with 0, regardless of the values of the x. If the x is [[1,2,3], [4,5,6], [7,8,9]] and paddings is [[1,1], [2,2]], then the Outputs is [[0,0,0,0,0,0,0], [0,0,1,2,3,0,0], [0,0,4,5,6,0,0], [0,0,7,8,9,0,0], [0,0,0,0,0,0,0]].

• If mode is “REFLECT”, it uses a way of symmetrical copying through the axis of symmetry to fill in. If the x is [[1,2,3], [4,5,6], [7,8,9]] and paddings is [[1,1], [2,2]], then the Outputs is [[6,5,4,5,6,5,4], [3,2,1,2,3,2,1], [6,5,4,5,6,5,4], [9,8,7,8,9,8,7], [6,5,4,5,6,5,4]].

• If mode is “SYMMETRIC”, the filling method is similar to the “REFLECT”. It is also copied according to the symmetry axis, except that it includes the symmetry axis. If the x is [[1,2,3], [4,5,6], [7,8,9]] and paddings is [[1,1], [2,2]], then the Outputs is [[2,1,1,2,3,3,2], [2,1,1,2,3,3,2], [5,4,4,5,6,6,5], [8,7,7,8,9,9,8], [8,7,7,8,9,9,8]].

Raises

• TypeError – If paddings is not a tuple.

• ValueError – If length of paddings is more than 4 or its shape is not (n, 2).

• ValueError – If mode is not one of ‘CONSTANT’, ‘REFLECT’, ‘SYMMETRIC’.

Supported Platforms:

Ascend GPU CPU

Examples

>>> from mindspore import Tensor
>>> import mindspore.nn as nn
>>> import numpy as np
>>> # If `mode` is "CONSTANT"
>>> class Net(nn.Cell):
...     def __init__(self):
...         super(Net, self).__init__()
...         self.pad = nn.Pad(paddings=((1, 1), (2, 2)), mode="CONSTANT")
...     def construct(self, x):
...         return self.pad(x)
>>> x = Tensor(np.array([[1, 2, 3], [4, 5, 6]]), mindspore.float32)
>>> pad = Net()
>>> output = pad(x)
>>> print(output)
[[0. 0. 0. 0. 0. 0. 0.]
 [0. 0. 1. 2. 3. 0. 0.]
 [0. 0. 4. 5. 6. 0. 0.]
 [0. 0. 0. 0. 0. 0. 0.]]
>>> # Another way to call
>>> pad = ops.Pad(paddings=((1, 1), (2, 2)))
>>> # From the above code, we can see following:
>>> # "paddings=((1, 1), (2, 2))",
>>> # paddings[0][0] = 1, indicates a row of values is filled top of the input data in the 1st dimension.
>>> # Shown as follows:
>>> # [[0. 0. 0.]
>>> #  [1. 2. 3.]
>>> #  [4. 5. 6.]]
>>> # paddings[0][1] = 1 indicates a row of values is filled below input data in the 1st dimension.
>>> # Shown as follows:
>>> # [[0. 0. 0.]
>>> #  [1. 2. 3.]
>>> #  [4. 5. 6.]
>>> #  [0. 0. 0.]]
>>> # paddings[1][0] = 2, indicates 2 rows of values is filled in front of input data in the 2nd dimension.
>>> # Shown as follows:
>>> # [[0. 0. 0. 0. 0.]
>>> #  [0. 0. 1. 2. 3.]
>>> #  [0. 0. 4. 5. 6.]
>>> #  [0. 0. 0. 0. 0.]]
>>> # paddings[1][1] = 2, indicates 2 rows of values is filled in front of input data in the 2nd dimension.
>>> # Shown as follows:
>>> # [[0. 0. 0. 0. 0. 0. 0.]
>>> #  [0. 0. 1. 2. 3. 0. 0.]
>>> #  [0. 0. 4. 5. 6. 0. 0.]
>>> #  [0. 0. 0. 0. 0. 0. 0.]]
>>> output = pad(x)
>>> print(output)
[[0. 0. 0. 0. 0. 0. 0.]
 [0. 0. 1. 2. 3. 0. 0.]
 [0. 0. 4. 5. 6. 0. 0.]
 [0. 0. 0. 0. 0. 0. 0.]]
>>> # if mode is "REFLECT"
>>> class Net(nn.Cell):
...     def __init__(self):
...         super(Net, self).__init__()
...         self.pad = nn.Pad(paddings=((1, 1), (2, 2)), mode="REFLECT")
...     def construct(self, x):
...         return self.pad(x)
>>> x = Tensor(np.array([[1, 2, 3], [4, 5, 6]]), mindspore.float32)
>>> pad = Net()
>>> output = pad(x)
>>> print(output)
[[6. 5. 4. 5. 6. 5. 4.]
 [3. 2. 1. 2. 3. 2. 1.]
 [6. 5. 4. 5. 6. 5. 4.]
 [3. 2. 1. 2. 3. 2. 1.]]
>>> # if mode is "SYMMETRIC"
>>> class Net(nn.Cell):
...     def __init__(self):
...         super(Net, self).__init__()
...         self.pad = nn.Pad(paddings=((1, 1), (2, 2)), mode="SYMMETRIC")
...     def construct(self, x):
...         return self.pad(x)
>>> x = Tensor(np.array([[1, 2, 3], [4, 5, 6]]), mindspore.float32)
>>> pad = Net()
>>> output = pad(x)
>>> print(output)
[[2. 1. 1. 2. 3. 3. 2.]
 [2. 1. 1. 2. 3. 3. 2.]
 [5. 4. 4. 5. 6. 6. 5.]
 [5. 4. 4. 5. 6. 6. 5.]]

class tinyms.layers.Unfold(ksizes, strides, rates, padding='valid')

Extracts patches from images. The input tensor must be a 4-D tensor and the data format is NCHW.

Parameters

• ksizes (Union[tuple[int], list[int]]) – The size of sliding window, must be a tuple or a list of integers, and the format is [1, ksize_row, ksize_col, 1].

• strides (Union[tuple[int], list[int]]) – Distance between the centers of the two consecutive patches, must be a tuple or list of int, and the format is [1, stride_row, stride_col, 1].

• rates (Union[tuple[int], list[int]]) – In each extracted patch, the gap between the corresponding dimension pixel positions, must be a tuple or a list of integers, and the format is [1, rate_row, rate_col, 1].

• padding (str) –

  The type of padding algorithm, is a string whose value is “same” or “valid”, not case sensitive. Default: “valid”.

  • same: Means that the patch can take the part beyond the original image, and this part is filled with 0.

  • valid: Means that the taken patch area must be completely covered in the original image.

Inputs:

• x (Tensor) - A 4-D tensor whose shape is [in_batch, in_depth, in_row, in_col] and data type is number.

Outputs:

Tensor, a 4-D tensor whose data type is same as x, and the shape is [out_batch, out_depth, out_row, out_col] where out_batch is the same as the in_batch.

\(out\_depth = ksize\_row * ksize\_col * in\_depth\)

\(out\_row = (in\_row - (ksize\_row + (ksize\_row - 1) * (rate\_row - 1))) // stride\_row + 1\)

\(out\_col = (in\_col - (ksize\_col + (ksize\_col - 1) * (rate\_col - 1))) // stride\_col + 1\)

Raises

• TypeError – If ksizes, strides or rates is neither a tuple nor list.

• ValueError – If shape of ksizes, strides or rates is not (1, x_row, x_col, 1).

• ValueError – If the second and third element of ksizes, strides or rates is less than 1.

Supported Platforms:

Ascend

Examples

>>> net = Unfold(ksizes=[1, 2, 2, 1], strides=[1, 2, 2, 1], rates=[1, 2, 2, 1])
>>> # As stated in the above code:
>>> # ksize_row = 2, ksize_col = 2, rate_row = 2, rate_col = 2, stride_row = 2, stride_col = 2.
>>> image = Tensor(np.ones([2, 3, 6, 6]), dtype=mstype.float16)
>>> # in_batch = 2, in_depth = 3, in_row = 6, in_col = 6.
>>> # Substituting the formula to get:
>>> # out_batch = in_batch = 2
>>> # out_depth = 2 * 2 * 3 = 12
>>> # out_row = (6 - (2 + (2 - 1) * (2 - 1))) // 2 + 1 = 2
>>> # out_col = (6 - (2 + (2 - 1) * (2 - 1))) // 2 + 1 = 2
>>> output = net(image)
>>> print(output.shape)
(2, 12, 2, 2)

class tinyms.layers.Tril

Returns a tensor with elements above the kth diagonal zeroed.

Inputs:

• x (Tensor) - The input tensor. The data type is Number. \((N,*)\) where \(*\) means, any number of additional dimensions.

• k (Int) - The index of diagonal. Default: 0

Outputs:

Tensor, has the same shape and type as input x.

Raises

• TypeError – If k is not an int.

• ValueError – If length of shape of x is less than 1.

Supported Platforms:

Ascend GPU CPU

Examples

>>> x = Tensor(np.array([[ 1,  2,  3,  4],
...                      [ 5,  6,  7,  8],
...                      [10, 11, 12, 13],
...                      [14, 15, 16, 17]]))
>>> tril = nn.Tril()
>>> result = tril(x)
>>> print(result)
[[ 1  0  0  0]
 [ 5  6  0  0]
 [10 11 12  0]
 [14 15 16 17]]
>>> x = Tensor(np.array([[ 1,  2,  3,  4],
...                      [ 5,  6,  7,  8],
...                      [10, 11, 12, 13],
...                      [14, 15, 16, 17]]))
>>> tril = nn.Tril()
>>> result = tril(x, 1)
>>> print(result)
[[ 1  2  0  0]
 [ 5  6  7  0]
 [10 11 12 13]
 [14 15 16 17]]
>>> x = Tensor(np.array([[ 1,  2,  3,  4],
...                      [ 5,  6,  7,  8],
...                      [10, 11, 12, 13],
...                      [14, 15, 16, 17]]))
>>> tril = nn.Tril()
>>> result = tril(x, 2)
>>> print(result)
[[ 1  2  3  0]
 [ 5  6  7  8]
 [10 11 12 13]
 [14 15 16 17]]
>>> x = Tensor(np.array([[ 1,  2,  3,  4],
...                      [ 5,  6,  7,  8],
...                      [10, 11, 12, 13],
...                      [14, 15, 16, 17]]))
>>> tril = nn.Tril()
>>> result = tril(x, -1)
>>> print(result)
[[ 0  0  0  0]
 [ 5  0  0  0]
 [10 11  0  0]
 [14 15 16  0]]

class tinyms.layers.Triu

Returns a tensor with elements below the kth diagonal zeroed.

Inputs:

• x (Tensor) - The input tensor. The data type is Number. \((N,*)\) where \(*\) means, any number of additional dimensions.

• k (Int) - The index of diagonal. Default: 0

Outputs:

Tensor, has the same type and shape as input x.

Raises

• TypeError – If k is not an int.

• ValueError – If length of shape of x is less than 1.

Supported Platforms:

Ascend GPU CPU

Examples

>>> x = Tensor(np.array([[ 1,  2,  3,  4],
...                      [ 5,  6,  7,  8],
...                      [10, 11, 12, 13],
...                      [14, 15, 16, 17]]))
>>> triu = nn.Triu()
>>> result = triu(x)
>>> print(result)
[[ 1  2  3  4]
 [ 0  6  7  8]
 [ 0  0 12 13]
 [ 0  0  0 17]]
>>> x = Tensor(np.array([[ 1,  2,  3,  4],
...                      [ 5,  6,  7,  8],
...                      [10, 11, 12, 13],
...                      [14, 15, 16, 17]]))
>>> triu = nn.Triu()
>>> result = triu(x, 1)
>>> print(result)
[[ 0  2  3  4]
 [ 0  0  7  8]
 [ 0  0  0 13]
 [ 0  0  0  0]]
>>> x = Tensor(np.array([[ 1,  2,  3,  4],
...                      [ 5,  6,  7,  8],
...                      [10, 11, 12, 13],
...                      [14, 15, 16, 17]]))
>>> triu = nn.Triu()
>>> result = triu(x, 2)
>>> print(result)
[[ 0  0  3  4]
 [ 0  0  0  8]
 [ 0  0  0  0]
 [ 0  0  0  0]]
>>> x = Tensor(np.array([[ 1,  2,  3,  4],
...                      [ 5,  6,  7,  8],
...                      [10, 11, 12, 13],
...                      [14, 15, 16, 17]]))
>>> triu = nn.Triu()
>>> result = triu(x, -1)
>>> print(result)
[[ 1  2  3  4]
 [ 5  6  7  8]
 [ 0 11 12 13]
 [ 0  0 16 17]]

class tinyms.layers.ResizeBilinear

Samples the input tensor to the given size or scale_factor by using bilinear interpolate.

Inputs:

• x (Tensor) - Tensor to be resized. Input tensor must be a 4-D tensor with shape \((batch, channels, height, width)\), with data type of float16 or float32.

• size (Union[tuple[int], list[int]]): A tuple or list of 2 int elements \((new\_height, new\_width)\),the new size of the tensor. One and only one of size and scale_factor can be set to None. Default: None.

• scale_factor (int): The scale factor of new size of the tensor. The value should be positive integer. One and only one of size and scale_factor can be set to None. Default: None.

• align_corners (bool): If true, rescale input by \((new\_height - 1) / (height - 1)\), which exactly aligns the 4 corners of images and resized images. If false, rescale by \(new\_height / height\). Default: False.

Outputs:

Resized tensor. If size is set, the result is 4-D tensor with shape \((batch, channels, new\_height, new\_width)\), and the data type is the same as x. If scale is set, the result is 4-D tensor with shape \((batch, channels, scale\_factor * height, scale\_factor * width)\) and the data type is the same as x.

Raises

• TypeError – If size is not one of tuple, list, None.

• TypeError – If scale_factor is neither int nor None.

• TypeError – If align_corners is not a bool.

• TypeError – If dtype of x is neither float16 nor float32.

• ValueError – If size and scale_factor are both None or not None.

• ValueError – If length of shape of x is not equal to 4.

• ValueError – If scale_factor is an int which is less than 0.

• ValueError – If size is a list or tuple whose length is not equal to 2.

Supported Platforms:

Ascend CPU GPU

Examples

>>> x = Tensor([[[[1, 2, 3, 4], [5, 6, 7, 8]]]], mindspore.float32)
>>> resize_bilinear = nn.ResizeBilinear()
>>> result = resize_bilinear(x, size=(5,5))
>>> print(x)
[[[[1. 2. 3. 4.]
   [5. 6. 7. 8.]]]]
>>> print(result)
[[[[1.        1.8       2.6       3.4       4.       ]
   [2.6       3.4       4.2000003 5.        5.6000004]
   [4.2       5.0000005 5.8       6.6       7.2      ]
   [5.        5.8       6.6       7.4       8.       ]
   [5.        5.8       6.6       7.4000006 8.       ]]]]
>>> print(result.shape)
(1, 1, 5, 5)

class tinyms.layers.MatrixDiag

Returns a batched diagonal tensor with a given batched diagonal values.

Assume x has \(k\) dimensions \([I, J, K, ..., N]\), then the output is a tensor of rank \(k+1\) with dimensions \([I, J, K, ..., N, N]\) where: \(output[i, j, k, ..., m, n] = 1\{m=n\} * x[i, j, k, ..., n]\)

Inputs:

• x (Tensor) - The diagonal values. It can be one of the following data types: float32, float16, int32, int8, and uint8. The shape is \((N,*)\) where \(*\) means, any number of additional dimensions.

Outputs:

Tensor, has the same type as input x. The shape must be x.shape + (x.shape[-1], ).

Raises

TypeError – If dtype of x is not one of float32, float16, int32, int8 or uint8.

Supported Platforms:

Ascend

Examples

>>> x = Tensor(np.array([1, -1]), mindspore.float32)
>>> matrix_diag = nn.MatrixDiag()
>>> output = matrix_diag(x)
>>> print(x.shape)
(2,)
>>> print(output)
[[ 1.  0.]
 [ 0. -1.]]
>>> print(output.shape)
(2, 2)
>>> x = Tensor(np.array([[1, -1], [1, -1]]), mindspore.float32)
>>> matrix_diag = nn.MatrixDiag()
>>> output = matrix_diag(x)
>>> print(x.shape)
(2, 2)
>>> print(output)
[[[ 1.  0.]
  [ 0. -1.]]
 [[ 1.  0.]
  [ 0. -1.]]]
>>> print(output.shape)
(2, 2, 2)
>>> x = Tensor(np.array([[1, -1, 1], [1, -1, 1]]), mindspore.float32)
>>> matrix_diag = nn.MatrixDiag()
>>> output = matrix_diag(x)
>>> print(x.shape)
(2, 3)
>>> print(output)
[[[ 1.  0.  0.]
  [ 0. -1.  0.]
  [ 0.  0.  1.]
 [[ 1.  0.  0.]
  [ 0. -1.  0.]
  [ 0.  0.  1.]]]
>>> print(output.shape)
(2, 3, 3)

class tinyms.layers.MatrixDiagPart

Returns the batched diagonal part of a batched tensor.

Assume x has \(k\) dimensions \([I, J, K, ..., M, N]\), then the output is a tensor of rank \(k-1\) with dimensions \([I, J, K, ..., min(M, N)]\) where: \(output[i, j, k, ..., n] = x[i, j, k, ..., n, n]\)

Inputs:

• x (Tensor) - The batched tensor. It can be one of the following data types: float32, float16, int32, int8, and uint8.

Outputs:

Tensor, has the same type as input x. The shape must be x.shape[:-2] + [min(x.shape[-2:])].

Raises

TypeError – If dtype of x is not one of float32, float16, int32, int8 or uint8.

Supported Platforms:

Ascend

Examples

>>> x = Tensor([[[-1, 0], [0, 1]],
...             [[-1, 0], [0, 1]],
...             [[-1, 0], [0, 1]]], mindspore.float32)
>>> matrix_diag_part = nn.MatrixDiagPart()
>>> output = matrix_diag_part(x)
>>> print(output)
[[-1.  1.]
 [-1.  1.]
 [-1.  1.]]
>>> x = Tensor([[-1, 0, 0, 1],
...             [-1, 0, 0, 1],
...             [-1, 0, 0, 1],
...             [-1, 0, 0, 1]], mindspore.float32)
>>> matrix_diag_part = nn.MatrixDiagPart()
>>> output = matrix_diag_part(x)
>>> print(output)
[-1 0 0 1]

class tinyms.layers.MatrixSetDiag

Modifies the batched diagonal part of a batched tensor.

Assume x has \(k+1\) dimensions \([I, J, K, ..., M, N]\) and diagonal has \(k\) dimensions \([I, J, K, ..., min(M, N)]\). Then the output is a tensor of rank \(k+1\) with dimensions \([I, J, K, ..., M, N]\) where:

\[output[i, j, k, ..., m, n] = diagnoal[i, j, k, ..., n]\ for\ m == n\]

\[output[i, j, k, ..., m, n] = x[i, j, k, ..., m, n]\ for\ m != n\]

Inputs:

• x (Tensor) - The batched tensor. Rank k+1, where k >= 1. It can be one of the following data types: float32, float16, int32, int8, and uint8.

• diagonal (Tensor) - The diagonal values. Must have the same type as input x. Rank k, where k >= 1.

Outputs:

Tensor, has the same type and shape as input x.

Raises

• TypeError – If dtype of x or diagonal is not one of float32, float16, int32, int8 or uint8.

• ValueError – If length of shape of x is less than 2.

• ValueError – If x_shape[-2] < x_shape[-1] and x_shape[:-1] != diagonal_shape.

• ValueError – If x_shape[-2] >= x_shape[-1] and x_shape[:-2] + x_shape[-1:] != diagonal_shape.

Supported Platforms:

Ascend

Examples

>>> x = Tensor([[[-1, 0], [0, 1]], [[-1, 0], [0, 1]], [[-1, 0], [0, 1]]], mindspore.float32)
>>> diagonal = Tensor([[-1., 2.], [-1., 1.], [-1., 1.]], mindspore.float32)
>>> matrix_set_diag = nn.MatrixSetDiag()
>>> output = matrix_set_diag(x, diagonal)
>>> print(output)
[[[-1.  0.]
  [ 0.  2.]]
 [[-1.  0.]
  [ 0.  1.]]
 [[-1.  0.]
  [ 0.  1.]]]

class tinyms.layers.L1Regularizer(scale)

Applies l1 regularization to weights.

l1 regularization makes weights sparsity

\[\text{loss}=\lambda * \text{reduce_sum}(\text{abs}(\omega))\]

Note

scale(regularization factor) should be a number which greater than 0

Parameters

scale (int, float) – l1 regularization factor which greater than 0.

Inputs:

• weights (Tensor) - The input of L1Regularizer with data type of float16 or float32. The shape is \((N,*)\) where \(*\) means, any number of additional dimensions.

Outputs:

Tensor, which dtype is higher precision data type between mindspore.float32 and weights dtype, and Tensor shape is ()

Raises

• TypeError – If scale is neither an int nor float.

• ValueError – If scale is not greater than 0.

• ValueError – If scale is math.inf or math.nan.

Supported Platforms:

Ascend GPU CPU

Examples

>>> scale = 0.5
>>> net = nn.L1Regularizer(scale)
>>> weights = Tensor(np.array([[1.0, -2.0], [-3.0, 4.0]]).astype(np.float32))
>>> output = net(weights)
>>> print(output.asnumpy())
5.0

class tinyms.layers.Roll(shift, axis)

Rolls the elements of a tensor along an axis.

The elements are shifted positively (towards larger indices) by the offset of shift along the dimension of axis. Negative shift values will shift elements in the opposite direction. Elements that roll passed the last position will wrap around to the first and vice versa. Multiple shifts along multiple axes may be specified.

Parameters

• shift (Union[list(int), tuple(int), int]) – Specifies the number of places by which elements are shifted positively (towards larger indices) along the specified dimension. Negative shifts will roll the elements in the opposite direction.

• axis (Union[list(int), tuple(int), int]) – Specifies the dimension indexes of shape to be rolled.

Inputs:

• input_x (Tensor) - Input tensor.

Outputs:

Tensor, has the same shape and type as input_x.

Raises

• TypeError – If shift is not an int, a tuple or a list.

• TypeError – If axis is not an int, a tuple or a list.

• TypeError – If element of shift is not an int.

• TypeError – If element of axis is not an int.

• ValueError – If axis is out of the range [-len(input_x.shape), len(input_x.shape)).

• ValueError – If length of shape of shift is not equal to length of shape of axis.

Supported Platforms:

Ascend

Examples

>>> input_x = Tensor(np.array([0, 1, 2, 3, 4]).astype(np.float32))
>>> op = nn.Roll(shift=2, axis=0)
>>> output = op(input_x)
>>> print(output)
[3. 4. 0. 1. 2.]
>>> input_x = Tensor(np.array([[0, 1, 2, 3, 4], [5, 6, 7, 8, 9]]).astype(np.float32))
>>> op = nn.Roll(shift=[1, -2], axis=[0, 1])
>>> output = op(input_x)
>>> print(output)
[[7. 8. 9. 5. 6.]
 [2. 3. 4. 0. 1.]]

class tinyms.layers.Embedding(vocab_size, embedding_size, use_one_hot=False, embedding_table='normal', dtype=mindspore.float32, padding_idx=None)

A simple lookup table that stores embeddings of a fixed dictionary and size.

This module is often used to store word embeddings and retrieve them using indices. The input to the module is a list of indices, and the output is the corresponding word embeddings.

Note

When ‘use_one_hot’ is set to True, the type of the x must be mindspore.int32.

Parameters

• vocab_size (int) – Size of the dictionary of embeddings.

• embedding_size (int) – The size of each embedding vector.

• use_one_hot (bool) – Specifies whether to apply one_hot encoding form. Default: False.

• embedding_table (Union[Tensor, str, Initializer, numbers.Number]) – Initializer for the embedding_table. Refer to class initializer for the values of string when a string is specified. Default: ‘normal’.

• dtype (mindspore.dtype) – Data type of x. Default: mindspore.float32.

• padding_idx (int, None) – When the padding_idx encounters index, the output embedding vector of this index will be initialized to zero. Default: None. The feature is inactivated.

Inputs:

• x (Tensor) - Tensor of shape \((\text{batch_size}, \text{x_length})\). The elements of the Tensor must be integer and not larger than vocab_size. Otherwise the corresponding embedding vector will be zero. The data type is int32 or int64.

Outputs:

Tensor of shape \((\text{batch_size}, \text{x_length}, \text{embedding_size})\).

Raises

• TypeError – If vocab_size or embedding_size is not an int.

• TypeError – If use_one_hot is not a bool.

• ValueError – If padding_idx is an int which not in range [0, vocab_size].

Supported Platforms:

Ascend GPU CPU

Examples

>>> net = nn.Embedding(20000, 768,  True)
>>> x = Tensor(np.ones([8, 128]), mindspore.int32)
>>> # Maps the input word IDs to word embedding.
>>> output = net(x)
>>> result = output.shape
>>> print(result)
(8, 128, 768)

class tinyms.layers.EmbeddingLookup(vocab_size, embedding_size, param_init='normal', target='CPU', slice_mode='batch_slice', manual_shapes=None, max_norm=None, sparse=True, vocab_cache_size=0)

Returns a slice of the input tensor based on the specified indices.

Note

When ‘target’ is set to ‘CPU’, this module will use P.EmbeddingLookup().add_prim_attr(‘primitive_target’, ‘CPU’) which specified ‘offset = 0’ to lookup table. When ‘target’ is set to ‘DEVICE’, this module will use P.Gather() which specified ‘axis = 0’ to lookup table. In field slice mode, the manual_shapes must be given. It is a tuple ,where the element is vocab[i], vocab[i] is the row numbers for i-th part.

Parameters

• vocab_size (int) – Size of the dictionary of embeddings.

• embedding_size (int) – The size of each embedding vector.

• param_init (Union[Tensor, str, Initializer, numbers.Number]) – Initializer for the embedding_table. Refer to class initializer for the values of string when a string is specified. Default: ‘normal’.

• target (str) – Specifies the target where the op is executed. The value must in [‘DEVICE’, ‘CPU’]. Default: ‘CPU’.

• slice_mode (str) – The slicing way in semi_auto_parallel/auto_parallel. The value must get through nn.EmbeddingLookup. Default: nn.EmbeddingLookup.BATCH_SLICE.

• manual_shapes (tuple) – The accompaniment array in field slice mode.

• max_norm (Union[float, None]) – A maximum clipping value. The data type must be float16, float32 or None. Default: None

• sparse (bool) – Using sparse mode. When ‘target’ is set to ‘CPU’, ‘sparse’ has to be true. Default: True.

• vocab_cache_size (int) – Cache size of the dictionary of embeddings. Default: 0. It is valid only in ‘DEVICE’ target. And the moment parameter of corresponding optimizer will also be set to the cache size. In addition, it should be noted that it will cost the ‘DEVICE’ memory, so suggests setting a reasonable value to avoid insufficient memory.

Inputs:

• input_indices (Tensor) - The shape of tensor is \((y_1, y_2, ..., y_S)\). Specifies the indices of elements of the original Tensor. Values can be out of range of embedding_table, and the exceeding part will be filled with 0 in the output. Values does not support negative and the result is undefined if values are negative. Input_indices must only be a 2d tensor in this interface when run in semi auto parallel/auto parallel mode.

Outputs:

Tensor, the shape of tensor is \((z_1, z_2, ..., z_N)\).

Raises

• TypeError – If vocab_size or embedding_size or vocab_cache_size is not an int.

• TypeError – If sparse is not a bool or manual_shapes is not a tuple.

• ValueError – If vocab_size or embedding_size is less than 1.

• ValueError – If vocab_cache_size is less than 0.

• ValueError – If target is neither ‘CPU’ nor ‘DEVICE’.

• ValueError – If slice_mode is not one of ‘batch_slice’ or ‘field_slice’ or ‘table_row_slice’ or ‘table_column_slice’.

• ValueError – If sparse is False and target is ‘CPU’.

• ValueError – If slice_mode is ‘field_slice’ and manual_shapes is None.

Supported Platforms:

Ascend GPU CPU

Examples

>>> input_indices = Tensor(np.array([[1, 0], [3, 2]]), mindspore.int32)
>>> result = nn.EmbeddingLookup(4,2)(input_indices)
>>> print(result.shape)
(2, 2, 2)

class tinyms.layers.MultiFieldEmbeddingLookup(vocab_size, embedding_size, field_size, param_init='normal', target='CPU', slice_mode='batch_slice', feature_num_list=None, max_norm=None, sparse=True, operator='SUM')

Returns a slice of input tensor based on the specified indices and the field ids. This operation supports looking up embeddings using multi hot and one hot fields simultaneously.

Note

When ‘target’ is set to ‘CPU’, this module will use P.EmbeddingLookup().add_prim_attr(‘primitive_target’, ‘CPU’) which specified ‘offset = 0’ to lookup table. When ‘target’ is set to ‘DEVICE’, this module will use P.Gather() which specified ‘axis = 0’ to lookup table. The vectors with the same field_ids will be combined by the ‘operator’, such as ‘SUM’, ‘MAX’ and ‘MEAN’. Ensure the input_values of the padded id is zero, so that they can be ignored. The final output will be zeros if the sum of absolute weight of the field is zero. This class only supports [‘table_row_slice’, ‘batch_slice’ and ‘table_column_slice’]. For the operation ‘MAX’ on device Ascend, there is a constrain where batch_size * (seq_length + field_size) < 3500.

Parameters

• vocab_size (int) – The size of the dictionary of embeddings.

• embedding_size (int) – The size of each embedding vector.

• field_size (int) – The field size of the final outputs.

• param_init (Union[Tensor, str, Initializer, numbers.Number]) – Initializer for the embedding_table. Refer to class initializer for the values of string when a string is specified. Default: ‘normal’.

• target (str) – Specifies the target where the op is executed. The value must in [‘DEVICE’, ‘CPU’]. Default: ‘CPU’.

• slice_mode (str) – The slicing way in semi_auto_parallel/auto_parallel. The value must get through nn.EmbeddingLookup. Default: nn.EmbeddingLookup.BATCH_SLICE.

• feature_num_list (tuple) – The accompaniment array in field slice mode. This is unused currently. Default: None.

• max_norm (Union[float, None]) – A maximum clipping value. The data type must be float16, float32 or None. Default: None

• sparse (bool) – Using sparse mode. When ‘target’ is set to ‘CPU’, ‘sparse’ has to be true. Default: True.

• operator (str) – The pooling method for the features in one field. Support ‘SUM’, ‘MEAN’ and ‘MAX’. Default: ‘SUM’.

Inputs:

• input_indices (Tensor) - The shape of tensor is \((batch\_size, seq\_length)\). Specifies the indices of elements of the original Tensor. Input_indices must be a 2d tensor in this interface. Type is Int32, Int64.

• input_values (Tensor) - The shape of tensor is \((batch\_size, seq\_length)\). Specifies the weights of elements of the input_indices. The lookout vector will multiply with the input_values. Type is Float32.

• field_ids (Tensor) - The shape of tensor is \((batch\_size, seq\_length)\). Specifies the field id of elements of the input_indices. Type is Int32.

Outputs:

Tensor, the shape of tensor is \((batch\_size, field\_size, embedding\_size)\). Type is Float32.

Raises

• TypeError – If vocab_size or embedding_size or field_size is not an int.

• TypeError – If sparse is not a bool or feature_num_list is not a tuple.

• ValueError – If vocab_size or embedding_size or field_size is less than 1.

• ValueError – If target is neither ‘CPU’ nor ‘DEVICE’.

• ValueError – If slice_mode is not one of ‘batch_slice’, ‘field_slice’, ‘table_row_slice’, ‘table_column_slice’.

• ValueError – If sparse is False and target is ‘CPU’.

• ValueError – If slice_mode is ‘field_slice’ and feature_num_list is None.

• ValueError – If operator is not one of ‘SUM’, ‘MAX’, ‘MEAN’.

Supported Platforms:

Ascend GPU

Examples

>>> input_indices = Tensor([[2, 4, 6, 0, 0], [1, 3, 5, 0, 0]], mindspore.int32)
>>> input_values = Tensor([[1, 1, 1, 0, 0], [1, 1, 1, 0, 0]], mindspore.float32)
>>> field_ids = Tensor([[0, 1, 1, 0, 0], [0, 0, 1, 0, 0]], mindspore.int32)
>>> net = nn.MultiFieldEmbeddingLookup(10, 2, field_size=2, operator='SUM', target='DEVICE')
>>> out = net(input_indices, input_values, field_ids)
>>> print(out.shape)
(2, 2, 2)

class tinyms.layers.AvgPool2d(kernel_size=1, stride=1, pad_mode='valid', data_format='NCHW')

2D average pooling for temporal data.

Applies a 2D average pooling over an input Tensor which can be regarded as a composition of 2D input planes.

Typically the input is of shape \((N_{in}, C_{in}, H_{in}, W_{in})\), AvgPool2d outputs regional average in the \((H_{in}, W_{in})\)-dimension. Given kernel size \(ks = (h_{ker}, w_{ker})\) and stride \(s = (s_0, s_1)\), the operation is as follows.

\[\text{output}(N_i, C_j, h, w) = \frac{1}{h_{ker} * w_{ker}} \sum_{m=0}^{h_{ker}-1} \sum_{n=0}^{w_{ker}-1} \text{input}(N_i, C_j, s_0 \times h + m, s_1 \times w + n)\]

Note

pad_mode for training only supports “same” and “valid”.

Parameters

• kernel_size (Union[int, tuple[int]]) – The size of kernel used to take the average value. The data type of kernel_size must be int and the value represents the height and width, or a tuple of two int numbers that represent height and width respectively. Default: 1.

• stride (Union[int, tuple[int]]) – The distance of kernel moving, an int number that represents the height and width of movement are both strides, or a tuple of two int numbers that represent height and width of movement respectively. Default: 1.

• pad_mode (str) –

  The optional value for pad mode, is “same” or “valid”, not case sensitive. Default: “valid”.

  • same: Adopts the way of completion. The height and width of the output will be the same as the input. The total number of padding will be calculated in horizontal and vertical directions and evenly distributed to top and bottom, left and right if possible. Otherwise, the last extra padding will be done from the bottom and the right side.

  • valid: Adopts the way of discarding. The possible largest height and width of output will be returned without padding. Extra pixels will be discarded.

• data_format (str) – The optional value for data format, is ‘NHWC’ or ‘NCHW’. Default: ‘NCHW’.

Inputs:

• x (Tensor) - Tensor of shape \((N, C_{in}, H_{in}, W_{in})\).

Outputs:

Tensor of shape \((N, C_{out}, H_{out}, W_{out})\).

Raises

• TypeError – If kernel_size or strides is neither int nor tuple.

• ValueError – If pad_mode is neither ‘valid’ nor ‘same’ with not case sensitive.

• ValueError – If data_format is neither ‘NCHW’ nor ‘NHWC’.

• ValueError – If kernel_size or strides is less than 1.

• ValueError – If length of shape of x is not equal to 4.

Supported Platforms:

Ascend GPU CPU

Examples

>>> pool = nn.AvgPool2d(kernel_size=3, stride=1)
>>> x = Tensor(np.random.randint(0, 10, [1, 2, 4, 4]), mindspore.float32)
>>> output = pool(x)
>>> print(output.shape)
(1, 2, 2, 2)

class tinyms.layers.MaxPool2d(kernel_size=1, stride=1, pad_mode='valid', data_format='NCHW')

2D max pooling operation for temporal data.

Applies a 2D max pooling over an input Tensor which can be regarded as a composition of 2D planes.

Typically the input is of shape \((N_{in}, C_{in}, H_{in}, W_{in})\), MaxPool2d outputs regional maximum in the \((H_{in}, W_{in})\)-dimension. Given kernel size \(ks = (h_{ker}, w_{ker})\) and stride \(s = (s_0, s_1)\), the operation is as follows.

\[\text{output}(N_i, C_j, h, w) = \max_{m=0, \ldots, h_{ker}-1} \max_{n=0, \ldots, w_{ker}-1} \text{input}(N_i, C_j, s_0 \times h + m, s_1 \times w + n)\]

Note

pad_mode for training only supports “same” and “valid”.

Parameters

• kernel_size (Union[int, tuple[int]]) – The size of kernel used to take the max value, is an int number that represents height and width are both kernel_size, or a tuple of two int numbers that represent height and width respectively. Default: 1.

• stride (Union[int, tuple[int]]) – The distance of kernel moving, an int number that represents the height and width of movement are both strides, or a tuple of two int numbers that represent height and width of movement respectively. Default: 1.

• pad_mode (str) –

  The optional value for pad mode, is “same” or “valid”, not case sensitive. Default: “valid”.

  • same: Adopts the way of completion. The height and width of the output will be the same as the input. The total number of padding will be calculated in horizontal and vertical directions and evenly distributed to top and bottom, left and right if possible. Otherwise, the last extra padding will be done from the bottom and the right side.

  • valid: Adopts the way of discarding. The possible largest height and width of output will be returned without padding. Extra pixels will be discarded.

• data_format (str) – The optional value for data format, is ‘NHWC’ or ‘NCHW’. Default: ‘NCHW’.

Inputs:

• x (Tensor) - Tensor of shape \((N, C_{in}, H_{in}, W_{in})\).

Outputs:

Tensor of shape \((N, C_{out}, H_{out}, W_{out})\).

Raises

• TypeError – If kernel_size or strides is neither int nor tuple.

• ValueError – If pad_mode is neither ‘valid’ nor ‘same’ with not case sensitive.

• ValueError – If data_format is neither ‘NCHW’ nor ‘NHWC’.

• ValueError – If kernel_size or strides is less than 1.

• ValueError – If length of shape of x is not equal to 4.

Supported Platforms:

Ascend GPU CPU

Examples

>>> pool = nn.MaxPool2d(kernel_size=3, stride=1)
>>> x = Tensor(np.random.randint(0, 10, [1, 2, 4, 4]), mindspore.float32)
>>> output = pool(x)
>>> print(output.shape)
(1, 2, 2, 2)

class tinyms.layers.AvgPool1d(kernel_size=1, stride=1, pad_mode='valid')

1D average pooling for temporal data.

Applies a 1D average pooling over an input Tensor which can be regarded as a composition of 1D input planes.

Typically the input is of shape \((N_{in}, C_{in}, L_{in})\), AvgPool1d outputs regional average in the \((L_{in})\)-dimension. Given kernel size \(ks = l_{ker}\) and stride \(s = s_0\), the operation is as follows.

\[\text{output}(N_i, C_j, l) = \frac{1}{l_{ker}} \sum_{n=0}^{l_{ker}-1} \text{input}(N_i, C_j, s_0 \times l + n)\]

Note

pad_mode for training only supports “same” and “valid”.

Parameters

• kernel_size (int) – The size of kernel window used to take the average value, Default: 1.

• stride (int) – The distance of kernel moving, an int number that represents the width of movement is strides, Default: 1.

• pad_mode (str) –

  The optional value for pad mode, is “same” or “valid”, not case sensitive. Default: “valid”.

  • same: Adopts the way of completion. The height and width of the output will be the same as the input. The total number of padding will be calculated in horizontal and vertical directions and evenly distributed to top and bottom, left and right if possible. Otherwise, the last extra padding will be done from the bottom and the right side.

  • valid: Adopts the way of discarding. The possible largest height and width of output will be returned without padding. Extra pixels will be discarded.

Inputs:

• x (Tensor) - Tensor of shape \((N, C_{in}, L_{in})\).

Outputs:

Tensor of shape \((N, C_{out}, L_{out})\).

Raises

• TypeError – If kernel_size or stride is not an int.

• ValueError – If pad_mode is neither ‘same’ nor ‘valid’ with not case sensitive.

• ValueError – If kernel_size or strides is less than 1.

• ValueError – If length of shape of x is not equal to 3.

Supported Platforms:

Ascend GPU CPU

Examples

>>> pool = nn.AvgPool1d(kernel_size=6, stride=1)
>>> x = Tensor(np.random.randint(0, 10, [1, 3, 6]), mindspore.float32)
>>> output = pool(x)
>>> result = output.shape
>>> print(result)
(1, 3, 1)

class tinyms.layers.MaxPool1d(kernel_size=1, stride=1, pad_mode='valid')

1D max pooling operation for temporal data.

Applies a 1D max pooling over an input Tensor which can be regarded as a composition of 1D planes.

Typically the input is of shape \((N_{in}, C_{in}, L_{in})\), MaxPool1d outputs regional maximum in the \((L_{in})\)-dimension. Given kernel size \(ks = (l_{ker})\) and stride \(s = (s_0)\), the operation is as follows.

\[\text{output}(N_i, C_j, l) = \max_{n=0, \ldots, l_{ker}-1} \text{input}(N_i, C_j, s_0 \times l + n)\]

Note

pad_mode for training only supports “same” and “valid”.

Parameters

• kernel_size (int) – The size of kernel used to take the max value, Default: 1.

• stride (int) – The distance of kernel moving, an int number that represents the width of movement is stride, Default: 1.

• pad_mode (str) –

  The optional value for pad mode, is “same” or “valid”, not case sensitive. Default: “valid”.

  • same: Adopts the way of completion. The total number of padding will be calculated in horizontal and vertical directions and evenly distributed to top and bottom, left and right if possible. Otherwise, the last extra padding will be done from the bottom and the right side.

  • valid: Adopts the way of discarding. The possible largest height and width of output will be returned without padding. Extra pixels will be discarded.

Inputs:

• x (Tensor) - Tensor of shape \((N, C, L_{in})\).

Outputs:

Tensor of shape \((N, C, L_{out}))\).

Raises

• TypeError – If kernel_size or strides is not an int.

• ValueError – If pad_mode is neither ‘valid’ nor ‘same’ with not case sensitive.

• ValueError – If data_format is neither ‘NCHW’ nor ‘NHWC’.

• ValueError – If kernel_size or strides is less than 1.

• ValueError – If length of shape of x is not equal to 4.

Supported Platforms:

Ascend GPU CPU

Examples

>>> max_pool = nn.MaxPool1d(kernel_size=3, stride=1)
>>> x = Tensor(np.random.randint(0, 10, [1, 2, 4]), mindspore.float32)
>>> output = max_pool(x)
>>> result = output.shape
>>> print(result)
(1, 2, 2)

class tinyms.layers.ImageGradients

Returns two tensors, the first is along the height dimension and the second is along the width dimension.

Assume an image shape is \(h*w\). The gradients along the height and the width are \(dy\) and \(dx\), respectively.

\[ \begin{align}\begin{aligned}dy[i] = \begin{cases} image[i+1, :]-image[i, :], &if\ 0<=i<h-1 \cr 0, &if\ i==h-1\end{cases}\\dx[i] = \begin{cases} image[:, i+1]-image[:, i], &if\ 0<=i<w-1 \cr 0, &if\ i==w-1\end{cases}\end{aligned}\end{align} \]

Inputs:

• images (Tensor) - The input image data, with format ‘NCHW’.

Outputs:

• dy (Tensor) - vertical image gradients, the same type and shape as input.

• dx (Tensor) - horizontal image gradients, the same type and shape as input.

Raises

ValueError – If length of shape of images is not equal to 4.

Supported Platforms:

Ascend GPU CPU

Examples

>>> net = nn.ImageGradients()
>>> image = Tensor(np.array([[[[1, 2], [3, 4]]]]), dtype=mindspore.int32)
>>> output = net(image)
>>> print(output)
(Tensor(shape=[1, 1, 2, 2], dtype=Int32, value=
[[[[2, 2],
   [0, 0]]]]), Tensor(shape=[1, 1, 2, 2], dtype=Int32, value=
[[[[1, 0],
   [1, 0]]]]))

class tinyms.layers.SSIM(max_val=1.0, filter_size=11, filter_sigma=1.5, k1=0.01, k2=0.03)

Returns SSIM index between two images.

Its implementation is based on Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004). Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing.

SSIM is a measure of the similarity of two pictures. Like PSNR, SSIM is often used as an evaluation of image quality. SSIM is a number between 0 and 1.The larger it is, the smaller the gap between the output image and the undistorted image, that is, the better the image quality. When the two images are exactly the same, SSIM=1.

\[\begin{split}l(x,y)&=\frac{2\mu_x\mu_y+C_1}{\mu_x^2+\mu_y^2+C_1}, C_1=(K_1L)^2.\\ c(x,y)&=\frac{2\sigma_x\sigma_y+C_2}{\sigma_x^2+\sigma_y^2+C_2}, C_2=(K_2L)^2.\\ s(x,y)&=\frac{\sigma_{xy}+C_3}{\sigma_x\sigma_y+C_3}, C_3=C_2/2.\\ SSIM(x,y)&=l*c*s\\&=\frac{(2\mu_x\mu_y+C_1)(2\sigma_{xy}+C_2}{(\mu_x^2+\mu_y^2+C_1)(\sigma_x^2+\sigma_y^2+C_2)}.\end{split}\]

Parameters

• max_val (Union[int, float]) – The dynamic range of the pixel values (255 for 8-bit grayscale images). Default: 1.0.

• filter_size (int) – The size of the Gaussian filter. Default: 11. The value must be greater than or equal to 1.

• filter_sigma (float) – The standard deviation of Gaussian kernel. Default: 1.5. The value must be greater than 0.

• k1 (float) – The constant used to generate c1 in the luminance comparison function. Default: 0.01.

• k2 (float) – The constant used to generate c2 in the contrast comparison function. Default: 0.03.

Inputs:

• img1 (Tensor) - The first image batch with format ‘NCHW’. It must be the same shape and dtype as img2.

• img2 (Tensor) - The second image batch with format ‘NCHW’. It must be the same shape and dtype as img1.

Outputs:

Tensor, has the same dtype as img1. It is a 1-D tensor with shape N, where N is the batch num of img1.

Raises

• TypeError – If max_val is neither int nor float.

• TypeError – If k1, k2 or filter_sigma is not a float.

• TypeError – If filter_size is not an int.

• ValueError – If max_val or filter_sigma is less than or equal to 0.

• ValueError – If filter_size is less than 0.

Supported Platforms:

Ascend GPU CPU

Examples

>>> import numpy as np
>>> import mindspore.nn as nn
>>> from mindspore import Tensor
>>> net = nn.SSIM()
>>> img1 = Tensor(np.ones([1, 3, 16, 16]).astype(np.float32))
>>> img2 = Tensor(np.ones([1, 3, 16, 16]).astype(np.float32))
>>> output = net(img1, img2)
>>> print(output)
[1.]

class tinyms.layers.MSSSIM(max_val=1.0, power_factors=(0.0448, 0.2856, 0.3001, 0.2363, 0.1333), filter_size=11, filter_sigma=1.5, k1=0.01, k2=0.03)

Returns MS-SSIM index between two images.

Its implementation is based on Wang, Zhou, Eero P. Simoncelli, and Alan C. Bovik. Multiscale structural similarity for image quality assessment. Signals, Systems and Computers, 2004.

\[\begin{split}l(x,y)&=\frac{2\mu_x\mu_y+C_1}{\mu_x^2+\mu_y^2+C_1}, C_1=(K_1L)^2.\\ c(x,y)&=\frac{2\sigma_x\sigma_y+C_2}{\sigma_x^2+\sigma_y^2+C_2}, C_2=(K_2L)^2.\\ s(x,y)&=\frac{\sigma_{xy}+C_3}{\sigma_x\sigma_y+C_3}, C_3=C_2/2.\\ MSSSIM(x,y)&=l^\alpha_M*{\prod_{1\leq j\leq M} (c^\beta_j*s^\gamma_j)}.\end{split}\]

Parameters

• max_val (Union[int, float]) – The dynamic range of the pixel values (255 for 8-bit grayscale images). Default: 1.0.

• power_factors (Union[tuple, list]) – Iterable of weights for each scal e. Default: (0.0448, 0.2856, 0.3001, 0.2363, 0.1333). Default values obtained by Wang et al.

• filter_size (int) – The size of the Gaussian filter. Default: 11.

• filter_sigma (float) – The standard deviation of Gaussian kernel. Default: 1.5.

• k1 (float) – The constant used to generate c1 in the luminance comparison function. Default: 0.01.

• k2 (float) – The constant used to generate c2 in the contrast comparison function. Default: 0.03.

Inputs:

• img1 (Tensor) - The first image batch with format ‘NCHW’. It must be the same shape and dtype as img2.

• img2 (Tensor) - The second image batch with format ‘NCHW’. It must be the same shape and dtype as img1.

Outputs:

Tensor, the value is in range [0, 1]. It is a 1-D tensor with shape N, where N is the batch num of img1.

Raises

• TypeError – If max_val is neither int nor float.

• TypeError – If power_factors is neither tuple nor list.

• TypeError – If k1, k2 or filter_sigma is not a float.

• TypeError – If filter_size is not an int.

• ValueError – If max_val or filter_sigma is less than or equal to 0.

• ValueError – If filter_size is less than 0.

• ValueError – If length of shape of img1 or img2 is not equal to 4.

Supported Platforms:

Ascend GPU

Examples

>>> import numpy as np
>>> import mindspore.nn as nn
>>> from mindspore import Tensor
>>> net = nn.MSSSIM(power_factors=(0.033, 0.033, 0.033))
>>> img1 = Tensor(np.ones((1, 3, 128, 128)).astype(np.float32))
>>> img2 = Tensor(np.ones((1, 3, 128, 128)).astype(np.float32))
>>> output = net(img1, img2)
>>> print(output)
[1.]

class tinyms.layers.PSNR(max_val=1.0)

Returns Peak Signal-to-Noise Ratio of two image batches.

It produces a PSNR value for each image in batch. Assume inputs are \(I\) and \(K\), both with shape \(h*w\). \(MAX\) represents the dynamic range of pixel values.

\[\begin{split}MSE&=\frac{1}{hw}\sum\limits_{i=0}^{h-1}\sum\limits_{j=0}^{w-1}[I(i,j)-K(i,j)]^2\\ PSNR&=10*log_{10}(\frac{MAX^2}{MSE})\end{split}\]

Parameters

max_val (Union[int, float]) – The dynamic range of the pixel values (255 for 8-bit grayscale images). The value must be greater than 0. Default: 1.0.

Inputs:

• img1 (Tensor) - The first image batch with format ‘NCHW’. It must be the same shape and dtype as img2.

• img2 (Tensor) - The second image batch with format ‘NCHW’. It must be the same shape and dtype as img1.

Outputs:

Tensor, with dtype mindspore.float32. It is a 1-D tensor with shape N, where N is the batch num of img1.

Raises

• TypeError – If max_val is neither int nor float.

• ValueError – If max_val is less than or equal to 0.

• ValueError – If length of shape of img1 or img2 is not equal to 4.

Supported Platforms:

Ascend GPU CPU

Examples

>>> net = nn.PSNR()
>>> img1 = Tensor([[[[1, 2, 3, 4], [1, 2, 3, 4]]]])
>>> img2 = Tensor([[[[3, 4, 5, 6], [3, 4, 5, 6]]]])
>>> output = net(img1, img2)
>>> print(output)
[-6.0206]

class tinyms.layers.CentralCrop(central_fraction)

Crops the central region of the images with the central_fraction.

Parameters

central_fraction (float) – Fraction of size to crop. It must be float and in range (0.0, 1.0].

Inputs:

• image (Tensor) - A 3-D tensor of shape [C, H, W], or a 4-D tensor of shape [N, C, H, W].

Outputs:

Tensor, 3-D or 4-D float tensor, according to the input.

Raises

• TypeError – If central_fraction is not a float.

• ValueError – If central_fraction is not in range (0, 1.0].

Supported Platforms:

Ascend GPU CPU

Examples

>>> net = nn.CentralCrop(central_fraction=0.5)
>>> image = Tensor(np.random.random((4, 3, 4, 4)), mindspore.float32)
>>> output = net(image)
>>> print(output.shape)
(4, 3, 2, 2)

class tinyms.layers.FakeQuantWithMinMaxObserver(min_init=-6, max_init=6, ema=False, ema_decay=0.999, per_channel=False, channel_axis=1, num_channels=1, quant_dtype=<QuantDtype.INT8: 'INT8'>, symmetric=False, narrow_range=False, quant_delay=0, neg_trunc=False, mode='DEFAULT')

Quantization aware operation which provides the fake quantization observer function on data with min and max.

The detail of the quantization mode DEFAULT is described as below:

The running min/max \(x_{min}\) and \(x_{max}\) are computed as:

\[\begin{split}\begin{array}{ll} \\ x_{min} = \begin{cases} \min(\min(X), 0) & \text{ if } ema = \text{False} \\ \min((1 - c) \min(X) + \text{c } x_{min}, 0) & \text{ if } \text{otherwise} \end{cases}\\ x_{max} = \begin{cases} \max(\max(X), 0) & \text{ if } ema = \text{False} \\ \max((1 - c) \max(X) + \text{c } x_{max}, 0) & \text{ if } \text{otherwise} \end{cases} \end{array}\end{split}\]

where X is the input tensor, and \(c\) is the ema_decay.

The scale and zero point zp is computed as:

\[\begin{split}\begin{array}{ll} \\ scale = \begin{cases} \frac{x_{max} - x_{min}}{Q_{max} - Q_{min}} & \text{ if } symmetric = \text{False} \\ \frac{2\max(x_{max}, \left | x_{min} \right |) }{Q_{max} - Q_{min}} & \text{ if } \text{otherwise} \end{cases}\\ zp\_min = Q_{min} - \frac{x_{min}}{scale} \\ zp = \left \lfloor \min(Q_{max}, \max(Q_{min}, zp\_min)) + 0.5 \right \rfloor \end{array}\end{split}\]

where \(Q_{max}\) and \(Q_{min}\) is decided by quant_dtype, for example, if quant_dtype=INT8, then \(Q_{max} = 127\) and \(Q_{min} = -128\).

The fake quant output is computed as:

\[\begin{split}\begin{array}{ll} \\ u_{min} = (Q_{min} - zp) * scale \\ u_{max} = (Q_{max} - zp) * scale \\ u_X = \left \lfloor \frac{\min(u_{max}, \max(u_{min}, X)) - u_{min}}{scale} + 0.5 \right \rfloor \\ output = u_X * scale + u_{min} \end{array}\end{split}\]

The detail of the quantization mode LEARNED_SCALE is described as below:

The fake quant output is computed as:

\[ \begin{align}\begin{aligned}\begin{split}\bar{X}=\left\{\begin{matrix} clip\left ( \frac{X}{maxq},0,1\right ) \qquad \quad if\quad neg\_trunc\\ clip\left ( \frac{X}{maxq},-1,1\right )\qquad \ if\quad otherwise \end{matrix}\right. \\\end{split}\\output=\frac{floor\left ( \bar{X}\ast Q_{max}+0.5 \right ) \ast scale }{Q_{max}}\end{aligned}\end{align} \]

where X is the input tensor. where \(Q_{max}\) (quant_max) is decided by quant_dtype and neg_trunc, for example, if quant_dtype=INT8 and neg_trunc works, \(Q_{max} = 256\) , otherwise math:Q_{max} = 127.

The maxq is updated by training, and its gradient is calculated as follows:

\[ \begin{align}\begin{aligned}\begin{split}\frac{\partial \ output}{\partial \ maxq} = \left\{\begin{matrix} -\frac{X}{maxq}+\left \lfloor \frac{X}{maxq} \right \rceil \qquad if\quad bound_{lower}< \frac{X}{maxq}< 1\\ -1 \qquad \quad \qquad \quad if\quad \frac{X}{maxq}\le bound_{lower}\\ 1 \qquad \quad \qquad \quad if\quad \frac{X}{maxq}\ge 1 \qquad \quad \end{matrix}\right. \\\end{split}\\\begin{split}bound_{lower}= \left\{\begin{matrix} 0\qquad \quad if\quad neg\_trunc\\ -1\qquad if\quad otherwise \end{matrix}\right.\end{split}\end{aligned}\end{align} \]

Then minq is computed as:

\[\begin{split}minq=\left\{\begin{matrix} 0 \qquad \qquad \quad if\quad neg\_trunc\\ -maxq\qquad if\quad otherwise \end{matrix}\right.\end{split}\]

When exporting, the scale and zero point zp is computed as:

\[\begin{split}scale=\frac{maxq}{quant\_max} ,\quad zp=0 \\\end{split}\]

zp is equal to 0 consistently, due to the LEARNED_SCALE`s symmetric nature.

Parameters

• min_init (int, float, list) – The initialized min value. Default: -6.

• max_init (int, float, list) – The initialized max value. Default: 6.

• ema (bool) – The exponential Moving Average algorithm updates min and max. Default: False.

• ema_decay (float) – Exponential Moving Average algorithm parameter. Default: 0.999.

• per_channel (bool) – Quantization granularity based on layer or on channel. Default: False.

• channel_axis (int) – Quantization by channel axis. Default: 1.

• num_channels (int) – declarate the min and max channel size, Default: 1.

• quant_dtype (QuantDtype) – The datatype of quantization, supporting 4 and 8bits. Default: QuantDtype.INT8.

• symmetric (bool) – Whether the quantization algorithm is symmetric or not. Default: False.

• narrow_range (bool) – Whether the quantization algorithm uses narrow range or not. Default: False.

• quant_delay (int) – Quantization delay parameters according to the global step. Default: 0.

• neg_trunc (bool) – Whether the quantization algorithm uses negative truncation or not. Default: False.

• mode (str) – Optional quantization mode, currently only DEFAULT`(QAT) and `LEARNED_SCALE are supported. Default: (“DEFAULT”)

Inputs:

• x (Tensor) - The input of FakeQuantWithMinMaxObserver. The input dimension is preferably 2D or 4D.

Outputs:

Tensor, with the same type and shape as the x.

Raises

• TypeError – If min_init or max_init is not int, float or list.

• TypeError – If quant_delay is not an int.

• ValueError – If quant_delay is less than 0.

• ValueError – If min_init is not less than max_init.

• ValueError – If mode is neither DEFAULT nor LEARNED_SCALE.

• ValueError – If mode is LEARNED_SCALE and symmetric is not True.

• ValueError – If mode is LEARNED_SCALE, and narrow_range is not True unless when neg_trunc is True.

Supported Platforms:

Ascend GPU

Examples

>>> import mindspore
>>> from mindspore import Tensor
>>> fake_quant = nn.FakeQuantWithMinMaxObserver()
>>> x = Tensor(np.array([[1, 2, 1], [-2, 0, -1]]), mindspore.float32)
>>> result = fake_quant(x)
>>> print(result)
[[ 0.9882355  1.9764705  0.9882355]
 [-1.9764705  0.        -0.9882355]]

extend_repr()

Display instance object as string.

reset(quant_dtype=<QuantDtype.INT8: 'INT8'>, min_init=-6, max_init=6)

Reset the quant max parameter (eg. 256) and the initial value of the minq parameter and maxq parameter, this function is currently only valid for LEARNED_SCALE mode.

class tinyms.layers.Conv2dBnFoldQuantOneConv(in_channels, out_channels, kernel_size, stride=1, pad_mode='same', padding=0, dilation=1, group=1, eps=1e-05, momentum=0.997, has_bias=False, weight_init='normal', bias_init='zeros', beta_init='zeros', gamma_init='ones', mean_init='zeros', var_init='ones', fake=True, quant_config=QuantConfig(weight=functools.partial(<class 'mindspore.nn.layer.quant.FakeQuantWithMinMaxObserver'>), activation=functools.partial(<class 'mindspore.nn.layer.quant.FakeQuantWithMinMaxObserver'>)), quant_dtype=<QuantDtype.INT8: 'INT8'>)

2D convolution which use the convolution layer statistics once to calculate Batch Normalization operation folded construct.

This part is a more detailed overview of Conv2d operation. For more details about Quantization, please refer to the implementation of class of FakeQuantWithMinMaxObserver, FakeQuantWithMinMaxObserver.

\[ \begin{align}\begin{aligned}w_{q}=quant(\frac{w}{\sqrt{var_{G}+\epsilon}}*\gamma )\\b=\frac{-\mu _{G} }{\sqrt{var_{G}+\epsilon }}*\gamma +\beta\\y=w_{q}\times x+b\end{aligned}\end{align} \]

where \(quant\) is the continuous execution of quant and dequant, you can refer to the implementation of subclass of FakeQuantWithMinMaxObserver, mindspore.nn.FakeQuantWithMinMaxObserver. mu _{G} and var_{G} represent the global mean and variance respectively.

Parameters

• in_channels (int) – The number of input channel \(C_{in}\).

• out_channels (int) – The number of output channel \(C_{out}\).

• kernel_size (Union[int, tuple[int]]) – Specifies the height and width of the 2D convolution window.

• stride (Union[int, tuple[int]]) – Specifies stride for all spatial dimensions with the same value. Default: 1.

• pad_mode (str) – Specifies padding mode. The optional values are “same”, “valid”, “pad”. Default: “same”.

• padding (Union[int, tuple[int]]) – Implicit paddings on both sides of the x. Default: 0.

• dilation (Union[int, tuple[int]]) – Specifies the dilation rate to use for dilated convolution. Default: 1.

• group (int) – Splits filter into groups, in_ channels and out_channels must be divisible by the number of groups. Default: 1.

• eps (float) – Parameters for Batch Normalization. Default: 1e-5.

• momentum (float) – Parameters for Batch Normalization op. Default: 0.997.

• has_bias (bool) – Specifies whether the layer uses a bias vector, which is temporarily invalid. Default: False.

• weight_init (Union[Tensor, str, Initializer, numbers.Number]) – Initializer for the convolution kernel. Default: ‘normal’.

• bias_init (Union[Tensor, str, Initializer, numbers.Number]) – Initializer for the bias vector. Default: ‘zeros’.

• beta_init (Union[Tensor, str, Initializer, numbers.Number]) – Initializer for the beta vector. Default: ‘zeros’.

• gamma_init (Union[Tensor, str, Initializer, numbers.Number]) – Initializer for the gamma vector. Default: ‘ones’.

• mean_init (Union[Tensor, str, Initializer, numbers.Number]) – Initializer for the mean vector. Default: ‘zeros’.

• var_init (Union[Tensor, str, Initializer, numbers.Number]) – Initializer for the variance vector. Default: ‘ones’.

• fake (bool) – Whether Conv2dBnFoldQuant Cell adds FakeQuantWithMinMaxObserver. Default: True.

• quant_config (QuantConfig) – Configures the types of quant observer and quant settings of weight and activation. Note that, QuantConfig is a special namedtuple, which is designed for quantization and can be generated by mindspore.compression.quant.create_quant_config() method. Default: QuantConfig with both items set to default FakeQuantWithMinMaxObserver.

• quant_dtype (QuantDtype) – Specifies the FakeQuant datatype. Default: QuantDtype.INT8.

Inputs:

• x (Tensor) - Tensor of shape \((N, C_{in}, H_{in}, W_{in})\).

Outputs:

Tensor of shape \((N, C_{out}, H_{out}, W_{out})\).

Raises

• TypeError – If in_channels, out_channels or group is not an int.

• TypeError – If kernel_size, stride, padding or dilation is neither an int nor a tuple.

• TypeError – If has_bias or fake is not a bool.

• TypeError – If data_format is not a string.

• ValueError – If in_channels, out_channels, kernel_size, stride or dilation is less than 1.

• ValueError – If padding is less than 0.

• ValueError – If pad_mode is not one of ‘same’, ‘valid’, ‘pad’.

Supported Platforms:

Ascend GPU

Examples

>>> import mindspore
>>> from mindspore.compression import quant
>>> from mindspore import Tensor
>>> qconfig = quant.create_quant_config()
>>> conv2d_bnfold = nn.Conv2dBnFoldQuantOneConv(1, 1, kernel_size=(2, 2), stride=(1, 1), pad_mode="valid",
...                                             weight_init="ones", quant_config=qconfig)
>>> x = Tensor(np.array([[[[1, 0, 3], [1, 4, 7], [2, 5, 2]]]]), mindspore.float32)
>>> result = conv2d_bnfold(x)
>>> print(result)
[[[[5.9296875 13.8359375]
   [11.859375 17.78125]]]]

extend_repr()

Display instance object as string.

class tinyms.layers.Conv2dBnFoldQuant(in_channels, out_channels, kernel_size, stride=1, pad_mode='same', padding=0, dilation=1, group=1, eps=1e-05, momentum=0.997, has_bias=False, weight_init='normal', bias_init='zeros', beta_init='zeros', gamma_init='ones', mean_init='zeros', var_init='ones', fake=True, quant_config=QuantConfig(weight=functools.partial(<class 'mindspore.nn.layer.quant.FakeQuantWithMinMaxObserver'>), activation=functools.partial(<class 'mindspore.nn.layer.quant.FakeQuantWithMinMaxObserver'>)), quant_dtype=<QuantDtype.INT8: 'INT8'>, freeze_bn=100000)

2D convolution with Batch Normalization operation folded construct.

This part is a more detailed overview of Conv2d operation. For more details about Quantization, please refer to the implementation of class of FakeQuantWithMinMaxObserver, FakeQuantWithMinMaxObserver.

\[ \begin{align}\begin{aligned}y = x\times w+ b\\w_{q}=quant(\frac{w}{\sqrt{Var[y]+\epsilon}}*\gamma )\\y_{out}= w_{q}\times x+\frac{b-E[y]}{\sqrt{Var[y]+\epsilon}}*\gamma +\beta\end{aligned}\end{align} \]

where \(quant\) is the continuous execution of quant and dequant. Two convolution and Batch Normalization operation are used here, the purpose of the first convolution and Batch Normalization is to count the mean E[y] and variance Var[y] of current batch output for quantization.

Parameters

• in_channels (int) – The number of input channel \(C_{in}\).

• out_channels (int) – The number of output channel \(C_{out}\).

• kernel_size (Union[int, tuple[int]]) – Specifies the height and width of the 2D convolution window.

• stride (Union[int, tuple[int]]) – Specifies stride for all spatial dimensions with the same value. Default: 1.

• pad_mode (str) – Specifies padding mode. The optional values are “same”, “valid”, “pad”. Default: “same”.

• padding (Union[int, tuple[int]]) – Implicit paddings on both sides of the x. Default: 0.

• dilation (Union[int, tuple[int]]) – Specifies the dilation rate to use for dilated convolution. Default: 1.

• group (int) – Splits filter into groups, in_ channels and out_channels must be divisible by the number of groups. Default: 1.

• eps (float) – Parameters for Batch Normalization. Default: 1e-5.

• momentum (float) – Parameters for Batch Normalization op. Default: 0.997.

• has_bias (bool) – Specifies whether the layer uses a bias vector. Default: False.

• weight_init (Union[Tensor, str, Initializer, numbers.Number]) – Initializer for the convolution kernel. Default: ‘normal’.

• bias_init (Union[Tensor, str, Initializer, numbers.Number]) – Initializer for the bias vector. Default: ‘zeros’.

• beta_init (Union[Tensor, str, Initializer, numbers.Number]) – Initializer for the beta vector. Default: ‘zeros’.

• gamma_init (Union[Tensor, str, Initializer, numbers.Number]) – Initializer for the gamma vector. Default: ‘ones’.

• mean_init (Union[Tensor, str, Initializer, numbers.Number]) – Initializer for the mean vector. Default: ‘zeros’.

• var_init (Union[Tensor, str, Initializer, numbers.Number]) – Initializer for the variance vector. Default: ‘ones’.

• fake (bool) – Whether Conv2dBnFoldQuant Cell adds FakeQuantWithMinMaxObserver. Default: True.

• quant_config (QuantConfig) – Configures the types of quant observer and quant settings of weight and activation. Note that, QuantConfig is a special namedtuple, which is designed for quantization and can be generated by mindspore.compression.quant.create_quant_config() method. Default: QuantConfig with both items set to default FakeQuantWithMinMaxObserver.

• quant_dtype (QuantDtype) – Specifies the FakeQuant datatype. Default: QuantDtype.INT8.

• freeze_bn (int) – The quantization freeze Batch Normalization op is according to the global step. Default: 100000.

Inputs:

• x (Tensor) - Tensor of shape \((N, C_{in}, H_{in}, W_{in})\).

Outputs:

Tensor of shape \((N, C_{out}, H_{out}, W_{out})\).

Raises

• TypeError – If in_channels, out_channels or group is not an int.

• TypeError – If kernel_size, stride, padding or dilation is neither an int nor a tuple.

• TypeError – If has_bias or fake is not a bool.

• ValueError – If in_channels, out_channels, kernel_size, stride or dilation is less than 1.

• ValueError – If padding is less than 0.

• ValueError – If pad_mode is not one of ‘same’, ‘valid’, ‘pad’.

• ValueError – If device_target in context is neither Ascend nor GPU.

Supported Platforms:

Ascend GPU

Examples

>>> import mindspore
>>> from mindspore.compression import quant
>>> from mindspore import Tensor
>>> qconfig = quant.create_quant_config()
>>> conv2d_bnfold = nn.Conv2dBnFoldQuant(1, 1, kernel_size=(2, 2), stride=(1, 1), pad_mode="valid",
...                                      weight_init="ones", quant_config=qconfig)
>>> x = Tensor(np.array([[[[1, 0, 3], [1, 4, 7], [2, 5, 2]]]]), mindspore.float32)
>>> result = conv2d_bnfold(x)
>>> print(result)
[[[[5.9296875 13.8359375]
   [11.859375 17.78125]]]]

extend_repr()

Display instance object as string.

class tinyms.layers.Conv2dBnWithoutFoldQuant(in_channels, out_channels, kernel_size, stride=1, pad_mode='same', padding=0, dilation=1, group=1, has_bias=False, eps=1e-05, momentum=0.997, weight_init='normal', bias_init='zeros', quant_config=QuantConfig(weight=functools.partial(<class 'mindspore.nn.layer.quant.FakeQuantWithMinMaxObserver'>), activation=functools.partial(<class 'mindspore.nn.layer.quant.FakeQuantWithMinMaxObserver'>)), quant_dtype=<QuantDtype.INT8: 'INT8'>)

2D convolution and batchnorm without fold with fake quantized construct.

This part is a more detailed overview of Conv2d operation. For more details about Quantization, please refer to the implementation of class of FakeQuantWithMinMaxObserver, mindspore.nn.FakeQuantWithMinMaxObserver.

\[ \begin{align}\begin{aligned}y =x\times quant(w)+ b\\y_{bn} =\frac{y-E[y] }{\sqrt{Var[y]+ \epsilon } } *\gamma + \beta\end{aligned}\end{align} \]

where \(quant\) is the continuous execution of quant and dequant, you can refer to the implementation of class of FakeQuantWithMinMaxObserver, mindspore.nn.FakeQuantWithMinMaxObserver.

Parameters

• in_channels (int) – The number of input channel \(C_{in}\).

• out_channels (int) – The number of output channel \(C_{out}\).

• kernel_size (Union[int, tuple[int]]) – Specifies the height and width of the 2D convolution window.

• stride (Union[int, tuple[int]]) – Specifies stride for all spatial dimensions with the same value. Default: 1.

• pad_mode (str) – Specifies padding mode. The optional values are “same”, “valid”, “pad”. Default: “same”.

• padding (Union[int, tuple[int]]) – Implicit paddings on both sides of the x. Default: 0.

• dilation (Union[int, tuple[int]]) – Specifies the dilation rate to use for dilated convolution. Default: 1.

• group (int) – Splits filter into groups, in_ channels and out_channels must be divisible by the number of groups. Default: 1.

• has_bias (bool) – Specifies whether the layer uses a bias vector. Default: False.

• eps (float) – Parameters for Batch Normalization. Default: 1e-5.

• momentum (float) – Parameters for Batch Normalization op. Default: 0.997.

• weight_init (Union[Tensor, str, Initializer, numbers.Number]) – Initializer for the convolution kernel. Default: ‘normal’.

• bias_init (Union[Tensor, str, Initializer, numbers.Number]) – Initializer for the bias vector. Default: ‘zeros’.

• quant_config (QuantConfig) – Configures the types of quant observer and quant settings of weight and activation. Note that, QuantConfig is a special namedtuple, which is designed for quantization and can be generated by mindspore.compression.quant.create_quant_config() method. Default: QuantConfig with both items set to default FakeQuantWithMinMaxObserver.

• quant_dtype (QuantDtype) – Specifies the FakeQuant datatype. Default: QuantDtype.INT8.

Inputs:

• x (Tensor) - Tensor of shape \((N, C_{in}, H_{in}, W_{in})\).

Outputs:

Tensor of shape \((N, C_{out}, H_{out}, W_{out})\).

Supported Platforms:

Ascend GPU

Raises

• TypeError – If in_channels, out_channels or group is not an int.

• TypeError – If kernel_size, stride, padding or dilation is neither an int nor a tuple.

• TypeError – If has_bias is not a bool.

• ValueError – If in_channels, out_channels, kernel_size, stride or dilation is less than 1.

• ValueError – If padding is less than 0.

• ValueError – If pad_mode is not one of ‘same’, ‘valid’, ‘pad’.

Examples

>>> import mindspore
>>> from mindspore.compression import quant
>>> from mindspore import Tensor
>>> qconfig = quant.create_quant_config()
>>> conv2d_no_bnfold = nn.Conv2dBnWithoutFoldQuant(1, 1, kernel_size=(2, 2), stride=(1, 1), pad_mode="valid",
...                                                weight_init='ones', quant_config=qconfig)
>>> x = Tensor(np.array([[[[1, 0, 3], [1, 4, 7], [2, 5, 2]]]]), mindspore.float32)
>>> result = conv2d_no_bnfold(x)
>>> print(result)
[[[[5.929658  13.835868]
   [11.859316  17.78116]]]]

extend_repr()

Display instance object as string.

class tinyms.layers.Conv2dQuant(in_channels, out_channels, kernel_size, stride=1, pad_mode='same', padding=0, dilation=1, group=1, has_bias=False, weight_init='normal', bias_init='zeros', quant_config=QuantConfig(weight=functools.partial(<class 'mindspore.nn.layer.quant.FakeQuantWithMinMaxObserver'>), activation=functools.partial(<class 'mindspore.nn.layer.quant.FakeQuantWithMinMaxObserver'>)), quant_dtype=<QuantDtype.INT8: 'INT8'>)

2D convolution with fake quantized operation layer.

This part is a more detailed overview of Conv2d operation. For more details about Quantization, please refer to the implementation of class of FakeQuantWithMinMaxObserver, mindspore.nn.FakeQuantWithMinMaxObserver.

Parameters

• in_channels (int) – The number of input channel \(C_{in}\).

• out_channels (int) – The number of output channel \(C_{out}\).

• kernel_size (Union[int, tuple[int]]) – Specifies the height and width of the 2D convolution window.

• stride (Union[int, tuple[int]]) – Specifies stride for all spatial dimensions with the same value. Default: 1.

• pad_mode (str) – Specifies padding mode. The optional values are “same”, “valid”, “pad”. Default: “same”.

• padding (Union[int, tuple[int]]) – Implicit paddings on both sides of the x. Default: 0.

• dilation (Union[int, tuple[int]]) – Specifies the dilation rate to use for dilated convolution. Default: 1.

• group (int) – Splits filter into groups, in_ channels and out_channels must be divisible by the number of groups. Default: 1.

• has_bias (bool) – Specifies whether the layer uses a bias vector. Default: False.

• weight_init (Union[Tensor, str, Initializer, numbers.Number]) – Initializer for the convolution kernel. Default: ‘normal’.

• bias_init (Union[Tensor, str, Initializer, numbers.Number]) – Initializer for the bias vector. Default: ‘zeros’.

• quant_config (QuantConfig) – Configures the types of quant observer and quant settings of weight and activation. Note that, QuantConfig is a special namedtuple, which is designed for quantization and can be generated by mindspore.compression.quant.create_quant_config() method. Default: QuantConfig with both items set to default FakeQuantWithMinMaxObserver.

• quant_dtype (QuantDtype) – Specifies the FakeQuant datatype. Default: QuantDtype.INT8.

Inputs:

• x (Tensor) - Tensor of shape \((N, C_{in}, H_{in}, W_{in})\). The input dimension is preferably 2D or 4D.

Outputs:

Tensor of shape \((N, C_{out}, H_{out}, W_{out})\).

Raises

• TypeError – If in_channels, out_channels or group is not an int.

• TypeError – If kernel_size, stride, padding or dilation is neither an int nor a tuple.

• TypeError – If has_bias is not a bool.

• ValueError – If in_channels, out_channels, kernel_size, stride or dilation is less than 1.

• ValueError – If padding is less than 0.

• ValueError – If pad_mode is not one of ‘same’, ‘valid’, ‘pad’.

Supported Platforms:

Ascend GPU

Examples

>>> import mindspore
>>> from mindspore.compression import quant
>>> from mindspore import Tensor
>>> qconfig = quant.create_quant_config()
>>> conv2d_quant = nn.Conv2dQuant(1, 1, kernel_size=(2, 2), stride=(1, 1), pad_mode="valid",
...                               weight_init='ones', quant_config=qconfig)
>>> x = Tensor(np.array([[[[1, 0, 3], [1, 4, 7], [2, 5, 2]]]]), mindspore.float32)
>>> result = conv2d_quant(x)
>>> print(result)
[[[[5.9296875  13.8359375]
   [11.859375  17.78125]]]]

extend_repr()

Display instance object as string.

class tinyms.layers.DenseQuant(in_channels, out_channels, weight_init='normal', bias_init='zeros', has_bias=True, activation=None, quant_config=QuantConfig(weight=functools.partial(<class 'mindspore.nn.layer.quant.FakeQuantWithMinMaxObserver'>), activation=functools.partial(<class 'mindspore.nn.layer.quant.FakeQuantWithMinMaxObserver'>)), quant_dtype=<QuantDtype.INT8: 'INT8'>)

The fully connected layer with fake quantized operation.

This part is a more detailed overview of Dense operation. For more details about Quantization, please refer to the implementation of class of FakeQuantWithMinMaxObserver, mindspore.nn.FakeQuantWithMinMaxObserver.

Parameters

• in_channels (int) – The dimension of the input space.

• out_channels (int) – The dimension of the output space.

• weight_init (Union[Tensor, str, Initializer, numbers.Number]) – The trainable weight_init parameter. The dtype is same as x. The values of str refer to the function initializer. Default: ‘normal’.

• bias_init (Union[Tensor, str, Initializer, numbers.Number]) – The trainable bias_init parameter. The dtype is same as x. The values of str refer to the function initializer. Default: ‘zeros’.

• has_bias (bool) – Specifies whether the layer uses a bias vector. Default: True.

• activation (Union[str, Cell, Primitive]) – The regularization function applied to the output of the layer, eg. ‘relu’. Default: None.

• quant_config (QuantConfig) – Configures the types of quant observer and quant settings of weight and activation. Note that, QuantConfig is a special namedtuple, which is designed for quantization and can be generated by mindspore.compression.quant.create_quant_config() method. Default: QuantConfig with both items set to default FakeQuantWithMinMaxObserver.

• quant_dtype (QuantDtype) – Specifies the FakeQuant datatype. Default: QuantDtype.INT8.

Inputs:

• x (Tensor) - Tensor of shape \((N, C_{in}, H_{in}, W_{in})\). The input dimension is preferably 2D or 4D.

Outputs:

Tensor of shape \((N, C_{out}, H_{out}, W_{out})\).

Raises

• TypeError – If in_channels, out_channels is not an int.

• TypeError – If has_bias is not a bool.

• TypeError – If activation is not str, Cell and Primitive.

• ValueError – If in_channels or out_channels is less than 1.

• ValueError – If the dims of weight_init is not equal to 2 or the first element of weight_init is not equal to out_channels or the second element of weight_init is not equal to in_channels.

• ValueError – If the dims of bias_init is not equal to 1 or the element of bias_init is not equal to out_channels.

Supported Platforms:

Ascend GPU

Examples

>>> import mindspore
>>> from mindspore.compression import quant
>>> from mindspore import Tensor
>>> qconfig = quant.create_quant_config()
>>> dense_quant = nn.DenseQuant(2, 1, weight_init='ones', quant_config=qconfig)
>>> x = Tensor(np.array([[1, 5], [3, 4]]), mindspore.float32)
>>> result = dense_quant(x)
>>> print(result)
[[5.929413]
 [6.9176483]]

construct(x)

Use operators to construct the Dense layer.

extend_repr()

A pretty print for Dense layer.

class tinyms.layers.ActQuant(activation, ema=False, ema_decay=0.999, fake_before=False, quant_config=QuantConfig(weight=functools.partial(<class 'mindspore.nn.layer.quant.FakeQuantWithMinMaxObserver'>), activation=functools.partial(<class 'mindspore.nn.layer.quant.FakeQuantWithMinMaxObserver'>)), quant_dtype=<QuantDtype.INT8: 'INT8'>)

Quantization aware training activation function.

Add the fake quantized operation to the end of activation operation, by which the output of activation operation will be truncated. For more details about Quantization, please refer to the implementation of subclass of FakeQuantWithMinMaxObserver, mindspore.nn.FakeQuantWithMinMaxObserver.

Parameters

• activation (Cell) – Activation cell.

• ema (bool) – The exponential Moving Average algorithm updates min and max. Default: False.

• ema_decay (float) – Exponential Moving Average algorithm parameter. Default: 0.999.

• fake_before (bool) – Whether add fake quantized operation before activation. Default: False.

• quant_config (QuantConfig) – Configures the types of quant observer and quant settings of weight and activation. Note that, QuantConfig is a special namedtuple, which is designed for quantization and can be generated by mindspore.compression.quant.create_quant_config() method. Default: QuantConfig with both items set to default FakeQuantWithMinMaxObserver.

• quant_dtype (QuantDtype) – Specifies the FakeQuant datatype. Default: QuantDtype.INT8.

Inputs:

• x (Tensor) - The input of ActQuant. The input dimension is preferably 2D or 4D.

Outputs:

Tensor, with the same type and shape as the x.

Raises

• TypeError – If activation is not an instance of Cell.

• TypeError – If fake_before is not a bool.

Supported Platforms:

Ascend GPU

Examples

>>> import mindspore
>>> from mindspore.compression import quant
>>> from mindspore import Tensor
>>> qconfig = quant.create_quant_config()
>>> act_quant = nn.ActQuant(nn.ReLU(), quant_config=qconfig)
>>> x = Tensor(np.array([[1, 2, -1], [-2, 0, -1]]), mindspore.float32)
>>> result = act_quant(x)
>>> print(result)
[[0.9882355 1.9764705 0.       ]
 [0.        0.        0.       ]]

class tinyms.layers.TensorAddQuant(ema_decay=0.999, quant_config=QuantConfig(weight=functools.partial(<class 'mindspore.nn.layer.quant.FakeQuantWithMinMaxObserver'>), activation=functools.partial(<class 'mindspore.nn.layer.quant.FakeQuantWithMinMaxObserver'>)), quant_dtype=<QuantDtype.INT8: 'INT8'>)

Adds fake quantized operation after TensorAdd operation.

This part is a more detailed overview of TensorAdd operation. For more details about Quantization, please refer to the implementation of class of FakeQuantWithMinMaxObserver, mindspore.nn.FakeQuantWithMinMaxObserver.

Parameters

• ema_decay (float) – Exponential Moving Average algorithm parameter. Default: 0.999.

• quant_config (QuantConfig) – Configures the types of quant observer and quant settings of weight and activation. Note that, QuantConfig is a special namedtuple, which is designed for quantization and can be generated by mindspore.compression.quant.create_quant_config() method. Default: QuantConfig with both items set to default FakeQuantWithMinMaxObserver.

• quant_dtype (QuantDtype) – Specifies the FakeQuant datatype. Default: QuantDtype.INT8.

Inputs:

• x1 (Tensor) - The first tensor of TensorAddQuant. The input dimension is preferably 2D or 4D.

• x2 (Tensor) - The second tensor of TensorAddQuant. Has the same shape with x1.

Outputs:

Tensor, with the same type and shape as the x1.

Raises

• TypeError – If ema_decay is not a float.

• ValueError – If the shape of x2 is different with x1.

Supported Platforms:

Ascend GPU

Examples

>>> import mindspore
>>> from mindspore.compression import quant
>>> from mindspore import Tensor
>>> qconfig = quant.create_quant_config()
>>> add_quant = nn.TensorAddQuant(quant_config=qconfig)
>>> x1 = Tensor(np.array([[1, 2, 1], [-2, 0, -1]]), mindspore.float32)
>>> x2 = Tensor(np.ones((2, 3)), mindspore.float32)
>>> output = add_quant(x1, x2)
>>> print(output)
[[ 1.9764705  3.011765   1.9764705]
 [-0.9882355  0.9882355  0.       ]]

class tinyms.layers.MulQuant(ema_decay=0.999, quant_config=QuantConfig(weight=functools.partial(<class 'mindspore.nn.layer.quant.FakeQuantWithMinMaxObserver'>), activation=functools.partial(<class 'mindspore.nn.layer.quant.FakeQuantWithMinMaxObserver'>)), quant_dtype=<QuantDtype.INT8: 'INT8'>)

Adds fake quantized operation after Mul operation.

This part is a more detailed overview of Mul operation. For more details about Quantization, please refer to the implementation of class of FakeQuantWithMinMaxObserver, mindspore.nn.FakeQuantWithMinMaxObserver.

Parameters

• ema_decay (float) – Exponential Moving Average algorithm parameter. Default: 0.999.

• quant_config (QuantConfig) – Configures the types of quant observer and quant settings of weight and activation. Note that, QuantConfig is a special namedtuple, which is designed for quantization and can be generated by mindspore.compression.quant.create_quant_config() method. Default: QuantConfig with both items set to default FakeQuantWithMinMaxObserver.

• quant_dtype (QuantDtype) – Specifies the FakeQuant datatype. Default: QuantDtype.INT8.

Inputs:

• x1 (Tensor) - The first tensor of MulQuant. The input dimension is preferably 2D or 4D.

• x2 (Tensor) - The second tensor of MulQuant. Has the same shape with x1.

Outputs:

Tensor, with the same type and shape as the x1.

Raises

• TypeError – If ema_decay is not a float.

• ValueError – If the shape of x2 is different with x1.

Supported Platforms:

Ascend GPU

Examples

>>> import mindspore
>>> from mindspore.compression import quant
>>> from mindspore import Tensor
>>> qconfig = quant.create_quant_config()
>>> mul_quant = nn.MulQuant(quant_config=qconfig)
>>> x1 = Tensor(np.array([[1, 2, 1], [-2, 0, -1]]), mindspore.float32)
>>> x2 = Tensor(np.ones((2, 3)) * 2, mindspore.float32)
>>> output = mul_quant(x1, x2)
>>> print(output)
[[ 1.9764705  4.0000005  1.9764705]
 [-4.         0.        -1.9764705]]

class tinyms.layers.ReduceLogSumExp(axis, keep_dims=False)

Reduces a dimension of a tensor by calculating exponential for all elements in the dimension, then calculate logarithm of the sum.

\[ReduceLogSumExp(x) = \log(\sum(e^x))\]

Parameters

• axis (Union[int, tuple(int), list(int)]) – (), reduce all dimensions. Only constant value is allowed.

• keep_dims (bool) – If True, keep these reduced dimensions and the length is 1. If False, don’t keep these dimensions. Default : False.

Inputs:

• x (Tensor) - The input tensor. With float16 or float32 data type.

Outputs:

Tensor, has the same dtype as the x.

• If axis is (), and keep_dims is False, the output is a 0-D tensor representing the sum of all elements in the input tensor.

• If axis is int, set as 2, and keep_dims is False, the shape of output is \((x_1, x_3, ..., x_R)\).

• If axis is tuple(int), set as (2, 3), and keep_dims is False, the shape of output is \((x_1, x_4, ..., x_R)\).

Raises

• TypeError – If axis is not one of int, list, tuple.

• TypeError – If keep_dims is not bool.

• TypeError – If dtype of x is neither float16 nor float32.

Supported Platforms:

Ascend GPU CPU

Examples

>>> x = Tensor(np.random.randn(3, 4, 5, 6).astype(np.float32))
>>> op = nn.ReduceLogSumExp(1, keep_dims=True)
>>> output = op(x)
>>> print(output.shape)
(3, 1, 5, 6)

class tinyms.layers.Range(start, limit=None, delta=1)

Creates a sequence of numbers in range [start, limit) with step size delta.

The size of output is \(\left \lfloor \frac{limit-start}{delta} \right \rfloor + 1\) and delta is the gap between two values in the tensor.

\[out_{i+1} = out_{i} +delta\]

Parameters

• start (Union[int, float]) – If limit is None, the value acts as limit in the range and first entry defaults to 0. Otherwise, it acts as first entry in the range.

• limit (Union[int, float]) – Acts as upper limit of sequence. If None, defaults to the value of start while set the first entry of the range to 0. It can not be equal to start. Default: None.

• delta (Union[int, float]) – Increment of the range. It can not be equal to zero. Default: 1.

Outputs:

Tensor, the dtype is int if the dtype of start, limit and delta all are int. Otherwise, dtype is float.

Supported Platforms:

Ascend GPU CPU

Examples

>>> net = nn.Range(1, 8, 2)
>>> output = net()
>>> print(output)
[1 3 5 7]

class tinyms.layers.LGamma

Calculates LGamma using Lanczos’ approximation referring to “A Precision Approximation of the Gamma Function”. The algorithm is:

\[\begin{split}\begin{array}{ll} \\ lgamma(z + 1) = \frac{(\log(2) + \log(pi))}{2} + (z + 1/2) * log(t(z)) - t(z) + A(z) \\ t(z) = z + kLanczosGamma + 1/2 \\ A(z) = kBaseLanczosCoeff + \sum_{k=1}^n \frac{kLanczosCoefficients[i]}{z + k} \end{array}\end{split}\]

However, if the input is less than 0.5 use Euler’s reflection formula:

\[lgamma(x) = \log(pi) - lgamma(1-x) - \log(abs(sin(pi * x)))\]

And please note that

\[lgamma(+/-inf) = +inf\]

Thus, the behaviour of LGamma follows:

• when x > 0.5, return log(Gamma(x))

• when x < 0.5 and is not an integer, return the real part of Log(Gamma(x)) where Log is the complex logarithm

• when x is an integer less or equal to 0, return +inf

• when x = +/- inf, return +inf

Inputs:

• x (Tensor) - The input tensor. Only float16, float32 are supported.

Outputs:

Tensor, has the same shape and dtype as the x.

Raises

TypeError – If dtype of x is neither float16 nor float32.

Supported Platforms:

Ascend GPU

Examples

>>> x = Tensor(np.array([2, 3, 4]).astype(np.float32))
>>> op = nn.LGamma()
>>> output = op(x)
>>> print(output)
[3.5762787e-07 6.9314754e-01 1.7917603e+00]

class tinyms.layers.DiGamma

Calculates Digamma using Lanczos’ approximation referring to “A Precision Approximation of the Gamma Function”. The algorithm is:

\[\begin{split}\begin{array}{ll} \\ digamma(z + 1) = log(t(z)) + A'(z) / A(z) - kLanczosGamma / t(z) \\ t(z) = z + kLanczosGamma + 1/2 \\ A(z) = kBaseLanczosCoeff + \sum_{k=1}^n \frac{kLanczosCoefficients[i]}{z + k} \\ A'(z) = \sum_{k=1}^n \frac{kLanczosCoefficients[i]}{{z + k}^2} \end{array}\end{split}\]

However, if the input is less than 0.5 use Euler’s reflection formula:

\[digamma(x) = digamma(1 - x) - pi * cot(pi * x)\]

Inputs:

• x (Tensor[Number]) - The input tensor. Only float16, float32 are supported.

Outputs:

Tensor, has the same shape and dtype as the x.

Raises

TypeError – If dtype of x is neither float16 nor float32.

Supported Platforms:

Ascend GPU

Examples

>>> x = Tensor(np.array([2, 3, 4]).astype(np.float32))
>>> op = nn.DiGamma()
>>> output = op(x)
>>> print(output)
[0.42278463  0.92278427 1.2561178]

class tinyms.layers.IGamma

Calculates lower regularized incomplete Gamma function. The lower regularized incomplete Gamma function is defined as:

\[P(a, x) = gamma(a, x) / Gamma(a) = 1 - Q(a, x)\]

where

\[gamma(a, x) = \int_0^x t^{a-1} \exp^{-t} dt\]

is the lower incomplete Gamma function.

Above \(Q(a, x)\) is the upper regularized complete Gamma function.

Inputs:

• a (Tensor) - The input tensor. With float32 data type. a should have the same dtype with x.

• x (Tensor) - The input tensor. With float32 data type. x should have the same dtype with a.

Outputs:

Tensor, has the same dtype as a and x.

Raises

TypeError – If dtype of input x and a is not float16 nor float32, or if x has different dtype with a.

Supported Platforms:

Ascend GPU

Examples

>>> a = Tensor(np.array([2.0, 4.0, 6.0, 8.0]).astype(np.float32))
>>> x = Tensor(np.array([2.0, 3.0, 4.0, 5.0]).astype(np.float32))
>>> igamma = nn.IGamma()
>>> output = igamma(a, x)
>>> print (output)
[0.593994  0.35276785  0.21486944  0.13337152]

class tinyms.layers.LBeta

This method avoids the numeric cancellation by explicitly decomposing lgamma into the Stirling approximation and an explicit log_gamma_correction, and cancelling the large terms from the Striling analytically.

This is semantically equal to

\[P(x, y) = lgamma(x) + lgamma(y) - lgamma(x + y).\]

The method is more accurate for arguments above 8. The reason for accuracy loss in the naive computation is catastrophic cancellation between the lgammas.

Inputs:

• x (Tensor) - The input tensor. With float16 or float32 data type. x should have the same dtype with y.

• y (Tensor) - The input tensor. With float16 or float32 data type. y should have the same dtype with x.

Outputs:

Tensor, has the same dtype as x and y.

Raises

TypeError – If dtype of x or y is neither float16 nor float32, or if x has different dtype with y.

Supported Platforms:

Ascend GPU

Examples

>>> x = Tensor(np.array([2.0, 4.0, 6.0, 8.0]).astype(np.float32))
>>> y = Tensor(np.array([2.0, 3.0, 14.0, 15.0]).astype(np.float32))
>>> lbeta = nn.LBeta()
>>> output = lbeta(y, x)
>>> print(output)
[-1.7917596  -4.094345  -12.000229  -14.754799]

class tinyms.layers.MatMul(**kwargs)

Multiplies matrix x1 by matrix x2.

nn.MatMul will be deprecated in future versions. Please use ops.matmul instead.

• If both x1 and x2 are 1-dimensional, the dot product is returned.

• If the dimensions of x1 and x2 are all not greater than 2, the matrix-matrix product will be returned. Note if one of ‘x1’ and ‘x2’ is 1-dimensional, the argument will first be expanded to 2 dimension. After the matrix multiply, the expanded dimension will be removed.

• If at least one of x1 and x2 is N-dimensional (N>2), the none-matrix dimensions(batch) of inputs will be broadcasted and must be broadcastable. Note if one of ‘x1’ and ‘x2’ is 1-dimensional, the argument will first be expanded to 2 dimension and then the none-matrix dimensions will be broadcasted. after the matrix multiply, the expanded dimension will be removed. For example, if x1 is a \((j \times 1 \times n \times m)\) tensor and x2 is b \((k \times m \times p)\) tensor, the output will be a \((j \times k \times n \times p)\) tensor.

Parameters

• transpose_x1 (bool) – If true, a is transposed before multiplication. Default: False.

• transpose_x2 (bool) – If true, b is transposed before multiplication. Default: False.

Inputs:

• x1 (Tensor) - The first tensor to be multiplied.

• x2 (Tensor) - The second tensor to be multiplied.

Outputs:

Tensor, the shape of the output tensor depends on the dimension of input tensors.

Raises

• TypeError – If transpose_x1 or transpose_x2 is not a bool.

• ValueError – If the column of matrix dimensions of x1 is not equal to the row of matrix dimensions of x2.

Supported Platforms:

Ascend GPU CPU

Examples

>>> net = nn.MatMul()
>>> x1 = Tensor(np.ones(shape=[3, 2, 3]), mindspore.float32)
>>> x2 = Tensor(np.ones(shape=[3, 4]), mindspore.float32)
>>> output = net(x1, x2)
>>> print(output.shape)
(3, 2, 4)

class tinyms.layers.Moments(axis=None, keep_dims=None)

Calculates the mean and variance of x.

The mean and variance are calculated by aggregating the contents of input_x across axes. If input_x is 1-D and axes = [0] this is just the mean and variance of a vector.

Parameters

• axis (Union[int, tuple(int)]) – Calculates the mean and variance along the specified axis. Default: None.

• keep_dims (bool) – If true, The dimension of mean and variance are identical with input’s. If false, don’t keep these dimensions. Default: None.

Inputs:

• x (Tensor) - The tensor to be calculated. Only float16 and float32 are supported. \((N,*)\) where \(*\) means,any number of additional dimensions.

Outputs:

• mean (Tensor) - The mean of x, with the same date type as input x.

• variance (Tensor) - The variance of x, with the same date type as input x.

Raises

• TypeError – If axis is not one of int, tuple, None.

• TypeError – If keep_dims is neither bool nor None.

• TypeError – If dtype of x is neither float16 nor float32.

Supported Platforms:

Ascend GPU CPU

Examples

>>> x = Tensor(np.array([[[[1, 2, 3, 4], [3, 4, 5, 6]]]]), mindspore.float32)
>>> net = nn.Moments(axis=0, keep_dims=True)
>>> output = net(x)
>>> print(output)
(Tensor(shape=[1, 1, 2, 4], dtype=Float32, value=
[[[[ 1.00000000e+00, 2.00000000e+00, 3.00000000e+00, 4.00000000e+00],
   [ 3.00000000e+00, 4.00000000e+00, 5.00000000e+00, 6.00000000e+00]]]]),
Tensor(shape=[1, 1, 2, 4], dtype=Float32, value=
[[[[ 0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00],
   [ 0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00]]]]))
>>> net = nn.Moments(axis=1, keep_dims=True)
>>> output = net(x)
>>> print(output)
(Tensor(shape=[1, 1, 2, 4], dtype=Float32, value=
[[[[ 1.00000000e+00, 2.00000000e+00, 3.00000000e+00, 4.00000000e+00],
   [ 3.00000000e+00, 4.00000000e+00, 5.00000000e+00, 6.00000000e+00]]]]),
Tensor(shape=[1, 1, 2, 4], dtype=Float32, value=
[[[[ 0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00],
   [ 0.00000000e+00, 0.00000000e+00, 0.00000000e+00, 0.00000000e+00]]]]))
>>> net = nn.Moments(axis=2, keep_dims=True)
>>> output = net(x)
>>> print(output)
(Tensor(shape=[1, 1, 1, 4], dtype=Float32, value=
[[[[ 2.00000000e+00, 3.00000000e+00, 4.00000000e+00, 5.00000000e+00]]]]),
Tensor(shape=[1, 1, 1, 4], dtype=Float32, value=
[[[[ 1.00000000e+00, 1.00000000e+00, 1.00000000e+00, 1.00000000e+00]]]]))
>>> net = nn.Moments(axis=3, keep_dims=True)
>>> output = net(x)
>>> print(output)
(Tensor(shape=[1, 1, 2, 1], dtype=Float32, value=
[[[[ 2.50000000e+00],
   [ 4.50000000e+00]]]]), Tensor(shape=[1, 1, 2, 1], dtype=Float32, value=
[[[[ 1.25000000e+00],
   [ 1.25000000e+00]]]]))

class tinyms.layers.MatInverse

Calculates the inverse of Positive-Definite Hermitian matrix using Cholesky decomposition.

Inputs:

• x (Tensor[Number]) - The input tensor. It must be a positive-definite matrix. With float16 or float32 data type.

Outputs:

Tensor, has the same dtype as the x.

Raises

TypeError – If dtype of x is neither float16 nor float32.

Supported Platforms:

GPU

Examples

>>> x = Tensor(np.array([[4, 12, -16], [12, 37, -43], [-16, -43, 98]]).astype(np.float32))
>>> op = nn.MatInverse()
>>> output = op(x)
>>> print(output)
[[49.36112  -13.555558  2.1111116]
 [-13.555558  3.7777784  -0.5555557]
 [2.1111116  -0.5555557  0.11111113]]

class tinyms.layers.MatDet

Calculates the determinant of Positive-Definite Hermitian matrix using Cholesky decomposition.

Inputs:

• x (Tensor[Number]) - The input tensor. It must be a positive-definite matrix. With float16 or float32 data type.

Outputs:

Tensor, has the same dtype as the x.

Raises

TypeError – If dtype of x is neither float16 nor float32.

Supported Platforms:

GPU

Examples

>>> x = Tensor(np.array([[4, 12, -16], [12, 37, -43], [-16, -43, 98]]).astype(np.float32))
>>> op = nn.MatDet()
>>> output = op(x)
>>> print(output)
35.999996

class tinyms.layers.Conv2dBnAct(in_channels, out_channels, kernel_size, stride=1, pad_mode='same', padding=0, dilation=1, group=1, has_bias=False, weight_init='normal', bias_init='zeros', has_bn=False, momentum=0.997, eps=1e-05, activation=None, alpha=0.2, after_fake=True)

A combination of convolution, Batchnorm, and activation layer.

This part is a more detailed overview of Conv2d operation.

Parameters

• in_channels (int) – The number of input channel \(C_{in}\).

• out_channels (int) – The number of output channel \(C_{out}\).

• kernel_size (Union[int, tuple]) – The data type is int or a tuple of 2 integers. Specifies the height and width of the 2D convolution window. Single int means the value is for both height and width of the kernel. A tuple of 2 ints means the first value is for the height and the other is for the width of the kernel.

• stride (int) – Specifies stride for all spatial dimensions with the same value. The value of stride must be greater than or equal to 1 and lower than any one of the height and width of the x. Default: 1.

• pad_mode (str) – Specifies padding mode. The optional values are “same”, “valid”, “pad”. Default: “same”.

• padding (int) – Implicit paddings on both sides of the x. Default: 0.

• dilation (int) – Specifies the dilation rate to use for dilated convolution. If set to be \(k > 1\), there will be \(k - 1\) pixels skipped for each sampling location. Its value must be greater than or equal to 1 and lower than any one of the height and width of the x. Default: 1.

• group (int) – Splits filter into groups, in_ channels and out_channels must be divisible by the number of groups. Default: 1.

• has_bias (bool) – Specifies whether the layer uses a bias vector. Default: False.

• weight_init (Union[Tensor, str, Initializer, numbers.Number]) – Initializer for the convolution kernel. It can be a Tensor, a string, an Initializer or a number. When a string is specified, values from ‘TruncatedNormal’, ‘Normal’, ‘Uniform’, ‘HeUniform’ and ‘XavierUniform’ distributions as well as constant ‘One’ and ‘Zero’ distributions are possible. Alias ‘xavier_uniform’, ‘he_uniform’, ‘ones’ and ‘zeros’ are acceptable. Uppercase and lowercase are both acceptable. Refer to the values of Initializer for more details. Default: ‘normal’.

• bias_init (Union[Tensor, str, Initializer, numbers.Number]) – Initializer for the bias vector. Possible Initializer and string are the same as ‘weight_init’. Refer to the values of Initializer for more details. Default: ‘zeros’.

• has_bn (bool) – Specifies to used batchnorm or not. Default: False.

• momentum (float) – Momentum for moving average for batchnorm, must be [0, 1]. Default:0.997

• eps (float) – Term added to the denominator to improve numerical stability for batchnorm, should be greater than 0. Default: 1e-5.

• activation (Union[str, Cell, Primitive]) – Specifies activation type. The optional values are as following: ‘softmax’, ‘logsoftmax’, ‘relu’, ‘relu6’, ‘tanh’, ‘gelu’, ‘sigmoid’, ‘prelu’, ‘leakyrelu’, ‘hswish’, ‘hsigmoid’. Default: None.

• alpha (float) – Slope of the activation function at x < 0 for LeakyReLU. Default: 0.2.

• after_fake (bool) – Determine whether there must be a fake quantization operation after Cond2dBnAct. Default: True.

Inputs:

• x (Tensor) - Tensor of shape \((N, C_{in}, H_{in}, W_{in})\). The data type is float32.

Outputs:

Tensor of shape \((N, C_{out}, H_{out}, W_{out})\). The data type is float32.

Raises

• TypeError – If in_channels, out_channels, stride, padding or dilation is not an int.

• TypeError – If has_bias is not a bool.

• ValueError – If in_channels or out_channels stride, padding or dilation is less than 1.

• ValueError – If pad_mode is not one of ‘same’, ‘valid’, ‘pad’.

Supported Platforms:

Ascend GPU CPU

Examples

>>> net = nn.Conv2dBnAct(120, 240, 4, has_bn=True, activation='relu')
>>> x = Tensor(np.ones([1, 120, 1024, 640]), mindspore.float32)
>>> result = net(x)
>>> output = result.shape
>>> print(output)
(1, 240, 1024, 640)

class tinyms.layers.DenseBnAct(in_channels, out_channels, weight_init='normal', bias_init='zeros', has_bias=True, has_bn=False, momentum=0.9, eps=1e-05, activation=None, alpha=0.2, after_fake=True)

A combination of Dense, Batchnorm, and the activation layer.

This part is a more detailed overview of Dense op.

Parameters

• in_channels (int) – The number of channels in the input space.

• out_channels (int) – The number of channels in the output space.

• weight_init (Union[Tensor, str, Initializer, numbers.Number]) – The trainable weight_init parameter. The dtype is same as x. The values of str refer to the function initializer. Default: ‘normal’.

• bias_init (Union[Tensor, str, Initializer, numbers.Number]) – The trainable bias_init parameter. The dtype is same as x. The values of str refer to the function initializer. Default: ‘zeros’.

• has_bias (bool) – Specifies whether the layer uses a bias vector. Default: True.

• has_bn (bool) – Specifies to use batchnorm or not. Default: False.

• momentum (float) – Momentum for moving average for batchnorm, must be [0, 1]. Default:0.9

• eps (float) – Term added to the denominator to improve numerical stability for batchnorm, should be greater than 0. Default: 1e-5.

• activation (Union[str, Cell, Primitive]) – Specifies activation type. The optional values are as following: ‘softmax’, ‘logsoftmax’, ‘relu’, ‘relu6’, ‘tanh’, ‘gelu’, ‘sigmoid’, ‘prelu’, ‘leakyrelu’, ‘hswish’, ‘hsigmoid’. Default: None.

• alpha (float) – Slope of the activation function at x < 0 for LeakyReLU. Default: 0.2.

• after_fake (bool) – Determine whether there must be a fake quantization operation after DenseBnAct. Default: True.

Inputs:

• x (Tensor) - Tensor of shape \((N, in\_channels)\). The data type is float32.

Outputs:

Tensor of shape \((N, out\_channels)\). The data type is float32.

Raises

• TypeError – If in_channels or out_channels is not an int.

• TypeError – If has_bias, has_bn or after_fake is not a bool.

• TypeError – If momentum or eps is not a float.

• ValueError – If momentum is not in range [0, 1.0].

Supported Platforms:

Ascend GPU CPU

Examples

>>> net = nn.DenseBnAct(3, 4)
>>> x = Tensor(np.random.randint(0, 255, [2, 3]), mindspore.float32)
>>> result = net(x)
>>> output = result.shape
>>> print(output)
(2, 4)

class tinyms.layers.TimeDistributed(layer, time_axis, reshape_with_axis=None)

The time distributed layer.

Time distributed is a wrapper which allows to apply a layer to every temporal slice of an input. And the x should be at least 3D. There are two cases in the implementation. When reshape_with_axis provided, the reshape method will be chosen, which is more efficient; otherwise, the method of dividing the inputs along time axis will be used, which is more general. For example, reshape_with_axis could not be provided when deal with Batch Normalization.

Parameters

• layer (Union[Cell, Primitive]) – The Cell or Primitive which will be wrapped.

• time_axis (int) – The axis of time_step.

• reshape_with_axis (int) – The axis which will be reshaped with time_axis. Default: None.

Inputs:

• x (Tensor) - Tensor of shape \((N, T, *)\), where \(*\) means any number of additional dimensions.

Outputs:

Tensor of shape \((N, T, *)\)

Supported Platforms:

Ascend GPU CPU

Raises

TypeError – If layer is not a Cell or Primitive.

Examples

>>> x = Tensor(np.random.random([32, 10, 3]), mindspore.float32)
>>> dense = nn.Dense(3, 6)
>>> net = nn.TimeDistributed(dense, time_axis=1, reshape_with_axis=0)
>>> output = net(x)
>>> print(output.shape)
(32, 10, 6)

class tinyms.layers.DenseThor(in_channels, out_channels, weight_init='normal', bias_init='zeros', has_bias=True, activation=None)

The dense connected layer and saving the information needed for THOR.

Applies dense connected layer for the input and saves the information A and G in the dense connected layer needed for THOR, the detail can be seen in paper: https://www.aaai.org/AAAI21Papers/AAAI-6611.ChenM.pdf This layer implements the operation as:

\[\text{outputs} = \text{activation}(\text{inputs} * \text{kernel} + \text{bias}),\]

where \(\text{activation}\) is the activation function , \(\text{kernel}\) is a weight matrix with the same data type as the inputs created by the layer, and \(\text{bias}\) is a bias vector with the same data type as the inputs created by the layer (only if has_bias is True).

Parameters

• in_channels (int) – The number of the input channels.

• out_channels (int) – The number of the output channels.

• weight_init (Union[Tensor, str, Initializer, numbers.Number]) – The trainable weight_init parameter. The dtype is same as x. The values of str refer to the function initializer. Default: ‘normal’.

• bias_init (Union[Tensor, str, Initializer, numbers.Number]) – The trainable bias_init parameter. The dtype is same as x. The values of str refer to the function initializer. Default: ‘zeros’.

• has_bias (bool) – Specifies whether the layer uses a bias vector. Default: True.

• activation (str) – activate function applied to the output of the fully connected layer, eg. ‘ReLU’. Default: None.

Inputs:

• x (Tensor) - Tensor of shape \((N, in\_channels)\).

Outputs:

Tensor of shape \((N, out\_channels)\).

Raises

ValueError – If the shape of weight_init or bias_init is incorrect.

Supported Platforms:

Ascend GPU

Examples

>>> x = Tensor(np.array([[1, 2, 3], [3, 4, 5]]), mindspore.float32)
>>> net = nn.DenseThor(3, 4, weight_init="ones")
>>> output = net(x)
>>> print(output)
[[  6.  6.  6.  6.]
 [ 12. 12. 12. 12. ]]

save_gradient(dout)

this function only for thor optimizer save_gradient

class tinyms.layers.Conv2dThor(in_channels, out_channels, kernel_size, stride=1, pad_mode='same', padding=0, dilation=1, group=1, has_bias=False, weight_init='normal', bias_init='zeros')

2D convolution layer and saving the information needed for THOR.

Applies a 2D convolution over an input tensor which is typically of shape \((N, C_{in}, H_{in}, W_{in})\), where \(N\) is batch size, \(C_{in}\) is channel number, and \(H_{in}, W_{in})\) are height and width. And saves the information A and G in the 2D convolution layer needed for THOR. The detail can be seen in paper: https://www.aaai.org/AAAI21Papers/AAAI-6611.ChenM.pdf

For each batch of shape \((C_{in}, H_{in}, W_{in})\), the formula is defined as:

\[out_j = \sum_{i=0}^{C_{in} - 1} ccor(W_{ij}, X_i) + b_j,\]

where \(ccor\) is the cross-correlation operator, \(C_{in}\) is the input channel number, \(j\) ranges from \(0\) to \(C_{out} - 1\), \(W_{ij}\) corresponds to the \(i\)-th channel of the \(j\)-th filter and \(out_{j}\) corresponds to the \(j\)-th channel of the output. \(W_{ij}\) is a slice of kernel and it has shape \((\text{ks_h}, \text{ks_w})\), where \(\text{ks_h}\) and \(\text{ks_w}\) are the height and width of the convolution kernel. The full kernel has shape \((C_{out}, C_{in} // \text{group}, \text{ks_h}, \text{ks_w})\), where group is the group number to split the input x in the channel dimension.

If the ‘pad_mode’ is set to be “valid”, the output height and width will be \(\left \lfloor{1 + \frac{H_{in} + 2 \times \text{padding} - \text{ks_h} - (\text{ks_h} - 1) \times (\text{dilation} - 1) }{\text{stride}}} \right \rfloor\) and \(\left \lfloor{1 + \frac{W_{in} + 2 \times \text{padding} - \text{ks_w} - (\text{ks_w} - 1) \times (\text{dilation} - 1) }{\text{stride}}} \right \rfloor\) respectively.

Note

For Ascend, the type of inputs should be subclass of Tensor[Float16], Tensor[Int8]. For GPU, the type of inputs should be subclass of Tensor[Float32].

Parameters

• in_channels (int) – The number of the input channel \(C_{in}\).

• out_channels (int) – The number of the output channel \(C_{out}\).

• kernel_size (Union[int, tuple[int]]) – The data type is int or a tuple of 2 integers. Specifies the height and width of the 2D convolution window. Single int means that the value is not only the height, but also the width of the kernel. A tuple of 2 integers means the height and the width of the kernel respectively.

• stride (Union[int, tuple[int]]) – The distance of kernel moving, an int number represents the height and width of movement, or a tuple of two int numbers that represent height and width of movement, respectively. Default: 1.

• pad_mode (str) –

  Specifies padding mode. The optional values are “same”, “valid”, “pad”. Default: “same”.

  • same: Adopts the way of completion. The shape of the output will be the same as the x. The total number of padding will be calculated in horizontal and vertical directions and evenly distributed to top and bottom, left and right if possible. Otherwise, the last extra padding will be done from the bottom and the right side. If this mode is set, padding must be 0.

  • valid: Adopts the way of discarding. The possible largest height and width of output will be returned without padding. Extra pixels will be discarded. If this mode is set, padding must be 0.

  • pad: Implicit paddings on both sides of the input x. The number of padding will be padded to the input Tensor borders. padding must be greater than or equal to 0.

• padding (Union[int, tuple[int]]) – Implicit paddings on both sides of the input x. If padding is an integer, the paddings of top, bottom, left and right are the same, equal to padding. If padding is a tuple with four integers, the paddings of top, bottom, left and right will be equal to padding[0], padding[1], padding[2], and padding[3] accordingly. Default: 0.

• dilation (Union[int, tuple[int]]) – The data type is int or a tuple of 2 integers. Specifies the dilation rate to use for dilated convolution. If set to be \(k > 1\), there will be \(k - 1\) pixels skipped for each sampling location. Its value must be greater or equal to 1 and bounded by the height and width of the input x. Default: 1.

• group (int) – Splits filter into groups, in_ channels and out_channels must be divisible by the number of groups. If the group is equal to in_channels and out_channels, this 2D convolution layer also can be called 2D depthwise convolution layer. Default: 1.

• has_bias (bool) – Specifies whether the layer uses a bias vector. Default: False.

• weight_init (Union[Tensor, str, Initializer, numbers.Number]) – Initializes the convolution kernel. It can be a Tensor, a string, an Initializer or a number. When a string is specified, values from ‘TruncatedNormal’, ‘Normal’, ‘Uniform’, ‘HeUniform’ and ‘XavierUniform’ distributions as well as constant ‘One’ and ‘Zero’ distributions are possible. Alias ‘xavier_uniform’, ‘he_uniform’, ‘ones’ and ‘zeros’ are acceptable. Uppercase and lowercase are both acceptable. Refer to the values of Initializer for more details. Default: ‘normal’.

• bias_init (Union[Tensor, str, Initializer, numbers.Number]) – Initializes the bias vector. Possible Initializer and string are the same as ‘weight_init’. Refer to the values of Initializer for more details. Default: ‘zeros’.

Inputs:

• x (Tensor) - Tensor of shape \((N, C_{in}, H_{in}, W_{in})\).

Outputs:

Tensor of shape \((N, C_{out}, H_{out}, W_{out})\).

Supported Platforms:

Ascend GPU

Examples

>>> net = nn.Conv2dThor(120, 240, 4, has_bias=False, weight_init='normal')
>>> # for Ascend
>>> x = Tensor(np.ones([1, 120, 1024, 640]), mindspore.float16)
>>> print(net(x).shape)
(1, 240, 1024, 640)

save_gradient(dout)

class tinyms.layers.EmbeddingThor(vocab_size, embedding_size, use_one_hot=False, embedding_table='normal', dtype=mindspore.float32, padding_idx=None)

A simple lookup table that stores embeddings of a fixed dictionary and size and saving the information needed for THOR.

This module is often used to store word embeddings and retrieve them using indices. The input to the module is a list of indices, and the output is the corresponding word embeddings. And saves the information A and G in the dense connected layer needed for THOR, the detail can be seen in paper: https://www.aaai.org/AAAI21Papers/AAAI-6611.ChenM.pdf

Note

When ‘use_one_hot’ is set to True, the type of the input x must be mindspore.int32.

Parameters

• vocab_size (int) – The size of the dictionary of embeddings.

• embedding_size (int) – The size of each embedding vector.

• use_one_hot (bool) – Specifies whether to apply one_hot encoding form. Default: False.

• embedding_table (Union[Tensor, str, Initializer, numbers.Number]) – Initializes the embedding_table. Refer to class initializer for the values of string when a string is specified. Default: ‘normal’.

• dtype (mindspore.dtype) – Data type of input x. Default: mindspore.float32.

• padding_idx (int, None) – When the padding_idx encounters index, the output embedding vector of this index will be initialized to zero. Default: None. The feature is inactivated.

Inputs:

• x (Tensor) - Tensor of input shape \((\text{batch_size}, \text{x_length})\). The elements of the Tensor must be integer and not larger than vocab_size. Otherwise the corresponding embedding vector will be zero.

Outputs:

Tensor of output shape \((\text{batch_size}, \text{x_length}, \text{embedding_size})\).

Supported Platforms:

Ascend GPU

Examples

>>> net = nn.EmbeddingThor(20000, 768,  True)
>>> x = Tensor(np.ones([8, 128]), mindspore.int32)
>>>
>>> # Maps the input word IDs to word embedding.
>>> output = net(x)
>>> output.shape
(8, 128, 768)

save_gradient(dout)

this function only for thor optimizer save_gradient

class tinyms.layers.EmbeddingLookupThor(vocab_size, embedding_size, param_init='normal', target='CPU', slice_mode='batch_slice', manual_shapes=None, max_norm=None, sparse=True, vocab_cache_size=0)

Returns a slice of the input tensor based on the specified indices and saving the information needed for THOR.

This module has the same function as EmbeddingLookup, but additionally saves the information A and G in the embeddinglookup layer needed for THOR, the detail can be seen in paper: https://www.aaai.org/AAAI21Papers/AAAI-6611.ChenM.pdf

Parameters

• vocab_size (int) – The size of the dictionary of embeddings.

• embedding_size (int) – The size of each embedding vector.

• param_init (Union[Tensor, str, Initializer, numbers.Number]) – Initializer for the embedding_table. Refer to class initializer for the values of string when a string is specified. Default: ‘normal’.

• target (str) – Specifies the target where the op is executed. The value must in [‘DEVICE’, ‘CPU’]. Default: ‘CPU’.

• slice_mode (str) – The slicing way in semi_auto_parallel/auto_parallel. The value must get through nn.EmbeddingLookup. Default: nn.EmbeddingLookup.BATCH_SLICE.

• manual_shapes (tuple) – The accompaniment array in field slice mode.

• max_norm (Union[float, None]) – A maximum clipping value. The data type must be float16, float32 or None. Default: None

• sparse (bool) – Using sparse mode. When ‘target’ is set to ‘CPU’, ‘sparse’ has to be true. Default: True.

• vocab_cache_size (int) – Cache size of the dictionary of embeddings. Default: 0. It is valid only in ‘DEVICE’ target. And the moment parameter of corresponding optimizer will also be set to the cache size. In addition, it should be noted that it will cost the ‘DEVICE’ memory, so suggests setting a reasonable value to avoid insufficient memory.

Inputs:

• input_indices (Tensor) - The shape of tensor is \((y_1, y_2, ..., y_S)\).

Outputs:

Tensor, the shape of tensor is \((z_1, z_2, ..., z_N)\).

Raises

• ValueError – If target is neither ‘CPU’ nor ‘DEVICE’.

• ValueError – If slice_mode is not one of ‘batch_slice’ or ‘field_slice’ or ‘table_row_slice’ or ‘table_column_slice’.

• ValueError – If sparse is False and target is ‘CPU’.

• ValueError – If slice_mode is ‘field_slice’ and manual_shapes is None.

• TypeError – If vocab_size or embedding_size or vocab_cache_size is not an int.

• TypeError – If sparse is not a bool or manual_shapes is not a tuple.

• ValueError – If vocab_size or embedding_size is less than 1.

• ValueError – If vocab_cache_size is less than 0.

Supported Platforms:

Ascend

Examples

>>> input_indices = Tensor(np.array([[1, 0], [3, 2]]), mindspore.int32)
>>> result = nn.EmbeddingLookup(4,2)(input_indices)
>>> print(result.shape)
(2, 2, 2)

save_gradient(dout)

this function only for thor optimizer save_gradient