tinyms.metrics

Metrics module provides functions to measure the performance of the machine learning models on the evaluation dataset. It’s used to choose the best model.

tinyms.metrics.names()

Gets the names of the metric methods.

Returns

List, the name list of metric methods.

tinyms.metrics.get_metric_fn(name, *args, **kwargs)

Gets the metric method based on the input name.

Parameters

• name (str) – The name of metric method. Refer to the ‘__factory__’ object for the currently supported metrics.

• args – Arguments for the metric function.

• kwargs – Keyword arguments for the metric function.

Returns

Metric object, class instance of the metric method.

Examples

>>> metric = nn.get_metric_fn('precision', eval_type='classification')

class tinyms.metrics.Accuracy(eval_type='classification')

Calculates the accuracy for classification and multilabel data.

The accuracy class creates two local variables, the correct number and the total number that are used to compute the frequency with which predictions matches labels. This frequency is ultimately returned as the accuracy: an idempotent operation that simply divides the correct number by the total number.

\[\text{accuracy} =\frac{\text{true_positive} + \text{true_negative}} {\text{true_positive} + \text{true_negative} + \text{false_positive} + \text{false_negative}}\]

Parameters

eval_type (str) – Metric to calculate the accuracy over a dataset, for classification (single-label), and multilabel (multilabel classification). Default: ‘classification’.

Examples

>>> x = Tensor(np.array([[0.2, 0.5], [0.3, 0.1], [0.9, 0.6]]), mindspore.float32)
>>> y = Tensor(np.array([1, 0, 1]), mindspore.float32)
>>> metric = nn.Accuracy('classification')
>>> metric.clear()
>>> metric.update(x, y)
>>> accuracy = metric.eval()
>>> print(accuracy)
0.6666666666666666

clear()

Clears the internal evaluation result.

eval()

Computes the accuracy.

Returns

Float, the computed result.

Raises

RuntimeError – If the sample size is 0.

update(*inputs)

Updates the internal evaluation result \(y_{pred}\) and \(y\).

Parameters

inputs – Input y_pred and y. y_pred and y are a Tensor, a list or an array. For the ‘classification’ evaluation type, y_pred is in most cases (not strictly) a list of floating numbers in range \([0, 1]\) and the shape is \((N, C)\), where \(N\) is the number of cases and \(C\) is the number of categories. Shape of y can be \((N, C)\) with values 0 and 1 if one-hot encoding is used or the shape is \((N,)\) with integer values if index of category is used. For ‘multilabel’ evaluation type, y_pred and y can only be one-hot encoding with values 0 or 1. Indices with 1 indicate the positive category. The shape of y_pred and y are both \((N, C)\).

Raises

ValueError – If the number of the inputs is not 2.

class tinyms.metrics.MAE

Calculates the mean absolute error.

Creates a criterion that measures the mean absolute error (MAE) between each element in the input: \(x\) and the target: \(y\).

\[\text{MAE} = \frac{\sum_{i=1}^n \|y_i - x_i\|}{n}\]

Here \(y_i\) is the prediction and \(x_i\) is the true value.

Note

The method update must be called with the form update(y_pred, y).

Examples

>>> x = Tensor(np.array([0.1, 0.2, 0.6, 0.9]), mindspore.float32)
>>> y = Tensor(np.array([0.1, 0.25, 0.7, 0.9]), mindspore.float32)
>>> error = nn.MAE()
>>> error.clear()
>>> error.update(x, y)
>>> result = error.eval()
>>> print(result)
0.037499990314245224

clear()

Clears the internal evaluation result.

eval()

Computes the mean absolute error.

Returns

Float, the computed result.

Raises

RuntimeError – If the number of the total samples is 0.

update(*inputs)

Updates the internal evaluation result \(y_{pred}\) and \(y\).

Parameters

inputs – Input y_pred and y for calculating mean absolute error where the shape of y_pred and y are both N-D and the shape are the same.

Raises

ValueError – If the number of the input is not 2.

class tinyms.metrics.MSE

Measures the mean squared error.

Creates a criterion that measures the mean squared error (squared L2 norm) between each element in the input: \(x\) and the target: \(y\).

\[\text{MSE}(x,\ y) = \frac{\sum_{i=1}^n(y_i - x_i)^2}{n}\]

where \(n\) is batch size.

Examples

>>> x = Tensor(np.array([0.1, 0.2, 0.6, 0.9]), mindspore.float32)
>>> y = Tensor(np.array([0.1, 0.25, 0.5, 0.9]), mindspore.float32)
>>> error = nn.MSE()
>>> error.clear()
>>> error.update(x, y)
>>> result = error.eval()

clear()

Clear the internal evaluation result.

eval()

Compute the mean squared error.

Returns

Float, the computed result.

Raises

RuntimeError – If the number of samples is 0.

update(*inputs)

Updates the internal evaluation result \(y_{pred}\) and \(y\).

Parameters

inputs – Input y_pred and y for calculating mean square error where the shape of y_pred and y are both N-D and the shape are the same.

Raises

ValueError – If the number of input is not 2.

class tinyms.metrics.Metric

Base class of metric.

Note

For examples of subclasses, please refer to the definition of class MAE, ‘Recall’ etc.

abstract clear()

An interface describes the behavior of clearing the internal evaluation result.

Note

All subclasses must override this interface.

abstract eval()

An interface describes the behavior of computing the evaluation result.

Note

All subclasses must override this interface.

abstract update(*inputs)

An interface describes the behavior of updating the internal evaluation result.

Note

All subclasses must override this interface.

Parameters

inputs – A variable-length input argument list.

class tinyms.metrics.Precision(eval_type='classification')

Calculates precision for classification and multilabel data.

The precision function creates two local variables, \(\text{true_positive}\) and \(\text{false_positive}\), that are used to compute the precision. This value is ultimately returned as the precision, an idempotent operation that simply divides \(\text{true_positive}\) by the sum of \(\text{true_positive}\) and \(\text{false_positive}\).

\[\text{precision} = \frac{\text{true_positive}}{\text{true_positive} + \text{false_positive}}\]

Note

In the multi-label cases, the elements of \(y\) and \(y_{pred}\) must be 0 or 1.

Parameters

eval_type (str) – Metric to calculate accuracy over a dataset, for classification or multilabel. Default: ‘classification’.

Examples

>>> x = Tensor(np.array([[0.2, 0.5], [0.3, 0.1], [0.9, 0.6]]))
>>> y = Tensor(np.array([1, 0, 1]))
>>> metric = nn.Precision('classification')
>>> metric.clear()
>>> metric.update(x, y)
>>> precision = metric.eval()
>>> print(precision)
[0.5 1. ]

clear()

Clears the internal evaluation result.

eval(average=False)

Computes the precision.

Parameters

average (bool) – Specify whether calculate the average precision. Default value is False.

Returns

Float, the computed result.

update(*inputs)

Updates the internal evaluation result with y_pred and y.

Parameters

inputs – Input y_pred and y. y_pred and y are Tensor, list or numpy.ndarray. For ‘classification’ evaluation type, y_pred is in most cases (not strictly) a list of floating numbers in range \([0, 1]\) and the shape is \((N, C)\), where \(N\) is the number of cases and \(C\) is the number of categories. Shape of y can be \((N, C)\) with values 0 and 1 if one-hot encoding is used or the shape is \((N,)\) with integer values if index of category is used. For ‘multilabel’ evaluation type, y_pred and y can only be one-hot encoding with values 0 or 1. Indices with 1 indicate positive category. The shape of y_pred and y are both \((N, C)\).

Raises

ValueError – If the number of input is not 2.

class tinyms.metrics.HausdorffDistance(distance_metric='euclidean', percentile=None, directed=False, crop=True)

Calculates the Hausdorff distance. Hausdorff distance is the maximum and minimum distance between two point sets. Given two feature sets A and B, the Hausdorff distance between two point sets A and B is defined as follows:

\[H(A, B) = \text{max}[h(A, B), h(B, A)] h(A, B) = \underset{a \in A}{\text{max}}\{\underset{b \in B}{\text{min}} \rVert a - b \rVert \} h(A, B) = \underset{b \in B}{\text{max}}\{\underset{a \in A}{\text{min}} \rVert b - a \rVert \}\]

Parameters

• distance_metric (string) – The parameter of calculating Hausdorff distance supports three measurement methods, “euclidean”, “chessboard” or “taxicab”. Default: “euclidean”.

• percentile (float) – Floating point numbers between 0 and 100. Specify the percentile parameter to get the percentile of the Hausdorff distance. Defaults: None.

• directed (bool) – It can be divided into directional and non directional Hausdorff distance, and the default is non directional Hausdorff distance, specify the percentile parameter to get the percentile of the Hausdorff distance. Default: False.

• crop (bool) – Crop input images and only keep the foregrounds. In order to maintain two inputs’ shapes, here the bounding box is achieved by (y_pred | y) which represents the union set of two images. Default: True.

Supported Platforms:

Ascend GPU CPU

Examples

>>> x = Tensor(np.array([[3, 0, 1], [1, 3, 0], [1, 0, 2]]))
>>> y = Tensor(np.array([[0, 2, 1], [1, 2, 1], [0, 0, 1]]))
>>> metric = nn.HausdorffDistance()
>>> metric.clear()
>>> metric.update(x, y, 0)
>>> mean_average_distance = metric.eval()
>>> print(mean_average_distance)
1.4142135623730951

clear()

Clears the internal evaluation result.

eval()

Calculate the no-directed or directed Hausdorff distance.

update(*inputs)

Updates the internal evaluation result ‘y_pred’, ‘y’ and ‘label_idx’.

Parameters

inputs –

Input ‘y_pred’, ‘y’ and ‘label_idx’. ‘y_pred’ and ‘y’ are Tensor or numpy.ndarray. ‘y_pred’ is the

predicted binary image. ‘y’ is the actual binary image. ‘label_idx’, the data type of label_idx is int.

Raises:

ValueError: If the number of the inputs is not 3.

class tinyms.metrics.Recall(eval_type='classification')

Calculates recall for classification and multilabel data.

The recall class creates two local variables, \(\text{true_positive}\) and \(\text{false_negative}\), that are used to compute the recall. This value is ultimately returned as the recall, an idempotent operation that simply divides \(\text{true_positive}\) by the sum of \(\text{true_positive}\) and \(\text{false_negative}\).

\[\text{recall} = \frac{\text{true_positive}}{\text{true_positive} + \text{false_negative}}\]

Note

In the multi-label cases, the elements of \(y\) and \(y_{pred}\) must be 0 or 1.

Parameters

eval_type (str) – Metric to calculate the recall over a dataset, for classification or multilabel. Default: ‘classification’.

Examples

>>> x = Tensor(np.array([[0.2, 0.5], [0.3, 0.1], [0.9, 0.6]]))
>>> y = Tensor(np.array([1, 0, 1]))
>>> metric = nn.Recall('classification')
>>> metric.clear()
>>> metric.update(x, y)
>>> recall = metric.eval()
>>> print(recall)
[1. 0.5]

clear()

Clears the internal evaluation result.

eval(average=False)

Computes the recall.

Parameters

average (bool) – Specify whether calculate the average recall. Default value is False.

Returns

Float, the computed result.

update(*inputs)

Updates the internal evaluation result with y_pred and y.

Parameters

inputs – Input y_pred and y. y_pred and y are a Tensor, a list or an array. For ‘classification’ evaluation type, y_pred is in most cases (not strictly) a list of floating numbers in range \([0, 1]\) and the shape is \((N, C)\), where \(N\) is the number of cases and \(C\) is the number of categories. Shape of y can be \((N, C)\) with values 0 and 1 if one-hot encoding is used or the shape is \((N,)\) with integer values if index of category is used. For ‘multilabel’ evaluation type, y_pred and y can only be one-hot encoding with values 0 or 1. Indices with 1 indicate positive category. The shape of y_pred and y are both \((N, C)\).

Raises

ValueError – If the number of input is not 2.

class tinyms.metrics.Fbeta(beta)

Calculates the fbeta score.

Fbeta score is a weighted mean of precison and recall.

\[F_\beta=\frac{(1+\beta^2) \cdot true\_positive} {(1+\beta^2) \cdot true\_positive +\beta^2 \cdot false\_negative + false\_positive}\]

Parameters

beta (Union[float, int]) – The weight of precision.

Examples

>>> x = Tensor(np.array([[0.2, 0.5], [0.3, 0.1], [0.9, 0.6]]))
>>> y = Tensor(np.array([1, 0, 1]))
>>> metric = nn.Fbeta(1)
>>> metric.clear()
>>> metric.update(x, y)
>>> fbeta = metric.eval()
>>> print(fbeta)
[0.66666667 0.66666667]

clear()

Clears the internal evaluation result.

eval(average=False)

Computes the fbeta.

Parameters

average (bool) – Whether to calculate the average fbeta. Default value is False.

Returns

Float, computed result.

update(*inputs)

Updates the internal evaluation result y_pred and y.

Parameters

inputs – Input y_pred and y. y_pred and y are Tensor, list or numpy.ndarray. y_pred is in most cases (not strictly) a list of floating numbers in range \([0, 1]\) and the shape is \((N, C)\), where \(N\) is the number of cases and \(C\) is the number of categories. y contains values of integers. The shape is \((N, C)\) if one-hot encoding is used. Shape can also be \((N,)\) if category index is used.

class tinyms.metrics.BleuScore(n_gram=4, smooth=False)

Calculates BLEU score of machine translated text with one or more references.

Parameters

• n_gram (int) – The n_gram value ranged from 1 to 4. Default: 4.

• smooth (bool) – Whether or not to apply smoothing. Default: False.

Example

>>> candidate_corpus = [['i', 'have', 'a', 'pen', 'on', 'my', 'desk']]
>>> reference_corpus = [[['i', 'have', 'a', 'pen', 'in', 'my', 'desk'],
...                      ['there', 'is', 'a', 'pen', 'on', 'the', 'desk']]]
>>> metric = BleuScore()
>>> metric.clear()
>>> metric.update(candidate_corpus, reference_corpus)
>>> bleu_score = metric.eval()
>>> print(bleu_score)
0.5946035575013605

clear()

Clear the internal evaluation result.

eval()

Computes the bleu score.

Returns

A numpy with bleu score.

update(*inputs)

Updates the internal evaluation result with candidate_corpus and reference_corpus.

Parameters

inputs – Input candidate_corpus and reference_corpus. candidate_corpus and reference_corpus are a list. The candidate_corpus is an iterable of machine translated corpus. The reference_corpus is an iterable of iterables of reference corpus.

Raises

ValueError – If the number of input is not 2.

class tinyms.metrics.CosineSimilarity(similarity='cosine', reduction='none', zero_diagonal=True)

Computes representation similarity

Parameters

• similarity (str) – ‘dot’ or ‘cosine’. Default: ‘cosine’

• reduction (str) – ‘none’, ‘sum’, ‘mean’ (all along dim -1). Default: ‘none’

• zero_diagonal (bool) – if True, the diagonals are set to zero. Default: True

Returns

A square matrix (input1, input1) with the similarity scores between all elements. If sum or mean are used, then returns (b, 1) with the reduced value for each row

Example

>>> test_data = np.array([[1, 3, 4, 7], [2, 4, 2, 5], [3, 1, 5, 8]])
>>> metric = CosineSimilarity()
>>> metric.clear()
>>> metric.update(test_data)
>>> square_matrix = metric.eval()
>>> print(square_matrix)
[[0.  0.94025615  0.95162452]
 [0.94025615  0.  0.86146098]
 [0.95162452  0.86146098  0.]]

clear()

Clears the internal evaluation result.

eval()

Computes the Cosine_Similarity square matrix.

Returns

A square matrix.

update(inputs)

Updates the internal evaluation result with ‘input1’.

Parameters

inputs – input_data input1. The input_data is a Tensor or an array.

class tinyms.metrics.OcclusionSensitivity(pad_val=0.0, margin=2, n_batch=128, b_box=None)

This function is used to calculate the occlusion sensitivity of the model for a given image. Occlusion sensitivity refers to how the probability of a given prediction changes with the change of the occluded part of the image.

For a given result, the output probability is the probability of a region.

The higher the value in the output image, the greater the decline of certainty, indicating that the occluded area is more important in the decision-making process.

Parameters

• pad_val (float) – What values need to be entered in the image when a part of the image is occluded. Default: 0.0.

• margin (Union[int, Sequence]) – Create a cuboid / cube around the voxel you want to occlude. Default: 2.

• n_batch (int) – number of images in a batch before inference. Default: 128.

• b_box (Sequence) – Bounding box on which to perform the analysis. The output image will also match in size. There should be a minimum and maximum for all dimensions except batch: [min1, max1, min2, max2,...]. If no bounding box is supplied, this will be the same size as the input image. If a bounding box is used, the output image will be cropped to this size. Default: None.

Example

>>> class DenseNet(nn.Cell):
...     def __init__(self):
...         super(DenseNet, self).__init__()
...         w = np.array([[0.1, 0.8, 0.1, 0.1],[1, 1, 1, 1]]).astype(np.float32)
...         b = np.array([0.3, 0.6]).astype(np.float32)
...         self.dense = nn.Dense(4, 2, weight_init=Tensor(w), bias_init=Tensor(b))
...
...     def construct(self, x):
...         return self.dense(x)
>>>
>>> model = DenseNet()
>>> test_data = np.array([[0.1, 0.2, 0.3, 0.4]]).astype(np.float32)
>>> label = np.array(1).astype(np.int32)
>>> metric = OcclusionSensitivity()
>>> metric.clear()
>>> metric.update(model, test_data, label)
>>> score = metric.eval()
>>> print(score)
[0.29999995    0.6    1    0.9]

clear()

Clears the internal evaluation result.

eval()

Computes the occlusion_sensitivity.

Returns

A numpy ndarray.

update(*inputs)

Updates input, including model, y_pred and label.

Inputs:

• model (nn.Cell) - classification model to use for inference.

• y_pred (Union[Tensor, list, np.ndarray]) - image to test. Should be tensor consisting of 1 batch, can be 2- or 3D.

• label (Union[int, Tensor]) - classification label to check for changes (normally the true label, but doesn’t have to be

Raises

ValueError – If the number of input is not 3.

class tinyms.metrics.F1

Calculates the F1 score. F1 is a special case of Fbeta when beta is 1. Refer to class mindspore.nn.Fbeta for more details.

\[F_1=\frac{2\cdot true\_positive}{2\cdot true\_positive + false\_negative + false\_positive}\]

Examples

>>> x = Tensor(np.array([[0.2, 0.5], [0.3, 0.1], [0.9, 0.6]]))
>>> y = Tensor(np.array([1, 0, 1]))
>>> metric = nn.F1()
>>> metric.update(x, y)
>>> result = metric.eval()
>>> print(result)
[0.66666667 0.66666667]

class tinyms.metrics.Dice(smooth=1e-05)

The Dice coefficient is a set similarity metric. It is used to calculate the similarity between two samples. The value of the Dice coefficient is 1 when the segmentation result is the best and 0 when the segmentation result is the worst. The Dice coefficient indicates the ratio of the area between two objects to the total area. The function is shown as follows:

\[dice = \frac{2 * (pred \bigcap true)}{pred \bigcup true}\]

Parameters

smooth (float) – A term added to the denominator to improve numerical stability. Should be greater than 0. Default: 1e-5.

Examples

>>> x = Tensor(np.array([[0.2, 0.5], [0.3, 0.1], [0.9, 0.6]]))
>>> y = Tensor(np.array([[0, 1], [1, 0], [0, 1]]))
>>> metric = Dice(smooth=1e-5)
>>> metric.clear()
>>> metric.update(x, y)
>>> dice = metric.eval()
>>> print(dice)
0.20467791371802546

clear()

Clears the internal evaluation result.

eval()

Computes the Dice.

Returns

Float, the computed result.

Raises

RuntimeError – If the total samples num is 0.

update(*inputs)

Updates the internal evaluation result \(y_pred\) and \(y\).

Parameters

inputs – Input y_pred and y. y_pred and y are Tensor, list or numpy.ndarray. y_pred is the predicted value, y is the true value. The shape of y_pred and y are both \((N, ...)\).

Raises

ValueError – If the number of the inputs is not 2.

class tinyms.metrics.ROC(class_num=None, pos_label=None)

Calculates the ROC curve. It is suitable for solving binary classification and multi classification problems. In the case of multiclass, the values will be calculated based on a one-vs-the-rest approach.

Parameters

• class_num (int) – Integer with the number of classes. For the problem of binary classification, it is not necessary to provide this argument. Default: None.

• pos_label (int) – Determine the integer of positive class. Default: None. For binary problems, it is translated to 1. For multiclass problems, this argument should not be set, as it is iteratively changed in the range [0,num_classes-1]. Default: None.

Examples

>>> # 1) binary classification example
>>> x = Tensor(np.array([3, 1, 4, 2]))
>>> y = Tensor(np.array([0, 1, 2, 3]))
>>> metric = ROC(pos_label=2)
>>> metric.clear()
>>> metric.update(x, y)
>>> fpr, tpr, thresholds = metric.eval()
>>> print(fpr)
[0. 0. 0.33333333 0.6666667 1.]
>>> print(tpr)
[0. 1. 1. 1. 1.]
>>> print(thresholds)
[5 4 3 2 1]
>>>
>>> # 2) multiclass classification example
>>> x = Tensor(np.array([[0.28, 0.55, 0.15, 0.05], [0.10, 0.20, 0.05, 0.05], [0.20, 0.05, 0.15, 0.05],
...                     [0.05, 0.05, 0.05, 0.75]]))
>>> y = Tensor(np.array([0, 1, 2, 3]))
>>> metric = ROC(class_num=4)
>>> metric.clear()
>>> metric.update(x, y)
>>> fpr, tpr, thresholds = metric.eval()
>>> print(fpr)
[array([0., 0., 0.33333333, 0.66666667, 1.]), array([0., 0.33333333, 0.33333333, 1.]),
array([0., 0.33333333, 1.]), array([0., 0., 1.])]
>>> print(tpr)
[array([0., 1., 1., 1., 1.]), array([0., 0., 1., 1.]), array([0., 1., 1.]), array([0., 1., 1.])]
print(thresholds)
[array([1.28, 0.28, 0.2, 0.1, 0.05]), array([1.55, 0.55, 0.2, 0.05]), array([1.15, 0.15, 0.05]),
array([1.75, 0.75, 0.05])]

clear()

Clear the internal evaluation result.

eval()

Computes the ROC curve.

Returns

A tuple, composed of fpr, tpr, and thresholds.

• fpr (np.array) - np.array with false positive rates. If multiclass, this is a list of such np.array, one for each class.

• tps (np.array) - np.array with true positive rates. If multiclass, this is a list of such np.array, one for each class.

• thresholds (np.array) - thresholds used for computing false- and true positive rates.

roc(y_pred, y, class_num=None, pos_label=None, sample_weights=None)

update(*inputs)

Update state with predictions and targets.

Parameters

inputs – Input y_pred and y. y_pred and y are Tensor, list or numpy.ndarray. In most cases (not strictly), y_pred is a list of floating numbers in range \([0, 1]\) and the shape is \((N, C)\), where \(N\) is the number of cases and \(C\) is the number of categories. y contains values of integers.

tinyms.metrics.auc(x, y, reorder=False)

Computes the Area Under the Curve (AUC) using the trapezoidal rule. This is a general function, given points on a curve. For computing the area under the ROC-curve.

Parameters

• x (Union[np.array, list]) – From the ROC curve(fpr), np.array with false positive rates. If multiclass, this is a list of such np.array, one for each class. The shape \((N)\).

• y (Union[np.array, list]) – From the ROC curve(tpr), np.array with true positive rates. If multiclass, this is a list of such np.array, one for each class. The shape \((N)\).

• reorder (boolean) – If True, assume that the curve is ascending in the case of ties, as for an ROC curve. If the curve is non-ascending, the result will be wrong. Default: False.

Returns

Compute result.

Return type

area (float)

Examples

>>> y_pred = np.array([[3, 0, 1], [1, 3, 0], [1, 0, 2]])
>>> y = np.array([[0, 2, 1], [1, 2, 1], [0, 0, 1]])
>>> metric = nn.ROC(pos_label=2)
>>> metric.clear()
>>> metric.update(y_pred, y)
>>> fpr, tpr, thre = metric.eval()
>>> output = auc(fpr, tpr)
>>> print(output)
0.5357142857142857

class tinyms.metrics.TopKCategoricalAccuracy(k)

Calculates the top-k categorical accuracy.

Note

The method update must receive input of the form \((y_{pred}, y)\). If some samples have the same accuracy, the first sample will be chosen.

Parameters

k (int) – Specifies the top-k categorical accuracy to compute.

Raises

• TypeError – If k is not int.

• ValueError – If k is less than 1.

Examples

>>> x = Tensor(np.array([[0.2, 0.5, 0.3, 0.6, 0.2], [0.1, 0.35, 0.5, 0.2, 0.],
...         [0.9, 0.6, 0.2, 0.01, 0.3]]), mindspore.float32)
>>> y = Tensor(np.array([2, 0, 1]), mindspore.float32)
>>> topk = nn.TopKCategoricalAccuracy(3)
>>> topk.clear()
>>> topk.update(x, y)
>>> output = topk.eval()
>>> print(output)
0.6666666666666666

clear()

Clear the internal evaluation result.

eval()

Computes the top-k categorical accuracy.

Returns

Float, computed result.

update(*inputs)

Updates the internal evaluation result y_pred and y.

Parameters

inputs – Input y_pred and y. y_pred and y are Tensor, list or numpy.ndarray. y_pred is in most cases (not strictly) a list of floating numbers in range \([0, 1]\) and the shape is \((N, C)\), where \(N\) is the number of cases and \(C\) is the number of categories. y contains values of integers. The shape is \((N, C)\) if one-hot encoding is used. Shape can also be \((N,)\) if category index is used.

class tinyms.metrics.Top1CategoricalAccuracy

Calculates the top-1 categorical accuracy. This class is a specialized class for TopKCategoricalAccuracy. Refer to class ‘TopKCategoricalAccuracy’ for more details.

Examples

>>> x = Tensor(np.array([[0.2, 0.5, 0.3, 0.6, 0.2], [0.1, 0.35, 0.5, 0.2, 0.],
...         [0.9, 0.6, 0.2, 0.01, 0.3]]), mindspore.float32)
>>> y = Tensor(np.array([2, 0, 1]), mindspore.float32)
>>> topk = nn.Top1CategoricalAccuracy()
>>> topk.clear()
>>> topk.update(x, y)
>>> output = topk.eval()
>>> print(output)
0.0

class tinyms.metrics.Top5CategoricalAccuracy

Calculates the top-5 categorical accuracy. This class is a specialized class for TopKCategoricalAccuracy. Refer to class ‘TopKCategoricalAccuracy’ for more details.

Examples

>>> x = Tensor(np.array([[0.2, 0.5, 0.3, 0.6, 0.2], [0.1, 0.35, 0.5, 0.2, 0.],
...            [0.9, 0.6, 0.2, 0.01, 0.3]]), mindspore.float32)
>>> y = Tensor(np.array([2, 0, 1]), mindspore.float32)
>>> topk = nn.Top5CategoricalAccuracy()
>>> topk.clear()
>>> topk.update(x, y)
>>> output = topk.eval()
>>> print(output)
1.0

class tinyms.metrics.Loss

Calculates the average of the loss. If method ‘update’ is called every \(n\) iterations, the result of evaluation will be:

\[loss = \frac{\sum_{k=1}^{n}loss_k}{n}\]

Examples

>>> x = Tensor(np.array(0.2), mindspore.float32)
>>> loss = nn.Loss()
>>> loss.clear()
>>> loss.update(x)
>>> result = loss.eval()

clear()

Clears the internal evaluation result.

eval()

Calculates the average of the loss.

Returns

Float, the average of the loss.

Raises

RuntimeError – If the total number is 0.

update(*inputs)

Updates the internal evaluation result.

Parameters

inputs – Inputs contain only one element, the element is loss. The dimension of loss must be 0 or 1.

Raises

• ValueError – If the length of inputs is not 1.

• ValueError – If the dimensions of loss is not 1.

class tinyms.metrics.MeanSurfaceDistance(symmetric=False, distance_metric='euclidean')

This function is used to compute the Average Surface Distance from y_pred to y under the default setting. Mean Surface Distance(MSD), the mean of the vector is taken. This tell us how much, on average, the surface varies between the segmentation and the GT.

Parameters

• distance_metric (string) – The parameter of calculating Hausdorff distance supports three measurement methods, “euclidean”, “chessboard” or “taxicab”. Default: “euclidean”.

• symmetric (bool) – if calculate the symmetric average surface distance between y_pred and y. In addition, if sets symmetric = True, the average symmetric surface distance between these two inputs will be returned. Defaults: False.

Examples

>>> x = Tensor(np.array([[3, 0, 1], [1, 3, 0], [1, 0, 2]]))
>>> y = Tensor(np.array([[0, 2, 1], [1, 2, 1], [0, 0, 1]]))
>>> metric = nn.MeanSurfaceDistance(symmetric=False, distance_metric="euclidean")
>>> metric.clear()
>>> metric.update(x, y, 0)
>>> mean_average_distance = metric.eval()
>>> print(mean_average_distance)
0.8047378541243649

clear()

Clears the internal evaluation result.

eval()

Calculate mean surface distance.

update(*inputs)

Updates the internal evaluation result ‘y_pred’, ‘y’ and ‘label_idx’.

Parameters

inputs –

Input ‘y_pred’, ‘y’ and ‘label_idx’. ‘y_pred’ and ‘y’ are Tensor or numpy.ndarray. ‘y_pred’ is the

predicted binary image. ‘y’ is the actual binary image. ‘label_idx’, the data type of label_idx is int.

Raises:

ValueError: If the number of the inputs is not 3.

class tinyms.metrics.RootMeanSquareDistance(symmetric=False, distance_metric='euclidean')

This function is used to compute the Residual Mean Square Distance from y_pred to y under the default setting. Residual Mean Square Distance(RMS), the mean is taken from each of the points in the vector, these residuals are squared (to remove negative signs), summed, weighted by the mean and then the square-root is taken. Measured in mm.

Parameters

• distance_metric (string) – The parameter of calculating Hausdorff distance supports three measurement methods, “euclidean”, “chessboard” or “taxicab”. Default: “euclidean”.

• symmetric (bool) – if calculate the symmetric average surface distance between y_pred and y. In addition, if sets symmetric = True, the average symmetric surface distance between these two inputs will be returned. Defaults: False.

Examples

>>> x = Tensor(np.array([[3, 0, 1], [1, 3, 0], [1, 0, 2]]))
>>> y = Tensor(np.array([[0, 2, 1], [1, 2, 1], [0, 0, 1]]))
>>> metric = nn.RootMeanSquareDistance(symmetric=False, distance_metric="euclidean")
>>> metric.clear()
>>> metric.update(x, y, 0)
>>> root_mean_square_distance = metric.eval()
>>> print(root_mean_square_distance)
1.0000000000000002

clear()

Clears the internal evaluation result.

eval()

Calculate residual mean square surface distance.

update(*inputs)

Updates the internal evaluation result ‘y_pred’, ‘y’ and ‘label_idx’.

Parameters

inputs –

Input ‘y_pred’, ‘y’ and ‘label_idx’. ‘y_pred’ and ‘y’ are Tensor or numpy.ndarray. ‘y_pred’ is the

predicted binary image. ‘y’ is the actual binary image. ‘label_idx’, the data type of label_idx is int.

Raises:

ValueError: If the number of the inputs is not 3.

class tinyms.metrics.Perplexity(ignore_label=None)

Computes perplexity. Perplexity is a measurement about how well a probability distribution or a model predicts a sample. A low perplexity indicates the model can predict the sample well. The function is shown as follows:

\[\begin{split}b^{\\big(-\\frac{1}{N} \\sum_{i=1}^N \\log_b q(x_i) \\big)} = \\exp \\big(-\\frac{1}{N} \\sum_{i=1}^N \\log q(x_i)\\big)\end{split}\]

Parameters

ignore_label (int) – Index of an invalid label to be ignored when counting. If set to None, it will include all entries. Default: -1.

Examples

>>> x = Tensor(np.array([[0.2, 0.5], [0.3, 0.1], [0.9, 0.6]]))
>>> y = Tensor(np.array([1, 0, 1]))
>>> metric = Perplexity(ignore_label=None)
>>> metric.clear()
>>> metric.update(x, y)
>>> perplexity = metric.eval()
>>> print(perplexity)
2.231443166940565

clear()

Clears the internal evaluation result.

eval()

Returns the current evaluation result.

Returns

float, the computed result.

Raises

RuntimeError – If the sample size is 0.

update(*inputs)

Updates the internal evaluation result: math:preds and :math:labels.

Parameters

inputs – Input preds and labels. preds and labels are Tensor, list or numpy.ndarray. preds is the predicted values, labels is the label of the data. The shape of preds and labels are both \((N, C)\).

Raises

ValueError – If the number of the inputs is not 2.

class tinyms.metrics.ConfusionMatrix(num_classes, normalize='no_norm', threshold=0.5)

Computes the confusion matrix. The performance matrix of measurement classification model is the model whose output is binary or multi class. The confusion matrix is calculated. An array of shape [BC4] is returned. The third dimension represents each channel of each sample in the input batch.Where B is the batch size and C is the number of classes to be calculated.

If you only want to find confusion matrix, use this class. If you want to find ‘PPV’, ‘TPR’, ‘TNR’, etc., use class ‘mindspore.metrics.ConfusionMatrixMetric’.

Parameters

• num_classes (int) – Number of classes in the dataset.

• normalize (str) –

  The parameter of calculating ConfusionMatrix supports four Normalization modes, Choose from:

  • ’no_norm’ (None) - No normalization is used. Default: None.

  • ’target’ (str) - Normalization based on target value.

  • ’prediction’ (str) - Normalization based on predicted value.

  • ’all’ (str) - Normalization over the whole matrix.

• threshold (float) – A threshold, which is used to compare with the input tensor. Default: 0.5.

Examples

>>> x = Tensor(np.array([1, 0, 1, 0]))
>>> y = Tensor(np.array([1, 0, 0, 1]))
>>> metric = nn.ConfusionMatrix(num_classes=2, normalize="no_norm", threshold=0.5)
>>> metric.clear()
>>> metric.update(x, y)
>>> output = metric.eval()
>>> print(output)
[[1. 1.]
 [1. 1.]]

clear()

Clears the internal evaluation result.

eval()

Computes confusion matrix.

Returns

numpy.ndarray, the computed result.

update(*inputs)

Update state with y_pred and y.

Parameters

inputs – Input y_pred and y. y_pred and y are a Tensor, a list or an array. y_pred is the predicted value, y is the true value. The shape of y_pred is \((N, C, ...)\) or \((N, ...)\). The shape of y is \((N, ...)\).

Raises

ValueError – If the number of the inputs is not 2.

class tinyms.metrics.ConfusionMatrixMetric(skip_channel=True, metric_name='sensitivity', calculation_method=False, decrease='mean')

The performance matrix of measurement classification model is the model whose output is binary or multi class. The correlation measure of confusion matrix was calculated from the full-scale tensor, and the average values of batch, class channel and iteration were collected. This function supports the calculation of all measures described below: the metric name in parameter metric_name.

If you want to use confusion matrix to calculate, such as ‘PPV’, ‘TPR’, ‘TNR’, use this class. If you only want to calculate confusion matrix, please use ‘mindspore.metrics.ConfusionMatrix’.

Parameters

• skip_channel (bool) – Whether to skip the measurement calculation on the first channel of the predicted output. Default: True.

• metric_name (str) – The names of indicators are in the following range. Of course, you can also set the industry common aliases for these indicators. Choose from: [“sensitivity”, “specificity”, “precision”, “negative predictive value”, “miss rate”, “fall out”, “false discovery rate”, “false omission rate”, “prevalence threshold”, “threat score”, “accuracy”, “balanced accuracy”, “f1 score”, “matthews correlation coefficient”, “fowlkes mallows index”, “informedness”, “markedness”].

• calculation_method (bool) – If true, the measurement for each sample is calculated first. If it is false, the confusion matrix of all samples is accumulated first. As for classification task, ‘calculation_method’ should be False. Default: False.

• decrease (str) – Define the mode to reduce the calculation result of one batch of data. Decrease is used only if calculation_method is True. Default: “mean”. Choose from: [“none”, “mean”, “sum”, “mean_batch”, “sum_batch”, “mean_channel”, “sum_channel”].

Examples

>>> metric = ConfusionMatrixMetric(skip_channel=True, metric_name="tpr",
...                                calculation_method=False, decrease="mean")
>>> metric.clear()
>>> x = Tensor(np.array([[[0], [1]], [[1], [0]]]))
>>> y = Tensor(np.array([[[0], [1]], [[0], [1]]]))
>>> metric.update(x, y)
>>> x = Tensor(np.array([[[0], [1]], [[1], [0]]]))
>>> y = Tensor(np.array([[[0], [1]], [[1], [0]]]))
>>> avg_output = metric.eval()
>>> print(avg_output)
[0.5]

clear()

Clears the internal evaluation result.

eval()

Computes confusion matrix metric.

Returns

ndarray, the computed result.

update(*inputs)

Update state with predictions and targets.

inputs:

Input y_pred and y. y_pred and y are a Tensor, a list or an array.

• y_pred (ndarray) - Input data to compute. It must be one-hot format and first dim is batch. The shape of y_pred is \((N, C, ...)\) or \((N, ...)\). As for classification tasks, y_pred should has the shape [BN] where N is larger than 1. As for segmentation tasks, the shape should be [BNHW] or [BNHWD].

• y (ndarray) - Compute the true value of the measure. It must be one-hot format and first dim is batch. The shape of y is \((N, C, ...)\).

Raises

ValueError – If the number of the inputs is not 2.