tinyms.metrics

Metrics module provides functions to measure the performance of the machine learning models on the evaluation dataset. It’s used to choose the best model.

class tinyms.metrics.AUCMetric

Calculates the auc value. Implement auc metric method.

Computes the Area Under the Curve (AUC) using the trapezoidal rule. This is a general function, given points on a curve. For computing the area under the ROC-curve.

参数:

• x (Union[np.array, list]) – From the ROC curve(fpr), np.array with false positive rates. If multiclass, this is a list of such np.array, one for each class. The shape \((N)\).

• y (Union[np.array, list]) – From the ROC curve(tpr), np.array with true positive rates. If multiclass, this is a list of such np.array, one for each class. The shape \((N)\).

• reorder (boolean) – If True, assume that the curve is ascending in the case of ties, as for an ROC curve. If the curve is non-ascending, the result will be wrong. Default: False.

返回:

Compute result.

返回类型:

area (float)

实际案例

>>> from tinyms.metrics import AUCMetric
>>>
>>> metric = AUCMetric()

clear()

Clear the internal evaluation result.

tinyms.metrics.names()

Gets all names of the metric methods.

返回:

List, the name list of metric methods.

Supported Platforms:

Ascend GPU CPU

tinyms.metrics.get_metric_fn(name, *args, **kwargs)

Gets the metric method based on the input name.

参数:

• name (str) – The name of metric method. Names can be obtained by mindspore.train.names() . object for the currently supported metrics.

• args – Arguments for the metric function.

• kwargs – Keyword arguments for the metric function.

返回:

Metric object, class instance of the metric method.

Supported Platforms:

Ascend GPU CPU

实际案例

>>> from mindspore import nn
>>> metric = nn.get_metric_fn('precision', eval_type='classification')

tinyms.metrics.get_metrics(metrics)

Get metrics used in evaluation.

参数:

metrics (Union[dict, set]) – Dict or set of metrics to be evaluated by the model during training and testing. eg: {‘accuracy’, ‘recall’}.

返回:

dict, the key is metric name, the value is class instance of metric method.

引发:

TypeError – If the type of argument ‘metrics’ is not None, dict or set.

class tinyms.metrics.Accuracy(eval_type='classification')

Calculates the accuracy for classification and multilabel data.

The accuracy class creates two local variables, the correct number and the total number that are used to compute the frequency with which y_pred matches y. This frequency is the accuracy.

\[\text{accuracy} =\frac{\text{true_positive} + \text{true_negative}} {\text{true_positive} + \text{true_negative} + \text{false_positive} + \text{false_negative}}\]

参数:

eval_type (str) – The metric to calculate the accuracy over a dataset. Supports ‘classification’ and ‘multilabel’. ‘classification’ means the dataset label is single. ‘multilabel’ means the dataset has multiple labels. Default: ‘classification’.

Supported Platforms:

Ascend GPU CPU

实际案例

>>> import numpy as np
>>> import mindspore
>>> from mindspore import Tensor
>>> from mindspore.train import Accuracy
>>>
>>> x = Tensor(np.array([[0.2, 0.5], [0.3, 0.1], [0.9, 0.6]]), mindspore.float32)
>>> y = Tensor(np.array([1, 0, 1]), mindspore.float32)
>>> metric = Accuracy('classification')
>>> metric.clear()
>>> metric.update(x, y)
>>> accuracy = metric.eval()
>>> print(accuracy)
0.6666666666666666

clear()

Clears the internal evaluation result.

eval()

Computes the accuracy.

返回:

np.float64, the computed result.

引发:

RuntimeError – If the sample size is 0.

update(*inputs)

Updates the local variables. For ‘classification’, if the index of the maximum of the predict value matches the label, the predict result is correct. For ‘multilabel’, the predict value match the label, the predict result is correct.

参数:

inputs –

Logits and labels. y_pred stands for logits, y stands for labels. y_pred and y must be a Tensor, a list or an array.

• For the ‘classification’ evaluation type, y_pred is a list of floating numbers in range \([0, 1]\) and the shape is \((N, C)\) in most cases (not strictly), where \(N\) is the number of cases and \(C\) is the number of categories. y must be in one-hot format that shape is \((N, C)\), or can be transformed to one-hot format that shape is \((N,)\).

• For ‘multilabel’ evaluation type, the value of y_pred and y can only be 0 or 1, indices with 1 indicate the positive category. The shape of y_pred and y are both \((N, C)\).

引发:

• ValueError – If the number of the inputs is not 2.

• ValueError – class numbers of last input predicted data and current predicted data not match.

class tinyms.metrics.MAE

Calculates the mean absolute error(MAE).

Creates a criterion that measures the MAE between each element in the input: \(x\) and the target: \(y\).

\[\text{MAE} = \frac{\sum_{i=1}^n \|{y\_pred}_i - y_i\|}{n}\]

where \(n\) is batch size.

Supported Platforms:

Ascend GPU CPU

实际案例

>>> import numpy as np
>>> import mindspore
>>> from mindspore import Tensor
>>> from mindspore.train import MAE
>>>
>>> x = Tensor(np.array([0.1, 0.2, 0.6, 0.9]), mindspore.float32)
>>> y = Tensor(np.array([0.1, 0.25, 0.7, 0.9]), mindspore.float32)
>>> error = MAE()
>>> error.clear()
>>> error.update(x, y)
>>> result = error.eval()
>>> print(result)
0.037499990314245224

clear()

Clears the internal evaluation result.

eval()

Computes the mean absolute error(MAE).

返回:

numpy.float64. The computed result.

引发:

RuntimeError – If the total number of samples is 0.

update(*inputs)

Updates the internal evaluation result \(y_{pred}\) and \(y\).

参数:

inputs – Input y_pred and y for calculating MAE where the shape of y_pred and y are both N-D and the shape should be the same.

引发:

ValueError – If the number of the input is not 2.

class tinyms.metrics.MSE

Measures the mean squared error(MSE).

Creates a criterion that measures the MSE (squared L2 norm) between each element in the prediction and the ground truth: \(x\) and: \(y\).

\[\text{MSE}(x,\ y) = \frac{\sum_{i=1}^n({y\_pred}_i - y_i)^2}{n}\]

where \(n\) is batch size.

Supported Platforms:

Ascend GPU CPU

实际案例

>>> import numpy as np
>>> import mindspore
>>> from mindspore import Tensor
>>> from mindspore.train import MSE
>>>
>>> x = Tensor(np.array([0.1, 0.2, 0.6, 0.9]), mindspore.float32)
>>> y = Tensor(np.array([0.1, 0.25, 0.5, 0.9]), mindspore.float32)
>>> error = MSE()
>>> error.clear()
>>> error.update(x, y)
>>> result = error.eval()
>>> print(result)
0.0031250009778887033

clear()

Clear the internal evaluation result.

eval()

Computes the mean squared error(MSE).

返回:

numpy.float64. The computed result.

引发:

RuntimeError – If the number of samples is 0.

update(*inputs)

Updates the internal evaluation result \(y_{pred}\) and \(y\).

参数:

inputs – Input y_pred and y for calculating the MSE where the shape of y_pred and y are both N-D and the shape should be the same.

引发:

ValueError – If the number of inputs is not 2.

class tinyms.metrics.Metric

Base class of metric, which is used to evaluate metrics.

The clear, update, and eval should be called when evaluating metric, and they should be overridden by subclasse. update will accumulate intermediate results in the evaluation process, eval will evaluate the final result, and clear will reinitialize the intermediate results.

Never use this class directly, but instantiate one of its subclasses instead, for examples, mindspore.train.MAE, mindspore.train.Recall etc.

Supported Platforms:

Ascend GPU CPU

abstract clear()

An interface describes the behavior of clearing the internal evaluation result.

注解

All subclasses must override this interface.

abstract eval()

An interface describes the behavior of computing the evaluation result.

注解

All subclasses must override this interface.

property indexes

Get the current indexes value. The default value is None and can be changed by set_indexes.

set_indexes(indexes)

This interface is to rearrange the inputs of update.

Given (label0, label1, logits), set the indexes to [2, 1] then the (logits, label1) will be the actually inputs of update.

注解

When customize a metric, decorate the update function with the decorator mindspore.train.rearrange_inputs() for the indexes to take effect.

参数:

indexes (List(int)) – The order of logits and labels to be rearranged.

Outputs:

Metric, its original Class instance.

引发:

ValueError – If the type of input ‘indexes’ is not a list or its elements are not all int.

实际案例

>>> import numpy as np
>>> from mindspore import Tensor
>>> from mindspore.train import Accuracy
>>>
>>> x = Tensor(np.array([[0.2, 0.5], [0.3, 0.1], [0.9, 0.6]]))
>>> y = Tensor(np.array([1, 0, 1]))
>>> y2 = Tensor(np.array([0, 0, 1]))
>>> metric = Accuracy('classification').set_indexes([0, 2])
>>> metric.clear()
>>> # indexes is [0, 2], using x as logits, y2 as label.
>>> metric.update(x, y, y2)
>>> accuracy = metric.eval()
>>> print(accuracy)
0.3333333333333333

abstract update(*inputs)

An interface describes the behavior of updating the internal evaluation result.

注解

All subclasses must override this interface.

参数:

inputs – A variable-length input argument list, usually are the logits and the corresponding labels.

tinyms.metrics.rearrange_inputs(func)

This decorator is used to rearrange the inputs according to its indexes attribute of the class.

This decorator is currently applied on the update of mindspore.train.Metric.

参数:

func (Callable) – A candidate function to be wrapped whose input will be rearranged.

返回:

Callable, used to exchange metadata between functions.

Supported Platforms:

Ascend GPU CPU

实际案例

>>> from mindspore.nn import rearrange_inputs
>>> class RearrangeInputsExample:
...     def __init__(self):
...         self._indexes = None
...
...     @property
...     def indexes(self):
...         return getattr(self, '_indexes', None)
...
...     def set_indexes(self, indexes):
...         self._indexes = indexes
...         return self
...
...     @rearrange_inputs
...     def update(self, *inputs):
...         return inputs
>>>
>>> rearrange_inputs_example = RearrangeInputsExample().set_indexes([1, 0])
>>> outs = rearrange_inputs_example.update(5, 9)
>>> print(outs)
(9, 5)

class tinyms.metrics.Precision(eval_type='classification')

Calculates precision for classification and multilabel data.

The precision function creates two local variables, \(\text{true_positive}\) and \(\text{false_positive}\), which are used to compute the precision. The calculation formula is:

\[\text{precision} = \frac{\text{true_positive}}{\text{true_positive} + \text{false_positive}}\]

注解

In the multi-label cases, the elements of \(y\) and \(y_{pred}\) must be 0 or 1.

参数:

eval_type (str) – ‘classification’ or ‘multilabel’ are supported. Default: ‘classification’.

Supported Platforms:

Ascend GPU CPU

实际案例

>>> import numpy as np
>>> from mindspore import Tensor
>>> from mindspore.train import Precision
>>>
>>> x = Tensor(np.array([[0.2, 0.5], [0.3, 0.1], [0.9, 0.6]]))
>>> y = Tensor(np.array([1, 0, 1]))
>>> metric = Precision('classification')
>>> metric.clear()
>>> metric.update(x, y)
>>> precision = metric.eval()
>>> print(precision)
[0.5 1. ]

clear()

Clears the internal evaluation result.

eval(average=False)

Computes the precision.

参数:

average (bool) – Specify whether calculate the average precision. Default: False.

返回:

numpy.float64, the computed result.

update(*inputs)

Updates the internal evaluation result with y_pred and y.

参数:

inputs – Input y_pred and y. y_pred and y are Tensor, list or numpy.ndarray. For ‘classification’ evaluation type, y_pred is in most cases (not strictly) a list of floating numbers in range \([0, 1]\) and the shape is \((N, C)\), where \(N\) is the number of cases and \(C\) is the number of categories. Shape of y can be \((N, C)\) with values 0 and 1 if one-hot encoding is used or the shape is \((N,)\) with integer values if index of category is used. For ‘multilabel’ evaluation type, y_pred and y can only be one-hot encoding with values 0 or 1. Indices with 1 indicate positive category. The shape of y_pred and y are both \((N, C)\).

引发:

ValueError – If the number of inputs is not 2.

class tinyms.metrics.HausdorffDistance(distance_metric='euclidean', percentile=None, directed=False, crop=True)

Calculates the Hausdorff distance. Hausdorff distance is the maximum and minimum distance between two point sets. Given two feature sets A and B, the Hausdorff distance between two point sets A and B is defined as follows:

\[\begin{split}\begin{array}{ll} \\ H(A, B) = \text{max}[h(A, B), h(B, A)]\\ h(A, B) = \underset{a \in A}{\text{max}}\{\underset{b \in B}{\text{min}} \rVert a - b \rVert \}\\ h(B, A) = \underset{b \in B}{\text{max}}\{\underset{a \in A}{\text{min}} \rVert b - a \rVert \} \end{array}\end{split}\]

where \(h(A,B)\) is the maximum distance of a set A to the nearest point in the set B, \(h(B, A)\) is the maximum distance of a set B to the nearest point in the set A. The distance calculation is oriented, which means that most of times \(h(A, B)\) is not equal to \(h(B, A)\). \(H(A, B)\) is the two-way Hausdorff distance.

参数:

• distance_metric (string) – Three distance measurement methods are supported: “euclidean”, “chessboard” or “taxicab”. Default: “euclidean”.

• percentile (float) – Floating point numbers between 0 and 100. Specify the percentile parameter to get the percentile of the Hausdorff distance. Default: None.

• directed (bool) – If True, it only calculates h(y_pred, y) distance, otherwise, max(h(y_pred, y), h(y, y_pred)) will be returned. Default: False.

• crop (bool) – Crop input images and only keep the foregrounds. In order to maintain two inputs’ shapes, here the bounding box is achieved by (y_pred | y) which represents the union set of two images. Default: True.

Supported Platforms:

Ascend GPU CPU

实际案例

>>> import numpy as np
>>> from mindspore import Tensor
>>> from mindspore.train import HausdorffDistance
>>>
>>> x = Tensor(np.array([[3, 0, 1], [1, 3, 0], [1, 0, 2]]))
>>> y = Tensor(np.array([[0, 2, 1], [1, 2, 1], [0, 0, 1]]))
>>> metric = HausdorffDistance()
>>> metric.clear()
>>> metric.update(x, y, 0)
>>> mean_average_distance = metric.eval()
>>> print(mean_average_distance)
1.4142135623730951

clear()

Clears the internal evaluation result.

eval()

Calculate the no-directed or directed Hausdorff distance.

返回:

numpy.float64, the hausdorff distance.

引发:

RuntimeError – If the update method is not called first, an error will be reported.

update(*inputs)

Updates the internal evaluation result with the inputs: ‘y_pred’, ‘y’ and ‘label_idx’.

参数:

inputs – Input ‘y_pred’, ‘y’ and ‘label_idx’. ‘y_pred’ and ‘y’ are a Tensor, list or numpy.ndarray. ‘y_pred’ is the predicted binary image. ‘y’ is the actual binary image. Data type of ‘label_idx’ is int or float.

引发:

• ValueError – If the number of the inputs is not 3.

• TypeError – If the data type of label_idx is not int or float.

• ValueError – If the value of label_idx is not in y_pred or y.

• ValueError – If y_pred and y have different shapes.

class tinyms.metrics.Recall(eval_type='classification')

Calculates recall for classification and multilabel data.

The recall class creates two local variables, \(\text{true_positive}\) and \(\text{false_negative}\), that are used to compute the recall. The calculation formula is:

\[\text{recall} = \frac{\text{true_positive}}{\text{true_positive} + \text{false_negative}}\]

注解

In the multi-label cases, the elements of \(y\) and \(y_{pred}\) must be 0 or 1.

参数:

eval_type (str) – ‘classification’ or ‘multilabel’ are supported. Default: ‘classification’. Default: ‘classification’.

Supported Platforms:

Ascend GPU CPU

实际案例

>>> import numpy as np
>>> from mindspore import Tensor
>>> from mindspore.train import Recall
>>>
>>> x = Tensor(np.array([[0.2, 0.5], [0.3, 0.1], [0.9, 0.6]]))
>>> y = Tensor(np.array([1, 0, 1]))
>>> metric = Recall('classification')
>>> metric.clear()
>>> metric.update(x, y)
>>> recall = metric.eval()
>>> print(recall)
[1. 0.5]

clear()

Clears the internal evaluation result.

eval(average=False)

Computes the recall.

参数:

average (bool) – Specify whether calculate the average recall. Default: False.

返回:

numpy.float64, the computed result.

update(*inputs)

Updates the internal evaluation result with y_pred and y.

参数:

inputs – Input y_pred and y. y_pred and y are a Tensor, a list or an array. For ‘classification’ evaluation type, y_pred is in most cases (not strictly) a list of floating numbers in range \([0, 1]\) and the shape is \((N, C)\), where \(N\) is the number of cases and \(C\) is the number of categories. Shape of y can be \((N, C)\) with values 0 and 1 if one-hot encoding is used or the shape is \((N,)\) with integer values if index of category is used. For ‘multilabel’ evaluation type, y_pred and y can only be one-hot encoding with values 0 or 1. Indices with 1 indicate positive category. The shape of y_pred and y are both \((N, C)\).

引发:

ValueError – If the number of inputs is not 2.

class tinyms.metrics.Fbeta(beta)

Calculates the Fbeta score.

Fbeta score is a weighted mean of precision and recall.

\[F_\beta=\frac{(1+\beta^2) \cdot true\_positive} {(1+\beta^2) \cdot true\_positive +\beta^2 \cdot false\_negative + false\_positive}\]

参数:

beta (Union[float, int]) – Beta coefficient in the F measure. beta should be greater than 0.

Supported Platforms:

Ascend GPU CPU

实际案例

>>> import numpy as np
>>> from mindspore import Tensor
>>> from mindspore.train import Fbeta
>>>
>>> x = Tensor(np.array([[0.2, 0.5], [0.3, 0.1], [0.9, 0.6]]))
>>> y = Tensor(np.array([1, 0, 1]))
>>> metric = Fbeta(1)
>>> metric.clear()
>>> metric.update(x, y)
>>> fbeta = metric.eval()
>>> print(fbeta)
[0.66666667 0.66666667]

clear()

Clears the internal evaluation result.

eval(average=False)

Computes the fbeta.

参数:

average (bool) – Whether to calculate the average fbeta. Default: False.

返回:

numpy.ndarray or numpy.float64, the computed result.

update(*inputs)

Updates the internal evaluation result y_pred and y.

参数:

inputs – Input y_pred and y. y_pred and y are Tensor, list or numpy.ndarray. y_pred is in most cases (not strictly) a list of floating numbers in range \([0, 1]\) and the shape is \((N, C)\), where \(N\) is the number of cases and \(C\) is the number of categories. y contains values of integers. The shape is \((N, C)\) if one-hot encoding is used. Shape can also be \((N,)\) if category index is used.

引发:

• ValueError – class numbers of last input predicted data and current predicted data not match.

• ValueError – If the predicted value and true value contain different classes.

class tinyms.metrics.BleuScore(n_gram=4, smooth=False)

Calculates the BLEU score. BLEU (bilingual evaluation understudy) is a metric for evaluating the quality of text translated by machine.

参数:

• n_gram (int) – The n_gram value ranges from 1 to 4. Default: 4.

• smooth (bool) – Whether or not to apply smoothing. Default: False.

引发:

ValueError – If the value range of n_gram is not from 1 to 4.

Supported Platforms:

Ascend GPU CPU

实际案例

>>> from mindspore.train import BleuScore
>>>
>>> candidate_corpus = [['i', 'have', 'a', 'pen', 'on', 'my', 'desk']]
>>> reference_corpus = [[['i', 'have', 'a', 'pen', 'in', 'my', 'desk'],
...                      ['there', 'is', 'a', 'pen', 'on', 'the', 'desk']]]
>>> metric = BleuScore()
>>> metric.clear()
>>> metric.update(candidate_corpus, reference_corpus)
>>> bleu_score = metric.eval()
>>> print(bleu_score)
0.5946035575013605

clear()

Clear the internal evaluation result.

eval()

Computes the bleu score.

返回:

numpy.float64, the bleu score.

引发:

RuntimeError – If the update method is not called first, an error will be reported.

update(*inputs)

Updates the internal evaluation result with candidate_corpus and reference_corpus.

参数:

inputs (iterator) – Input candidate_corpus and reference_corpus. candidate_corpus and reference_corpus are both a list. The candidate_corpus is an iterable of machine translated corpus. The reference_corpus is an iterable object of iterables of reference corpus.

引发:

• ValueError – If the number of inputs is not 2.

• ValueError – If the lengths of candidate_corpus and reference_corpus are not equal.

class tinyms.metrics.CosineSimilarity(similarity='cosine', reduction='none', zero_diagonal=True)

Computes representation similarity.

参数:

• similarity (str) – ‘dot’ or ‘cosine’. Default: ‘cosine’.

• reduction (str) – ‘none’, ‘sum’, ‘mean’ (all along dim -1). Default: ‘none’.

• zero_diagonal (bool) – If True, diagonals of results will be set to zero. Default: True.

Supported Platforms:

Ascend GPU CPU

实际案例

>>> import numpy as np
>>> from mindspore.train import CosineSimilarity
>>>
>>> test_data = np.array([[1, 3, 4, 7], [2, 4, 2, 5], [3, 1, 5, 8]])
>>> metric = CosineSimilarity()
>>> metric.clear()
>>> metric.update(test_data)
>>> square_matrix = metric.eval()
>>> print(square_matrix)
[[0.  0.94025615  0.95162452]
 [0.94025615  0.  0.86146098]
 [0.95162452  0.86146098  0.]]

clear()

Clears the internal evaluation result.

eval()

Computes the similarity matrix.

返回:

numpy.ndarray. The similarity matrix.

引发:

RuntimeError – If the update method is not called first, an error will be reported.

update(inputs)

Updates the internal evaluation result with ‘inputs’.

参数:

inputs (Union[Tensor, list, numpy.ndarray]) – The input matrix.

class tinyms.metrics.OcclusionSensitivity(pad_val=0.0, margin=2, n_batch=128, b_box=None)

Calculates the occlusion sensitivity of the model for a given image, which illustrates which parts of an image are most important for a network’s classification.

Occlusion sensitivity refers to how the predicted probability changes with the change of the occluded part of an image. The higher the value in the output image is, the greater the decline of certainty, indicating that the occluded area is more important in the decision-making process.

参数:

• pad_val (float) – The padding value of the occluded part in an image. Default: 0.0.

• margin (Union[int, Sequence]) – Create a cuboid / cube around the voxel you want to occlude. Default: 2.

• n_batch (int) – number of images in a batch. Default: 128.

• b_box (Sequence) – Bounding box on which to perform the analysis. The output image will also match in size. There should be a minimum and maximum for all dimensions except batch: [min1, max1, min2, max2,...]. If no bounding box is supplied, this will be the same size as the input image. If a bounding box is used, the output image will be cropped to this size. Default: None.

Supported Platforms:

Ascend GPU CPU

实际案例

>>> import numpy as np
>>> from mindspore import nn, Tensor
>>> from mindspore.train import OcclusionSensitivity
>>>
>>> class DenseNet(nn.Cell):
...     def __init__(self):
...         super(DenseNet, self).__init__()
...         w = np.array([[0.1, 0.8, 0.1, 0.1],[1, 1, 1, 1]]).astype(np.float32)
...         b = np.array([0.3, 0.6]).astype(np.float32)
...         self.dense = nn.Dense(4, 2, weight_init=Tensor(w), bias_init=Tensor(b))
...
...     def construct(self, x):
...         return self.dense(x)
>>>
>>> model = DenseNet()
>>> test_data = np.array([[0.1, 0.2, 0.3, 0.4]]).astype(np.float32)
>>> label = np.array(1).astype(np.int32)
>>> metric = OcclusionSensitivity()
>>> metric.clear()
>>> metric.update(model, test_data, label)
>>> score = metric.eval()
>>> print(score)
[0.29999995    0.6    1.    0.9]

clear()

Clears the internal evaluation result.

eval()

Computes the occlusion_sensitivity.

返回:

A numpy ndarray.

引发:

RuntimeError – If the update method is not called first, an error will be reported.

update(*inputs)

Updates input, including model, y_pred and label.

参数:

inputs – y_pred and label are a Tensor, list or numpy.ndarray. y_pred: a batch of images to test, which could be 2D or 3D. label: classification labels to check for changes. label is normally the true label, but doesn’t have to be. model is the neural network.

引发:

• ValueError – If the number of inputs is not 3.

• RuntimeError – If y_pred.shape[0] is not 1.

• RuntimeError – If the number of labels is different from the number of batches.

class tinyms.metrics.F1

Calculates the F1 score. F1 is a special case of Fbeta when beta is 1. Refer to class mindspore.train.Fbeta for more details.

\[F_1=\frac{2\cdot true\_positive}{2\cdot true\_positive + false\_negative + false\_positive}\]

Supported Platforms:

Ascend GPU CPU

实际案例

>>> import numpy as np
>>> from mindspore import Tensor
>>> from mindspore.train import F1
>>>
>>> x = Tensor(np.array([[0.2, 0.5], [0.3, 0.1], [0.9, 0.6]]))
>>> y = Tensor(np.array([1, 0, 1]))
>>> metric = F1()
>>> metric.update(x, y)
>>> result = metric.eval()
>>> print(result)
[0.66666667 0.66666667]

class tinyms.metrics.Dice(smooth=1e-05)

The Dice coefficient is a set similarity metric. It is used to calculate the similarity between two samples. The value of the Dice coefficient is 1 when the segmentation result is the best and is 0 when the segmentation result is the worst. The Dice coefficient indicates the ratio of the area between two objects to the total area. The function is shown as follows:

\[dice = \frac{2 * (pred \bigcap true)}{pred \bigcup true}\]

参数:

smooth (float) – A term added to the denominator to improve numerical stability. Should be greater than 0. Default: 1e-5.

Supported Platforms:

Ascend GPU CPU

实际案例

>>> import numpy as np
>>> from mindspore import Tensor
>>> from mindspore.train import Dice
>>>
>>> x = Tensor(np.array([[0.2, 0.5], [0.3, 0.1], [0.9, 0.6]]))
>>> y = Tensor(np.array([[0, 1], [1, 0], [0, 1]]))
>>> metric = Dice(smooth=1e-5)
>>> metric.clear()
>>> metric.update(x, y)
>>> dice = metric.eval()
>>> print(dice)
0.20467791371802546

clear()

Clears the internal evaluation result.

eval()

Computes the Dice.

返回:

Float, the computed result.

引发:

RuntimeError – If the total number of samples is 0.

update(*inputs)

Updates the internal evaluation result y_pred and y.

参数:

inputs (tuple) – Input y_pred and y. y_pred and y are Tensor, list or numpy.ndarray. y_pred is the predicted value, y is the true value. The shape of y_pred and y are both \((N, ...)\).

引发:

• ValueError – If the number of the inputs is not 2.

• ValueError – If y_pred and y do not have the same shape.

class tinyms.metrics.ROC(class_num=None, pos_label=None)

Calculates the ROC curve. It is suitable for solving binary classification and multi classification problems. In the case of multiclass, the values will be calculated based on a one-vs-the-rest approach.

参数:

• class_num (int) – The number of classes. It is not necessary to provide this argument under the binary classification scenario. Default: None.

• pos_label (int) – Determine the integer of positive class. For binary problems, it is translated to 1 by default. For multiclass problems, this argument should not be set, as it will iteratively changed in the range [0,num_classes-1]. Default: None.

Supported Platforms:

Ascend GPU CPU

实际案例

>>> import numpy as np
>>> from mindspore import Tensor
>>> from mindspore.train import ROC
>>>
>>> # 1) binary classification example
>>> x = Tensor(np.array([3, 1, 4, 2]))
>>> y = Tensor(np.array([0, 1, 2, 3]))
>>> metric = ROC(pos_label=2)
>>> metric.clear()
>>> metric.update(x, y)
>>> fpr, tpr, thresholds = metric.eval()
>>> print(fpr)
[0. 0. 0.33333333 0.6666667 1.]
>>> print(tpr)
[0. 1. 1. 1. 1.]
>>> print(thresholds)
[5 4 3 2 1]
>>>
>>> # 2) multiclass classification example
>>> x = Tensor(np.array([[0.28, 0.55, 0.15, 0.05], [0.10, 0.20, 0.05, 0.05], [0.20, 0.05, 0.15, 0.05],
...                     [0.05, 0.05, 0.05, 0.75]]))
>>> y = Tensor(np.array([0, 1, 2, 3]))
>>> metric = ROC(class_num=4)
>>> metric.clear()
>>> metric.update(x, y)
>>> fpr, tpr, thresholds = metric.eval()
>>> print(fpr)
[array([0., 0., 0.33333333, 0.66666667, 1.]), array([0., 0.33333333, 0.33333333, 1.]),
array([0., 0.33333333, 1.]), array([0., 0., 1.])]
>>> print(tpr)
[array([0., 1., 1., 1., 1.]), array([0., 0., 1., 1.]), array([0., 1., 1.]), array([0., 1., 1.])]
>>> print(thresholds)
[array([1.28, 0.28, 0.2, 0.1, 0.05]), array([1.55, 0.55, 0.2, 0.05]), array([1.15, 0.15, 0.05]),
array([1.75, 0.75, 0.05])]

clear()

Clear the internal evaluation result.

eval()

Computes the ROC curve.

返回:

A tuple, composed of fpr, tpr, and thresholds.

• fpr (np.array) - False positive rate. In binary classification case, a fpr numpy array under different thresholds will be returned, otherwise in multiclass case, a list of fpr numpy arrays will be returned and each element represents one category.

• tpr (np.array) - True positive rates. n binary classification case, a tps numpy array under different thresholds will be returned, otherwise in multiclass case, a list of tps numpy arrays will be returned and each element represents one category.

• thresholds (np.array) - Thresholds used for computing fpr and tpr.

引发:

RuntimeError – If the update method is not called first, an error will be reported.

update(*inputs)

Update state with predictions and targets.

参数:

inputs – Input y_pred and y. y_pred and y are Tensor, list or numpy.ndarray. In most cases (not strictly), y_pred is a list of floating numbers in range \([0, 1]\) and the shape is \((N, C)\), where \(N\) is the number of cases and \(C\) is the number of categories. y contains values of integers. The shape is \((N, C)\) if one-hot encoding is used. Shape can also be \((N,)\) if category index is used.

tinyms.metrics.auc(x, y, reorder=False)

Computes the AUC(Area Under the Curve) using the trapezoidal rule. This is a general function, given points on a curve, for computing the area under the ROC-curve.

参数:

• x (Union[np.array, list]) – From the ROC curve(fpr), np.array with false positive rates. If multiclass, this is a list of such np.array, one for each class. The shape \((N)\).

• y (Union[np.array, list]) – From the ROC curve(tpr), np.array with true positive rates. If multiclass, this is a list of such np.array, one for each class. The shape \((N)\).

• reorder (bool) – If False, x must rise or fall monotonously. If True, x will be sorted in ascending order. Default: False.

返回:

float, the area under the ROC-curve.

Supported Platforms:

Ascend GPU CPU

实际案例

>>> import numpy as np
>>> from mindspore.train import ROC, auc
>>>
>>> y_pred = np.array([[3, 0, 1], [1, 3, 0], [1, 0, 2]])
>>> y = np.array([[0, 2, 1], [1, 2, 1], [0, 0, 1]])
>>> metric = ROC(pos_label=2)
>>> metric.clear()
>>> metric.update(y_pred, y)
>>> fpr, tpr, thre = metric.eval()
>>> output = auc(fpr, tpr)
>>> print(output)
0.5357142857142857

class tinyms.metrics.TopKCategoricalAccuracy(k)

Calculates the top-k categorical accuracy.

参数:

k (int) – Specifies the top-k categorical accuracy to compute.

引发:

• TypeError – If k is not int.

• ValueError – If k is less than 1.

Supported Platforms:

Ascend GPU CPU

实际案例

>>> import mindspore
>>> import numpy as np
>>> from mindspore import Tensor
>>> from mindspore.train import TopKCategoricalAccuracy
>>>
>>> x = Tensor(np.array([[0.2, 0.5, 0.3, 0.6, 0.2], [0.1, 0.35, 0.5, 0.2, 0.],
...         [0.9, 0.6, 0.2, 0.01, 0.3]]), mindspore.float32)
>>> y = Tensor(np.array([2, 0, 1]), mindspore.float32)
>>> topk = TopKCategoricalAccuracy(3)
>>> topk.clear()
>>> topk.update(x, y)
>>> output = topk.eval()
>>> print(output)
0.6666666666666666

clear()

Clear the internal evaluation result.

eval()

Computes the top-k categorical accuracy.

返回:

numpy.float64, computed result.

update(*inputs)

Updates the internal evaluation result y_pred and y.

参数:

inputs – Input y_pred and y. y_pred and y are Tensor, list or numpy.ndarray. y_pred is in most cases (not strictly) a list of floating numbers in range \([0, 1]\) and the shape is \((N, C)\), where \(N\) is the number of cases and \(C\) is the number of categories. y contains values of integers. The shape is \((N, C)\) if one-hot encoding is used. Shape can also be \((N,)\) if category index is used.

注解

The method update must receive input of the form \((y_{pred}, y)\). If some samples have the same accuracy, the first sample will be chosen.

class tinyms.metrics.Top1CategoricalAccuracy

Calculates the top-1 categorical accuracy. This class is a specialized class for TopKCategoricalAccuracy. Refer to TopKCategoricalAccuracy for more details.

Supported Platforms:

Ascend GPU CPU

实际案例

>>> import numpy as np
>>> from mindspore import Tensor
>>> from mindspore.train import Top1CategoricalAccuracy
>>>
>>> x = Tensor(np.array([[0.2, 0.5, 0.3, 0.6, 0.2], [0.1, 0.35, 0.5, 0.2, 0.],
...         [0.9, 0.6, 0.2, 0.01, 0.3]]), mindspore.float32)
>>> y = Tensor(np.array([2, 0, 1]), mindspore.float32)
>>> topk = Top1CategoricalAccuracy()
>>> topk.clear()
>>> topk.update(x, y)
>>> output = topk.eval()
>>> print(output)
0.0

class tinyms.metrics.Top5CategoricalAccuracy

Calculates the top-5 categorical accuracy. This class is a specialized class for TopKCategoricalAccuracy. Refer to TopKCategoricalAccuracy for more details.

Supported Platforms:

Ascend GPU CPU

实际案例

>>> import numpy as np
>>> from mindspore import nn, Tensor
>>>
>>> x = Tensor(np.array([[0.2, 0.5, 0.3, 0.6, 0.2], [0.1, 0.35, 0.5, 0.2, 0.],
...            [0.9, 0.6, 0.2, 0.01, 0.3]]), mindspore.float32)
>>> y = Tensor(np.array([2, 0, 1]), mindspore.float32)
>>> topk = nn.Top5CategoricalAccuracy()
>>> topk.clear()
>>> topk.update(x, y)
>>> output = topk.eval()
>>> print(output)
1.0

class tinyms.metrics.Loss

Calculates the average of the loss. If method ‘update’ is called every \(n\) iterations, the result of evaluation will be:

\[loss = \frac{\sum_{k=1}^{n}loss_k}{n}\]

Supported Platforms:

Ascend GPU CPU

实际案例

>>> import numpy as np
>>> import mindspore
>>> from mindspore import Tensor
>>> from mindspore.train import Loss
>>>
>>> x = Tensor(np.array(0.2), mindspore.float32)
>>> loss = Loss()
>>> loss.clear()
>>> loss.update(x)
>>> result = loss.eval()
>>> print(result)
0.20000000298023224

clear()

Clears the internal evaluation result.

eval()

Calculates the average of the loss.

返回:

numpy.float64. The average of the loss.

引发:

RuntimeError – If the total number is 0.

update(*inputs)

Updates the internal evaluation result.

参数:

inputs – Inputs contain only one element, the element is loss. The dimension of loss must be 0 or 1.

引发:

• ValueError – If the length of inputs is not 1.

• ValueError – If the dimension of loss is not 1 or 0.

class tinyms.metrics.MeanSurfaceDistance(symmetric=False, distance_metric='euclidean')

Computes the Average Surface Distance from y_pred to y under the default setting. It measures how much, on average, the surface varies between the segmentation and the GT (ground truth).

Given two sets A and B, S(A) denotes the set of surface voxels of A, the shortest distance of an arbitrary voxel v to S(A) is defined as:

\[{\text{dis}}\left (v, S(A)\right ) = \underset{s_{A} \in S(A)}{\text{min }}\rVert v - s_{A} \rVert\]

The Average Surface Distance from set(B) to set(A) is given by:

\[AvgSurDis(B \rightarrow A) = \frac{\sum_{s_{B} \in S(B)}^{} {\text{dis} \left ( s_{B}, S(A) \right )} } {\left | S(B) \right |}\]

Where the ||*|| denotes a distance measure. |*| denotes the number of elements.

The mean of surface distance from set(B) to set(A) and from set(A) to set(B) is:

\[MeanSurDis(A \leftrightarrow B) = \frac{\sum_{s_{A} \in S(A)}^{} {\text{dis} \left ( s_{A}, S(B) \right )} + \sum_{s_{B} \in S(B)}^{} {\text{dis} \left ( s_{B}, S(A) \right )} }{\left | S(A) \right | + \left | S(B) \right |}\]

参数:

• distance_metric (string) – Three measurement methods are supported: “euclidean”, “chessboard” or “taxicab”. Default: “euclidean”.

• symmetric (bool) – Whether to calculate the Mean Surface Distance between y_pred and y. If False, it only calculates \(AvgSurDis({y\_pred} \rightarrow y)\), otherwise, the mean of distance from y_pred to y and from y to y_pred, i.e. \(MeanSurDis(y\_pred \leftrightarrow y)\), will be returned. Default: False.

Supported Platforms:

Ascend GPU CPU

实际案例

>>> import numpy as np
>>> from mindspore import Tensor
>>> from mindspore.train import MeanSurfaceDistance
>>> x = Tensor(np.array([[3, 0, 1], [1, 3, 0], [1, 0, 2]]))
>>> y = Tensor(np.array([[0, 2, 1], [1, 2, 1], [0, 0, 1]]))
>>> metric = MeanSurfaceDistance(symmetric=False, distance_metric="euclidean")
>>> metric.clear()
>>> metric.update(x, y, 0)
>>> mean_average_distance = metric.eval()
>>> print(mean_average_distance)
0.8047378541243649

clear()

Clears the internal evaluation result.

eval()

Calculate mean surface distance.

返回:

numpy.float64. The mean surface distance value.

引发:

RuntimeError – If the update method is not called first, an error will be reported.

update(*inputs)

Updates the internal evaluation result ‘y_pred’, ‘y’ and ‘label_idx’.

参数:

inputs – Input ‘y_pred’, ‘y’ and ‘label_idx’. ‘y_pred’ and ‘y’ are a Tensor, list or numpy.ndarray. ‘y_pred’ is the predicted binary image. ‘y’ is the actual binary image. ‘label_idx’, the data type of label_idx is int.

引发:

• ValueError – If the number of the inputs is not 3.

• TypeError – If the data type of label_idx is not int or float.

• ValueError – If the value of label_idx is not in y_pred or y.

• ValueError – If y_pred and y have different shapes.

class tinyms.metrics.RootMeanSquareDistance(symmetric=False, distance_metric='euclidean')

Computes the Root Mean Square Surface Distance from y_pred to y under the default setting.

Given two sets A and B, S(A) denotes the set of surface voxels of A, the shortest distance of an arbitrary voxel v to S(A) is defined as:

\[{\text{dis}}\left (v, S(A)\right ) = \underset{s_{A} \in S(A)}{\text{min }}\rVert v - s_{A} \rVert\]

The Root Mean Square Surface Distance from set(B) to set(A) is:

\[RmsSurDis(B \rightarrow A) = \sqrt{\frac{\sum_{s_{B} \in S(B)}^{} {\text{dis}^2 \left ( s_{B}, S(A) \right )} }{\left | S(B) \right |}}\]

Where the ||*|| denotes a distance measure. |*| denotes the number of elements.

The Root Mean Square Surface Distance from set(B) to set(A) and from set(A) to set(B) is:

\[RmsSurDis(A \leftrightarrow B) = \sqrt{\frac{\sum_{s_{A} \in S(A)}^{} {\text{dis} \left ( s_{A}, S(B) \right ) ^{2}} + \sum_{s_{B} \in S(B)}^{} {\text{dis} \left ( s_{B}, S(A) \right ) ^{2}}}{\left | S(A) \right | + \left | S(B) \right |}}\]

参数:

• distance_metric (string) – Three measurement methods are supported: “euclidean”, “chessboard” or “taxicab”. Default: “euclidean”.

• symmetric (bool) – Whether to calculate the symmetric average root mean square distance between y_pred and y. If False, only calculates \(RmsSurDis(y\_pred, y)\) surface distance, otherwise, the mean of distance from y_pred to y and from y to y_pred, i.e. \(RmsSurDis({y\_pred} \leftrightarrow y)\) will be returned. Default: False.

Supported Platforms:

Ascend GPU CPU

实际案例

>>> import numpy as np
>>> from mindspore import Tensor
>>> from mindspore.train import RootMeanSquareDistance
>>>
>>> x = Tensor(np.array([[3, 0, 1], [1, 3, 0], [1, 0, 2]]))
>>> y = Tensor(np.array([[0, 2, 1], [1, 2, 1], [0, 0, 1]]))
>>> metric = RootMeanSquareDistance(symmetric=False, distance_metric="euclidean")
>>> metric.clear()
>>> metric.update(x, y, 0)
>>> root_mean_square_distance = metric.eval()
>>> print(root_mean_square_distance)
1.0000000000000002

clear()

Clears the internal evaluation result.

eval()

Calculate Root Mean Square Distance.

返回:

numpy.float64, root mean square surface distance.

引发:

RuntimeError – If the update method is not called first, an error will be reported.

update(*inputs)

Updates the internal evaluation result ‘y_pred’, ‘y’ and ‘label_idx’.

参数:

inputs – Input ‘y_pred’, ‘y’ and ‘label_idx’. ‘y_pred’ and ‘y’ are Tensor, list or numpy.ndarray. ‘y_pred’ is the predicted binary image. ‘y’ is the actual binary image. ‘label_idx’, the data type of label_idx is int.

引发:

• ValueError – If the number of the inputs is not 3.

• TypeError – If the data type of label_idx is not int or float.

• ValueError – If the value of label_idx is not in y_pred or y.

• ValueError – If y_pred and y have different shapes.

class tinyms.metrics.Perplexity(ignore_label=None)

Computes perplexity. Perplexity is a measurement about how well a probability distribution or a model predicts a sample. A low perplexity indicates the model can predict the sample well. The function is shown as follows:

\[PP(W)=P(w_{1}w_{2}...w_{N})^{-\frac{1}{N}}=\sqrt[N]{\frac{1}{P(w_{1}w_{2}...w_{N})}}\]

Where \(w\) represents words in corpus.

参数:

ignore_label (Union[int, None]) – Index of an invalid label to be ignored when counting. If set to None, it will include all entries. Default: None.

Supported Platforms:

Ascend GPU CPU

实际案例

>>> import numpy as np
>>> from mindspore import Tensor
>>> from mindspore.train import Perplexity
>>> x = Tensor(np.array([[0.2, 0.5], [0.3, 0.1], [0.9, 0.6]]))
>>> y = Tensor(np.array([1, 0, 1]))
>>> metric = Perplexity(ignore_label=None)
>>> metric.clear()
>>> metric.update(x, y)
>>> perplexity = metric.eval()
>>> print(perplexity)
2.231443166940565

clear()

Clears the internal evaluation result.

eval()

Returns the current evaluation result.

返回:

numpy.float64. The computed result.

引发:

RuntimeError – If the sample size is 0.

update(*inputs)

Updates the internal evaluation result preds and labels.

参数:

inputs – Input preds and labels. preds and labels are a Tensor, list or numpy.ndarray. preds is the predicted values, labels is the labels of the data. The shape of preds and labels are both \((N, C)\).

引发:

• ValueError – If the number of the inputs is not 2.

• RuntimeError – If preds and labels have different lengths.

• RuntimeError – If label shape is not equal to pred shape.

class tinyms.metrics.ConfusionMatrix(num_classes, normalize='no_norm', threshold=0.5)

Computes the confusion matrix, which is commonly used to evaluate the performance of classification models, including binary classification and multiple classification.

If you only need confusion matrix, use this class. If you want to calculate other metrics, such as ‘PPV’, ‘TPR’, ‘TNR’, etc., use class mindspore.train.ConfusionMatrixMetric .

参数:

• num_classes (int) – Number of classes in the dataset.

• normalize (str) –

  Normalization mode for confusion matrix. Default: “no_norm”. Choose from:

  • ’no_norm’ (None) - No Normalization is used. Default: None.

  • ’target’ (str) - Normalization based on target value.

  • ’prediction’ (str) - Normalization based on predicted value.

  • ’all’ (str) - Normalization over the whole matrix.

• threshold (float) – The threshold used to compare with the input tensor. Default: 0.5.

Supported Platforms:

Ascend GPU CPU

实际案例

>>> import numpy as np
>>> from mindspore import Tensor
>>> from mindspore.train import ConfusionMatrix
>>>
>>> x = Tensor(np.array([1, 0, 1, 0]))
>>> y = Tensor(np.array([1, 0, 0, 1]))
>>> metric = ConfusionMatrix(num_classes=2, normalize='no_norm', threshold=0.5)
>>> metric.clear()
>>> metric.update(x, y)
>>> output = metric.eval()
>>> print(output)
[[1. 1.]
 [1. 1.]]

clear()

Clears the internal evaluation result.

eval()

Computes confusion matrix.

返回:

numpy.ndarray, the computed result.

update(*inputs)

Update state with y_pred and y.

参数:

inputs (tuple) – Input y_pred and y. y_pred and y are a Tensor, list or numpy.ndarray. y_pred is the predicted value, y is the true value. The shape of y_pred is \((N, C, ...)\) or \((N, ...)\). The shape of y is \((N, ...)\).

引发:

• ValueError – If the number of inputs is not 2.

• ValueError – If the dim of y_pred and y are not equal.

class tinyms.metrics.ConfusionMatrixMetric(skip_channel=True, metric_name='sensitivity', calculation_method=False, decrease='mean')

Computes metrics related to confusion matrix. The calculation based on full-scale tensor, average values of batch, class channel and iteration are collected. All metrics supported by the interface are listed in comments of metric_name.

If you want to calculate metrics related to confusion matrix, such as ‘PPV’, ‘TPR’, ‘TNR’, use this class. If you only want to calculate confusion matrix, please use mindspore.train.ConfusionMatrix .

参数:

• skip_channel (bool) – Whether to skip the measurement calculation on the first channel of the predicted output. Default: True.

• metric_name (str) – Names of supported metrics , users can also set the industry common aliases for them. Choose from: [“sensitivity”, “specificity”, “precision”, “negative predictive value”, “miss rate”, “fall out”, “false discovery rate”, “false omission rate”, “prevalence threshold”, “threat score”, “accuracy”, “balanced accuracy”, “f1 score”, “matthews correlation coefficient”, “fowlkes mallows index”, “informedness”, “markedness”]. Default: “sensitivity”.

• calculation_method (bool) – If true, the measurement for each sample will be calculated first. If not, the confusion matrix of all samples will be accumulated first. As for classification task, ‘calculation_method’ should be False. Default: False.

• decrease (str) – The reduction method on data batch. decrease takes effect only when calculation_method is True. Default: “mean”. Choose from: [“none”, “mean”, “sum”, “mean_batch”, “sum_batch”, “mean_channel”, “sum_channel”].

Supported Platforms:

Ascend GPU CPU

实际案例

>>> import numpy as np
>>> from mindspore import Tensor
>>> from mindspore.train import ConfusionMatrixMetric
>>>
>>> metric = ConfusionMatrixMetric(skip_channel=True, metric_name="tpr",
...                                   calculation_method=False, decrease="mean")
>>> metric.clear()
>>> x = Tensor(np.array([[[0], [1]], [[1], [0]]]))
>>> y = Tensor(np.array([[[0], [1]], [[0], [1]]]))
>>> metric.update(x, y)
>>> avg_output = metric.eval()
>>> print(avg_output)
[0.5]

clear()

Clears the internal evaluation result.

eval()

Computes confusion matrix metric.

返回:

ndarray, the computed result.

update(*inputs)

Update state with predictions and targets.

参数:

inputs (tuple) –

Input y_pred and y. y_pred and y are a Tensor, list or numpy.ndarray.

• y_pred (ndarray): The batch data shape is \((N, C, ...)\) or \((N, ...)\), representing onehot format or category index format respectively. As for classification tasks, y_pred should have the shape [BN] where N is larger than 1. As for segmentation tasks, the shape should be [BNHW] or [BNHWD].

• y (ndarray): It must be one-hot format. The batch data shape is \((N, C, ...)\).

引发:

ValueError – If the number of the inputs is not 2.