moval

class moval.MOVAL(mode: str = 'classification', metric: str = 'accuracy', confidence_scores: str = 'max_class_probability-conf', estim_algorithm: str = 'ac-model', class_specific: bool = False, approximate: bool = False, approximate_boundary: int = 30)

MOVAL model defined as part of a scikit-learn-like API.

Attributes:

mode (str):

The given task to estimate model performance.

Default: classification

metric (str):

The performance metric we optimize moval model to align.

Default: accuracy Metric can be accuracy | sensitivity | precision | f1score | auc This is only effective when class_specific is True, otherwise moval will always align overall accuracy.

confidence_scores (str):

The method to calculate the confidence scores. We provide a list of confidence score calculation methods which can be displayed by running moval.models.get_conf_options().

Default: max_class_probability-conf

estim_algorithm (str):

The algorithm to estimate model performance. We also provide a list of estimation algorithm which can be displayed by running moval.models.get_estim_options().

Default: ac-model

class_specific (bool):

If True, the calculation will match class-wise confidence to class-wise accuracy.

Default: False

approximate (bool):

If True, we crop the image and label map to accelerate the optimization of segmentation.

Default: False

approximate_boundary (int):

The enlarged regions of the region to crop.

Default: 30

Example:

>>> import moval
>>> moval_model = moval.MOVAL(
                    mode = "classification",
                    metric = "accuracy",
                    confidence_scores = "max_class_probability-conf",
                    estim_algorithm = "ac-model",
                    class_specific = False
                    )

classmethod crop(logits: List[Iterable], gt: List[Iterable], approximate_boundary)

Crop the image and label map to accelrate the optimization process.

Here we do the cropping based on the label map (gt). We first find the bounding box and enlarge the region with boundary = 30 pixels.

Args:

logits: The network output (logits) of a list of n (d, H, W, (D)). gt: The cooresponding annotation of a list of n (H, W, (D)). approximate_boundary: The enlarged regions of the region to crop.

Return:

logits_post: The cropped network output (logits) of a list of n (d, H', W', (D')). gt_post: The cooresponding cropped annotation of a list of n (H', W', (D')).

estimate(logits: List[Iterable] | ndarray, gt_guide: ndarray | None = None)

The function to estimate the model performance. This will call other corresponding metric calculation functions.

Args:

logits: A numpy array of size (n, d) for classification or a list of n (d, H, W, (D)) for segmentation. gt_guide: A numpy array of size (n, d) for segmentation, indicating the existing of object d in sample n.

If False, it means that there isn’t any manuel label of class d in this sample.

Returns:

estim: estimated metric (float) for classification tasks,

or class-wise estimated metric of shape (d-1, ) for segmentation tasks.

Example:

>>> import moval
>>> import numpy as np
>>> logits = np.random.randn(1000, 10)
>>> gt = np.random.randint(0, 10, (1000))
>>> moval_model = moval.MOVAL()
>>> moval_model.fit(logits, gt)
>>> estim_acc = moval_model.estimate(logits)

estimate_accuracy(logits: List[Iterable] | ndarray, gt_guide: ndarray | None = None)

Estimate accuracy using logits.

Args:

logits: A numpy array of size (n, d) for classification or a list of n (d, H, W, (D)) for segmentation. gt_guide: A numpy array of size (n, d) for segmentation, indicating the existing of object d in sample n.

If False, it means that there isn’t any manuel label of class d in this sample.

Returns:

estim: estimated accuracy (float).

Example:

>>> import moval
>>> import numpy as np
>>> logits = np.random.randn(1000, 10)
>>> gt = np.random.randint(0, 10, (1000))
>>> moval_model = moval.MOVAL()
>>> moval_model.fit(logits, gt)
>>> estim_acc = moval_model.estimate_accuracy(logits)

estimate_auc(logits: List[Iterable] | ndarray, gt_guide: ndarray | None = None)

Estimate model’s AUC using logits.

Args:

logits: A numpy array of size (n, d) for classification or a list of n (d, H, W, (D)) for segmentation. gt_guide: A numpy array of size (n, d) for segmentation, indicating the existing of object d in sample n.

If False, it means that there isn’t any manuel label of class d in this sample.

Returns:

estim: estimated average AUC (float) for classification tasks,

or estimated AUC of shape (d-1, ) for segmentation tasks.

Example:

>>> import moval
>>> import numpy as np
>>> logits = np.random.randn(1000, 10)
>>> gt = np.random.randint(0, 10, (1000))
>>> moval_model = moval.MOVAL()
>>> moval_model.fit(logits, gt)
>>> estim_auc = moval_model.estimate_auc(logits)

estimate_f1score(logits: List[Iterable] | ndarray, gt_guide: ndarray | None = None)

Estimate model’s F1score using logits.

Args:

logits: A numpy array of size (n, d) for classification or a list of n (d, H, W, (D)) for segmentation. gt_guide: A numpy array of size (n, d) for segmentation, indicating the existing of object d in sample n.

If False, it means that there isn’t any manuel label of class d in this sample.

Returns:

estim: estimated average F1score (float) for classification tasks,

or estimated dsc of shape (d-1, ) for segmentation tasks.

Example:

>>> import moval
>>> import numpy as np
>>> logits = np.random.randn(1000, 10)
>>> gt = np.random.randint(0, 10, (1000))
>>> moval_model = moval.MOVAL()
>>> moval_model.fit(logits, gt)
>>> estim_f1score = moval_model.estimate_f1score(logits)

estimate_precision(logits: List[Iterable] | ndarray, gt_guide: ndarray | None = None)

Estimate model’s sensitivity using logits.

Args:

logits: A numpy array of size (n, d) for classification or a list of n (d, H, W, (D)) for segmentation. gt_guide: A numpy array of size (n, d) for segmentation, indicating the existing of object d in sample n.

If False, it means that there isn’t any manuel label of class d in this sample.

Returns:

estim: estimated average precision (float) for classification tasks,

or estimated precision of shape (d-1, ) for segmentation tasks.

Example:

>>> import moval
>>> import numpy as np
>>> logits = np.random.randn(1000, 10)
>>> gt = np.random.randint(0, 10, (1000))
>>> moval_model = moval.MOVAL()
>>> moval_model.fit(logits, gt)
>>> estim_precision = moval_model.estimate_precision(logits)

estimate_sensitivity(logits: List[Iterable] | ndarray, gt_guide: ndarray | None = None)

Estimate model’s sensitivity using logits.

Args:

logits: A numpy array of size (n, d) for classification or a list of n (d, H, W, (D)) for segmentation. gt_guide: A numpy array of size (n, d) for segmentation, indicating the existing of object d in sample n.

If False, it means that there isn’t any manuel label of class d in this sample.

Returns:

estim: estimated sensitivity of shape (d, ).

Example:

>>> import moval
>>> import numpy as np
>>> logits = np.random.randn(1000, 10)
>>> gt = np.random.randint(0, 10, (1000))
>>> moval_model = moval.MOVAL()
>>> moval_model.fit(logits, gt)
>>> estim_sensitivity = moval_model.estimate_sensitivity(logits)

fit(logits: List[Iterable] | ndarray, gt: List[Iterable] | ndarray, batch: int = 1) → MOVAL

Fit the estimator to the given dataset by minimzing the calibration error.

Args:

inp: The network output (logits) of shape (n, d) for classification and a list of n (d, H, W, (D)) for segmentation. gt: The cooresponding annotation of shape (n, ) for classification and a list of n (H, W, (D)) for segmentation. batch: To match the group-wise accuracy with group-wise confidence score. batch is the group size. This is useful when validation data per case is few.

Note:

It only makes sense if the class frequency is similar between class index to use batch > 1!

Return:

self

Example:

>>> import moval
>>> import numpy as np
>>> logits = np.random.randn(1000, 10)
>>> gt = np.random.randint(0, 10, (1000))
>>> moval_model = moval.MOVAL()
>>> moval_model.fit(logits, gt)

classmethod get_case_perf(model: <module 'moval.models' from '/home/docs/checkouts/readthedocs.org/user_builds/moval/envs/latest/lib/python3.10/site-packages/moval/models/__init__.py'>, metric: str, inp: ~typing.List[~typing.Iterable] | ~numpy.ndarray, probability: ~typing.List[~typing.Iterable] | ~numpy.ndarray, gt: ~typing.List[~typing.Iterable] | ~numpy.ndarray)

Store the estimated results of n fitted data.

Note:

For segmentaiton tasks, we only save the average dsc of all the foreground classes.

Args:

model: The fitted moval model. metric: The performance metric to follow. inp: The network output (logits) of shape (n, d) for classification and a list of n (d, H, W, (D)) for segmentation. probability: The calculated probability of shape (n, d) for classification and a list of n (d, H, W, (D)) for segmentation. gt: The cooresponding annotation of shape (n, ) for classification and a list of n (H, W, (D)) for segmentation.

Returns:

fitted_perf: a list contain estimated results of shape of len n. Each element is a scalar if estimated accuracy, otherwise (d, ).

get_probability(logits: List[Iterable] | ndarray)

Obtain the the probability give the logits.

Note:

It will ensemble (not exactly ensemble, but calculate the mean of probability) moval-ensemble. Here we utilize calculate_probability for classification, while calculate_probability_appr for segmentation. Because the sample number of segmentation tasks is too large.

Args:

logits: A numpy array of size (n, d) for classification or a list of n (d, H, W, (D)) for segmentation.

Examples:

>>> import moval
>>> import numpy as np
>>> logits = np.random.randn(1000, 10)
>>> moval_model = moval.MOVAL(confidence_scores = "moval-ensemble")
>>> probability = moval_model.get_probability(logits)

classmethod load(filename: str) → MOVAL

load parameters from disk.

Args:

filename: The path to the file in which to save the parameters.

Return:

The loaded model, or a list of loaded models when requiring model ensembling.

Example:

>>> import moval
>>> import numpy as np
>>> logits = np.random.randn(1000, 10)
>>> gt = np.random.randint(0, 10, (1000))
>>> moval_model = moval.MOVAL()
>>> moval_model.fit(logits, gt)
>>> moval_model.save('./foo.pkl')
>>> loaded_model = moval.MOVAL.load('./foo.pkl')
>>> estim_acc = loaded_model.estimate(logits)

classmethod probability_aggregation(probabilities)

Calcualte the mean of multiple probabilities.

Args:

probabilities: The predicted probability.

This is a list of k (n, d) for clasfication and a list of k lists of n (d, H, W, (D)) for segmentation.

Returns:

probability_agg: The average probability of shape (n, d) for classification and a list of n (d, H, W, (D)) for segmentation.

save(filename: str)

Save current parameters to disk using pickle.

Args:

filename: The path to the file in which to save the parameters.

Return:

The saved parameters, or a list of saved paramters when requiring model ensembling.

Examples:

>>> import moval
>>> import numpy as np
>>> logits = np.random.randn(1000, 10)
>>> gt = np.random.randint(0, 10, (1000))
>>> moval_model = moval.MOVAL()
>>> moval_model.fit(logits, gt)
>>> moval_model.save('./foo.pkl')