sklearn_evaluation.plot#

The Plot API supports both functional and object-oriented (OOP) interfaces. While the functional API allows you to quickly generate out-of-the-box plots and is the easiest to get started with, the OOP API offers more flexibility to compare models using a simple synatx, i.e, plot1 + plot2; or to customize the style and elements in the plot.

Object Oriented API#

ConfusionMatrix#

class sklearn_evaluation.plot.ConfusionMatrix(cm, *, target_names=None, normalize=False, cmap=None)#

Plot confusion matrix.

Examples

Plot and Compare Confusion Matrix for multiple classifiers:

from sklearn import datasets
from sklearn.ensemble import RandomForestClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split

from sklearn_evaluation import plot

# generate data
X, y = datasets.make_classification(
    1000, 20, n_informative=10, class_sep=0.80, n_classes=3, random_state=0
)

# split data into train and test
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)

est = RandomForestClassifier()
est.fit(X_train, y_train)
y_pred = est.predict(X_test)

# plot for classifier 1
tree_cm = plot.ConfusionMatrix.from_raw_data(y_test, y_pred)

est = DecisionTreeClassifier()
est.fit(X_train, y_train)
y_pred = est.predict(X_test)

# plot for classifier 2
forest_cm = plot.ConfusionMatrix.from_raw_data(y_test, y_pred)

# Compare
tree_cm + forest_cm

# Diff
forest_cm - tree_cm
../_images/confusion_matrix_oop_00.png
../_images/confusion_matrix_oop_01.png
../_images/confusion_matrix_oop_02.png
../_images/confusion_matrix_oop_03.png

Notes

Changed in version 0.9: Added cmap argument

classmethod from_dump(path)#

Instantiates a plot object from a path to a JSON file. A default implementation is provided, but you might override it.

classmethod from_raw_data(y_true, y_pred, target_names=None, normalize=False, cmap=None)#

Notes

Changed in version 0.9: Added cmap argument.

plot(ax=None)#

All plotting related code must be here with one optional argument ax=None. Must assign, self.ax_, and self.figure_ attributes and return self.

InteractiveConfusionMatrix#

class sklearn_evaluation.plot.InteractiveConfusionMatrix(cm, *, target_names=None, interactive_data=None)#

Plot interactive confusion matrix.

Notes

New in version 0.11.3.

classmethod from_dump(path)#

Instantiates a plot object from a path to a JSON file. A default implementation is provided, but you might override it.

classmethod from_raw_data(y_true, y_pred, X_test=None, feature_names=None, feature_subset=None, nsample=5, target_names=None, normalize=False)#

Plot confusion matrix.

See also

ConfusionMatrix

Parameters
  • y_true (array-like, shape = [n_samples]) – Correct target values (ground truth).

  • y_pred (array-like, shape = [n_samples]) – Target predicted classes (estimator predictions).

  • X_test (array-like, shape = [n_samples, n_features], optional) – Defaults to None. If X_test is passed interactive data is displayed upon clicking on each quadrant of the confusion matrix.

  • feature_names (list of feature names, optional) – feature_names can be passed if X_test passed is a numpy array. If not passed, feature names are generated like [Feature 0, Feature 1, .. , Feature N]

  • feature_subset (list of features, optional) – subset of features to display in the tables. If not passed first 5 columns are selected.

  • nsample (int, optional) – Defaults to 5. Number of sample observations to display in the interactive table if X_test is passed.

  • target_names (list) – List containing the names of the target classes. List must be in order e.g. ['Label for class 0', 'Label for class 1']. If None, generic labels will be generated e.g. ['Class 0', 'Class 1']

  • normalize (bool) – Normalize the confusion matrix

Examples

Click here to see the user guide.

plot()#

All plotting related code must be here with one optional argument ax=None. Must assign, self.ax_, and self.figure_ attributes and return self.

PrecisionRecall#

class sklearn_evaluation.plot.PrecisionRecall(precision, recall, label=None)#

Plot precision recall curve.

Parameters
  • precision (array-like, shape = [n_samples], when task is binary classification,) – or shape = [n_classes, n_samples], when task is multiclass classification.

  • recall (array-like, shape = [n_samples], when task is binary classification.) – or shape = [n_classes, n_samples], when task is multiclass classification.

  • label (string when task is binary classification, optional) – list of strings when task is multiclass classification this is used for labelling the curves. Defaults to precision recall. Make sure that the order of the labels corresponds to the order in which recall/precision arrays are passed to the constructor.

  • ax (matplotlib Axes) – Axes object to draw the plot onto, otherwise uses current Axes

Examples

Plot a Precision-Recall Curve:

from sklearn import datasets
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

from sklearn_evaluation import plot

# generate data
X, y = datasets.make_classification(
    n_samples=2000, n_features=6, n_informative=4, class_sep=0.1
)

# split data into train and test
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

est = RandomForestClassifier()
est.fit(X_train, y_train)

# y_pred = est.predict(X_test)
y_score = est.predict_proba(X_test)
y_true = y_test

# plot precision recall curve
pr = plot.PrecisionRecall.from_raw_data(y_true, y_score)
../_images/precision_recall_oop.png

Compare Precision-Recall Curves of two classifiers:

from sklearn import datasets
from sklearn.ensemble import RandomForestClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split

from sklearn_evaluation import plot

# generate data
X, y = datasets.make_classification(
    n_samples=200, n_features=10, n_informative=5, class_sep=0.65
)

# split data into train and test
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

est = RandomForestClassifier()
est.fit(X_train, y_train)

y_pred = est.predict(X_test)
y_score = est.predict_proba(X_test)
y_true = y_test

# precision recall plot for random forest
forest_pr = plot.PrecisionRecall.from_raw_data(
    y_true, y_score, label="Random forest classifier"
)

est = DecisionTreeClassifier()
est.fit(X_train, y_train)
y_pred = est.predict(X_test)
y_score = est.predict_proba(X_test)
y_true = y_test

# precision recall plot for decision tree
tree_pr = plot.PrecisionRecall.from_raw_data(
    y_true, y_score, label="Decision Tree classifier"
)

# compare two precision recall curves
forest_pr + tree_pr
../_images/precision_recall_add_00.png
../_images/precision_recall_add_01.png
../_images/precision_recall_add_02.png

Notes

New in version 0.10.1.

classmethod from_raw_data(y_true, y_score, *, label=None)#

Plot precision-recall curve from raw data.

Parameters
  • y_true (array-like, shape = [n_samples]) – Correct target values (ground truth).

  • y_score (array-like, shape = [n_samples] or [n_samples, 2] for binary) – classification or [n_samples, n_classes] for multiclass Target scores (estimator predictions).

  • label (string or list, optional) – labels for the curves

Notes

It is assumed that the y_score parameter columns are in order. For example, if y_true = [2, 2, 1, 0, 0, 1, 2], then the first column in y_score must contain the scores for class 0, second column for class 1 and so on.

plot(ax=None)#

Create the plot :param ax: An Axes object to add the plot to :type ax: matplotlib.Axes

ROC#

class sklearn_evaluation.plot.ROC(fpr, tpr, label=None)#

Plot ROC curve

Parameters
  • fpr (ndarray of shape (>2,), list of lists or list of numbers) – Increasing false positive rates such that element i is the false positive rate of predictions with score >= thresholds[i].

  • tpr (ndarray of shape (>2,), list of lists or list of numbers) – Increasing true positive rates such that element i is the true positive rate of predictions with score >= thresholds[i].

  • label (list of str, default: None) – Set curve labels

  • ax (matplotlib Axes, default: None) – Axes object to draw the plot onto, otherwise uses current Axes

  • seealso: (..) – roc():

Examples

Plot a ROC Curve for binary classification:

from sklearn import datasets
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

from sklearn_evaluation import plot

# generate data
data = datasets.make_classification(200, 10, n_informative=5, class_sep=0.65)
X = data[0]
y = data[1]

# split data into train and test
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)

est = RandomForestClassifier()
est.fit(X_train, y_train)

y_score = est.predict_proba(X_test)
y_true = y_test

# plot the roc curve
roc = plot.ROC.from_raw_data(y_true, y_score)
roc
../_images/roc_binary_classification.png

Compare ROC Curves of two binary classifiers:

from sklearn import datasets
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

from sklearn_evaluation import plot

# generate data
X, y = datasets.make_classification(200, 10, n_informative=5, class_sep=0.65)

# split data into train and test
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)

est = RandomForestClassifier()
est.fit(X_train, y_train)

y_score = est.predict_proba(X_test)
y_true = y_test

roc = plot.ROC.from_raw_data(y_true, y_score)

# create another dataset
X_, y_ = datasets.make_classification(200, 10, n_informative=5, class_sep=0.15)

# split data into train and test
X_train_, X_test_, y_train_, y_test_ = train_test_split(X_, y_, test_size=0.3)

est_ = RandomForestClassifier()
est_.fit(X_train_, y_train_)

y_score_ = est.predict_proba(X_test_)
y_true_ = y_test_

roc2 = plot.ROC.from_raw_data(y_true_, y_score_)

# Compare both classifiers
roc + roc2
../_images/roc_comparison_00.png
../_images/roc_comparison_01.png
../_images/roc_comparison_02.png

Plot a ROC Curve for multi-class classification:

import numpy as np
from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

from sklearn_evaluation import plot

# load data
iris = load_iris()
X, y = iris.data, iris.target
y = iris.target_names[y]

random_state = np.random.RandomState(0)
n_samples, n_features = X.shape

X = np.concatenate([X, random_state.randn(n_samples, 200 * n_features)], axis=1)
(
    X_train,
    X_test,
    y_train,
    y_test,
) = train_test_split(X, y, test_size=0.5, stratify=y, random_state=0)

classifier = LogisticRegression()
y_score = classifier.fit(X_train, y_train).predict_proba(X_test)

# plot roc curve
plot.ROC.from_raw_data(y_test, y_score)
../_images/roc_multi_classification.png

Notes

New in version 0.8.4.

classmethod from_dump(path)#

Instantiates a plot object from a path to a JSON file. A default implementation is provided, but you might override it.

classmethod from_raw_data(y_true, y_score, ax=None)#

Takes raw unaggregated (for an example of aggregated vs unaggregated data see the constructor docstring) data, compute statistics and initializes the object. This is the method that users typically use. (e.g., they pass y_true, and y_pred here, we aggregate and call the constructor).

Apart from input data, this method must have the same argument as the constructor.

All arguments beyond the input data must be keyword-only (add a * argument between the input and the rest of the arguments).

plot(ax=None)#

All plotting related code must be here with one optional argument ax=None. Must assign, self.ax_, and self.figure_ attributes and return self.

ClassificationReport#

class sklearn_evaluation.plot.ClassificationReport(matrix, keys, *, target_names=None)#

Examples

Plot a Classification Report:

from sklearn import datasets
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

from sklearn_evaluation import plot

# generate data
X, y = datasets.make_classification(200, 10, n_informative=5, class_sep=0.65)

# split data into train and test
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)

y_pred_rf = RandomForestClassifier().fit(X_train, y_train).predict(X_test)
y_pred_lr = LogisticRegression().fit(X_train, y_train).predict(X_test)

target_names = ["Not spam", "Spam"]

# report for random forest
cr_rf = plot.ClassificationReport.from_raw_data(
    y_test, y_pred_rf, target_names=target_names
)

# report for logistic regression
cr_lr = plot.ClassificationReport.from_raw_data(
    y_test, y_pred_lr, target_names=target_names
)

# how better it is the random forest?
cr_rf - cr_lr

# compare both reports
cr_rf + cr_lr
../_images/ClassificationReport_00.png
../_images/ClassificationReport_01.png
../_images/ClassificationReport_02.png
../_images/ClassificationReport_03.png
classmethod from_dump(path)#

Instantiates a plot object from a path to a JSON file. A default implementation is provided, but you might override it.

classmethod from_raw_data(y_true, y_pred, *, target_names=None, sample_weight=None, zero_division=0)#

Takes raw unaggregated (for an example of aggregated vs unaggregated data see the constructor docstring) data, compute statistics and initializes the object. This is the method that users typically use. (e.g., they pass y_true, and y_pred here, we aggregate and call the constructor).

Apart from input data, this method must have the same argument as the constructor.

All arguments beyond the input data must be keyword-only (add a * argument between the input and the rest of the arguments).

plot(ax=None)#

All plotting related code must be here with one optional argument ax=None. Must assign, self.ax_, and self.figure_ attributes and return self.

CalibrationCurve#

class sklearn_evaluation.plot.CalibrationCurve(mean_predicted_value, fraction_of_positives, label=None, cmap='nipy_spectral')#
Parameters
  • mean_predicted_value (ndarray of shape (n_bins,) or smaller) – The mean predicted probability in each bin.

  • fraction_of_positives (ndarray of shape (n_bins,) or smaller) – The proportion of samples whose class is the positive class, in each bin.

  • label (list of str, optional) – A list of strings, where each string refers to the name of the classifier that produced the corresponding probability estimates in probabilities. If None, the names β€œClassifier 1”, β€œClassifier 2”, etc. will be used.

  • cmap (string or matplotlib.colors.Colormap instance, optional) – Colormap used for plotting the projection. View Matplotlib Colormap documentation for available options. https://matplotlib.org/users/colormaps.html

Examples

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.naive_bayes import GaussianNB

from sklearn_evaluation import plot

X, y = make_classification(
    n_samples=20000, n_features=2, n_informative=2, n_redundant=0, random_state=0
)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.33, random_state=0
)

rf = RandomForestClassifier()
lr = LogisticRegression()
nb = GaussianNB()

rf_probas = rf.fit(X_train, y_train).predict_proba(X_test)
lr_probas = lr.fit(X_train, y_train).predict_proba(X_test)
nb_probas = nb.fit(X_train, y_train).predict_proba(X_test)

probabilities = [rf_probas, lr_probas, nb_probas]

clf_names = [
    "Random Forest",
    "Logistic Regression",
    "Gaussian Naive Bayes",
]

plot.CalibrationCurve.from_raw_data(y_test, probabilities, label=clf_names)
../_images/calibration_curve_oop.png
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression

from sklearn_evaluation import plot


def make_dataset(n_samples):
    X, y = make_classification(
        n_samples=n_samples,
        n_features=2,
        n_informative=2,
        n_redundant=0,
        random_state=0,
    )
    return train_test_split(X, y, test_size=0.33, random_state=0)


# sample size 1k
X_train, X_test, y_train, y_test1 = make_dataset(n_samples=1000)
probs1 = LogisticRegression().fit(X_train, y_train).predict_proba(X_test)

# sample size 10k
X_train, X_test, y_train, y_test2 = make_dataset(n_samples=10000)
probs2 = LogisticRegression().fit(X_train, y_train).predict_proba(X_test)

# if you want plot probability curves for different sample sizes, pass
# a list with the true labels per each element in the probabilities
# argument
plot.CalibrationCurve.from_raw_data(
    [y_test1, y_test2], [probs1, probs2], label=["1k samples", "10k samples"]
)
../_images/calibration_curve_diff_sample_size.png
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.naive_bayes import GaussianNB

from sklearn_evaluation import plot

X, y = make_classification(
    n_samples=20000, n_features=2, n_informative=2, n_redundant=0, random_state=0
)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.33, random_state=0
)

rf = RandomForestClassifier()
lr = LogisticRegression()
nb = GaussianNB()

rf_probas = rf.fit(X_train, y_train).predict_proba(X_test)
lr_probas = lr.fit(X_train, y_train).predict_proba(X_test)
nb_probas = nb.fit(X_train, y_train).predict_proba(X_test)

probabilities = [rf_probas, lr_probas, nb_probas]

clf_names = [
    "Random Forest",
    "Logistic Regression",
    "Gaussian Naive Bayes",
]

cc1 = plot.CalibrationCurve.from_raw_data(y_test, [rf_probas], label=["Random Forest"])
cc2 = plot.CalibrationCurve.from_raw_data(
    y_test,
    [lr_probas, nb_probas],
    label=["Logistic Regression", "Gaussian Naive Bayes"],
)
cc1 + cc2
../_images/calibration_curve_add_00.png
../_images/calibration_curve_add_01.png
../_images/calibration_curve_add_02.png

Notes

New in version 0.11.1.

classmethod from_raw_data(y_true, probabilities, *, label=None, n_bins=10, cmap='nipy_spectral')#

Plots calibration curves for a set of classifier probability estimates. Calibration curves help determining whether you can interpret predicted probabilities as confidence level. For example, if we take a well-calibrated and take the instances where the score is 0.8, 80% of those instanes should be from the positive class. This function only works for binary classifiers.

Parameters
  • y_true (array-like, shape = [n_samples] or list with array-like:) – Ground truth (correct) target values. If passed a single array- object, it assumes all the probabilities have the same shape as y_true. If passed a list, it expects y_true[i] to have the same size as probabilities[i]

  • probabilities (list of array-like, shape (n_samples, 2) or (n_samples,)) – A list containing the outputs of binary classifiers’ predict_proba() method or decision_function() method.

  • label (list of str, optional)) – A list of strings, where each string refers to the name of the classifier that produced the corresponding probability estimates in probabilities. If None, the names β€œClassifier 1”, β€œClassifier 2”, etc. will be used.

  • n_bins (int, optional, default=10) – Number of bins. A bigger number requires more data.

  • cmap (string or matplotlib.colors.Colormap instance, optional) – Colormap used for plotting the projection. View Matplotlib Colormap documentation for available options. https://matplotlib.org/users/colormaps.html

plot(ax=None)#

Create the plot :param ax: An Axes object to add the plot to :type ax: matplotlib.Axes

Rank1D#

class sklearn_evaluation.plot.Rank1D(algorithm='shapiro', features=None, figsize=(7, 7), orient='h', color='g', ax=None)#

Rank1D computes a score for each feature in the data set with a specific metric or algorithm (e.g. Shapiro-Wilk) then returns the features ranked as a bar plot.

Parameters
  • algorithm (one of {'shapiro', }, default: 'shapiro') – The ranking algorithm to use, default is β€˜Shapiro-Wilk.

  • features (list) – A list of feature names to use. If a DataFrame is passed features is None, feature names are selected as the columns of the DataFrame.

  • figsize (tuple, optional) – (width, height) for specifying the size of the plot.

  • orient ('h' or 'v', default='h') – Specifies a horizontal or vertical bar chart.

  • color (string) – Specify color for barchart

  • ax (matplotlib Axes, default: None) – The axis to plot the figure on. If None is passed in the current axes will be used (or generated if required).

ranks_#

An array of rank scores with shape (n,), where n is the number of features.

Type

ndarray

Examples

Visualize the Feature Rankings:

import matplotlib.pyplot as plt
from sklearn_evaluation.plot import Rank1D

from sklearn.datasets import load_breast_cancer as load_data

# load some data
X, y = load_data(return_X_y=True)

features = [
    "mean radius",
    "mean texture",
    "mean perimeter",
    "mean area",
    "mean smoothness",
    "mean compactness",
    "mean concavity",
    "mean concave points",
    "mean symmetry",
    "mean fractal dimension",
    "radius error",
    "texture error",
    "perimeter error",
    "area error",
    "smoothness error",
    "compactness error",
    "concavity error",
    "concave points error",
    "symmetry error",
    "fractal dimension error",
    "worst radius",
    "worst texture",
    "worst perimeter",
    "worst area",
    "worst smoothness",
    "worst compactness",
    "worst concavity",
    "worst concave points",
    "worst symmetry",
    "worst fractal dimension",
]

# plot feature rankings
rank1d = Rank1D(features=features, figsize=(14, 7))
rank1d.feature_ranks(X)
plt.show()
../_images/feature_ranking_1D.png

Notes

New in version 0.8.4.

feature_ranks(X)#
Parameters

X (array-like, shape (n_samples, n_features)) – Feature dataset to be ranked. Refer https://numpy.org/doc/stable/glossary.html#term-array-like

Returns

ax – Axes containing the plot

Return type

matplotlib Axes

feature_ranks_custom_algorithm(ranks)#

This method is useful if user wants to use custom algorithm for feature ranking.

Parameters

ranks (ndarray) – An n-dimensional, symmetric array of rank scores, where n is the number of features. E.g. for 1D ranking, it is (n,), for a 2D ranking it is (n,n).

Returns

ax – Axes containing the plot

Return type

matplotlib Axes

Rank2D#

class sklearn_evaluation.plot.Rank2D(algorithm='pearson', features=None, colormap='RdBu_r', figsize=(7, 7), ax=None)#

Rank2D performs pairwise comparisons of each feature in the data set with a specific metric or algorithm (e.g. Pearson correlation) then returns them ranked as a lower left triangle diagram.

Parameters
  • algorithm (str, default: 'pearson') – The ranking algorithm to use, one of: β€˜pearson’, β€˜covariance’, β€˜spearman’, or β€˜kendalltau’.

  • features (list) – A list of feature names to use. If a DataFrame is passed features is None, feature names are selected as the columns of the DataFrame.

  • colormap (string or cmap, default: 'RdBu_r') – optional string or matplotlib cmap to colorize lines Use either color to colorize the lines on a per class basis or colormap to color them on a continuous scale.

  • figsize (tuple, optional) – (width, height) for specifying the size of the plot

  • ax (matplotlib Axes, default: None) – The axis to plot the figure on. If None is passed in the current axes will be used (or generated if required).

ranks_#

An array of rank scores with shape (n,n), where n is the number of features.

Type

ndarray

Examples

Visualize the Feature Rankings by Pairwise Comparison:

import matplotlib.pyplot as plt
from sklearn_evaluation.plot import Rank2D

from sklearn.datasets import load_breast_cancer as load_data

# load some data
X, y = load_data(return_X_y=True)

features = [
    "mean radius",
    "mean texture",
    "mean perimeter",
    "mean area",
    "mean smoothness",
    "mean compactness",
    "mean concavity",
    "mean concave points",
    "mean symmetry",
    "mean fractal dimension",
    "radius error",
    "texture error",
    "perimeter error",
    "area error",
    "smoothness error",
    "compactness error",
    "concavity error",
    "concave points error",
    "symmetry error",
    "fractal dimension error",
    "worst radius",
    "worst texture",
    "worst perimeter",
    "worst area",
    "worst smoothness",
    "worst compactness",
    "worst concavity",
    "worst concave points",
    "worst symmetry",
    "worst fractal dimension",
]

# plot feature rankings
rank2d = Rank2D(features=features, figsize=(14, 14))
rank2d.feature_ranks(X)
plt.show()
../_images/feature_ranking_2D.png

Notes

New in version 0.8.4.

feature_ranks(X)#
Parameters

X (array-like, shape (n_samples, n_features)) – Feature dataset to be ranked. Refer https://numpy.org/doc/stable/glossary.html#term-array-like

Returns

ax – Axes containing the plot

Return type

matplotlib Axes

feature_ranks_custom_algorithm(ranks)#

This method is useful if user wants to use custom algorithm for feature ranking.

Parameters

ranks (ndarray) – An n-dimensional, symmetric array of rank scores, where n is the number of features. E.g. for 1D ranking, it is (n,), for a 2D ranking it is (n,n).

Returns

ax – Axes containing the plot

Return type

matplotlib Axes

Functional API#

calibration_curve#

sklearn_evaluation.plot.calibration_curve(y_true, probabilities, clf_names=None, n_bins=10, cmap='nipy_spectral', ax=None)#

Plots calibration curves for a set of classifier probability estimates. Calibration curves help determining whether you can interpret predicted probabilities as confidence level. For example, if we take a well-calibrated and take the instances where the score is 0.8, 80% of those instanes should be from the positive class. This function only works for binary classifiers.

Parameters
  • y_true (array-like, shape = [n_samples] or list with array-like:) – Ground truth (correct) target values. If passed a single array- object, it assumes all the probabilities have the same shape as y_true. If passed a list, it expects y_true[i] to have the same size as probabilities[i]

  • probabilities (list of array-like, shape (n_samples, 2) or (n_samples,)) – A list containing the outputs of binary classifiers’ predict_proba() method or decision_function() method.

  • clf_names (list of str, optional)) – A list of strings, where each string refers to the name of the classifier that produced the corresponding probability estimates in probabilities. If None, the names β€œClassifier 1”, β€œClassifier 2”, etc. will be used.

  • n_bins (int, optional, default=10) – Number of bins. A bigger number requires more data.

  • cmap (string or matplotlib.colors.Colormap instance, optional) – Colormap used for plotting the projection. View Matplotlib Colormap documentation for available options. https://matplotlib.org/users/colormaps.html

  • ax (matplotlib Axes) – Axes object to draw the plot onto, otherwise uses current Axes

Returns

ax – Axes containing the plot

Return type

matplotlib Axes

Examples

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.naive_bayes import GaussianNB

from sklearn_evaluation import plot

# generate data
X, y = make_classification(
    n_samples=20000, n_features=2, n_informative=2, n_redundant=0, random_state=0
)

# split data into train and test
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.33, random_state=0
)

rf = RandomForestClassifier()
lr = LogisticRegression()
nb = GaussianNB()

rf_probas = rf.fit(X_train, y_train).predict_proba(X_test)
lr_probas = lr.fit(X_train, y_train).predict_proba(X_test)
nb_probas = nb.fit(X_train, y_train).predict_proba(X_test)

# list of probabilities for different classifier
probabilities = [rf_probas, lr_probas, nb_probas]

clf_names = [
    "Random Forest",
    "Logistic Regression",
    "Gaussian Naive Bayes",
]

# plot calibration curve
plot.calibration_curve(y_test, probabilities, clf_names=clf_names)
../_images/calibration_curve.png

classification_report#

sklearn_evaluation.plot.classification_report(y_true, y_pred, *, target_names=None, sample_weight=None, zero_division=0, ax=None)#

Classification report

Parameters
  • y_true (array-like, shape = [n_samples]) – Correct target values (ground truth)

  • y_pred (array-like, shape = [n_samples]) – Target predicted classes (estimator predictions)

  • target_names (list) – List containing the names of the target classes. List must be in order e.g. ['Label for class 0', 'Label for class 1']. If None, generic labels will be generated e.g. ['Class 0', 'Class 1']

  • sample_weight (array-like of shape (n_samples,), default=None) – Sample weights.

  • zero_division (bool, 0 or 1) – Sets the value to return when there is a zero division.

  • ax (matplotlib Axes) – Axes object to draw the plot onto, otherwise uses current Axes

Returns

ax – Axes containing the plot

Return type

matplotlib Axes

Examples

Plot a Classification Report for binary classification:

from sklearn import datasets
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

from sklearn_evaluation import plot

# generate data
X, y = datasets.make_classification(200, 10, n_informative=5, class_sep=0.65)

# split data into train and test
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)

est = RandomForestClassifier()
est.fit(X_train, y_train)

y_pred = est.predict(X_test)

# plot classification report for binary classification
plot.classification_report(y_test, y_pred, target_names=["Not spam", "Spam"])
../_images/classification_report.png

Plot a Classification Report for multi-class classification:

from sklearn import datasets
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

from sklearn_evaluation import plot

# generate data
X, y = datasets.make_classification(
    200, 10, n_informative=5, class_sep=0.65, n_classes=3
)

# split data into train and test
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)

est = RandomForestClassifier()
est.fit(X_train, y_train)

y_pred = est.predict(X_test)

# plot classification report for multi-class classification
plot.classification_report(y_test, y_pred)
../_images/classification_report_multiclass.png

confusion_matrix#

sklearn_evaluation.plot.confusion_matrix(y_true, y_pred, target_names=None, normalize=False, cmap=None, ax=None)#

Plot confusion matrix.

See also

ConfusionMatrix

Parameters
  • y_true (array-like, shape = [n_samples]) – Correct target values (ground truth).

  • y_pred (array-like, shape = [n_samples]) – Target predicted classes (estimator predictions).

  • target_names (list) – List containing the names of the target classes. List must be in order e.g. ['Label for class 0', 'Label for class 1']. If None, generic labels will be generated e.g. ['Class 0', 'Class 1']

  • ax (matplotlib Axes) – Axes object to draw the plot onto, otherwise uses current Axes

  • normalize (bool) – Normalize the confusion matrix

  • cmap (matplotlib Colormap) – If None uses a modified version of matplotlib’s OrRd colormap.

Returns

ax – Axes containing the plot

Return type

matplotlib Axes

Examples

Plot a Confusion Matrix for binary classifier:

from sklearn import datasets
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

from sklearn_evaluation import plot

# generate data
X, y = datasets.make_classification(200, 10, n_informative=5, class_sep=0.65)

# split data into train and test
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)

est = RandomForestClassifier()
est.fit(X_train, y_train)

y_pred = est.predict(X_test)
y_true = y_test

# plot confusion matrix
plot.confusion_matrix(y_true, y_pred)
../_images/confusion_matrix.png

cumulative_gain#

sklearn_evaluation.plot.cumulative_gain(y_true, y_score, figsize=None, title_fontsize='large', text_fontsize='medium', ax=None)#

Generates the Cumulative Gains Plot from labels and scores/probabilities The cumulative gains chart is used to determine the effectiveness of a binary classifier. The implementation here works only for binary classification.

Parameters
  • y_true (array-like, shape=[n_samples,]) – Ground truth (correct) target values. Refer: https://numpy.org/doc/stable/glossary.html#term-array-like

  • y_score (array-like, shape=[n_samples, n_classes]) – Prediction probabilities for each class returned by a classifier. Refer: https://numpy.org/doc/stable/glossary.html#term-array-like

  • figsize (2-tuple, optional) – Tuple denoting figure size of the plot e.g. (6, 6). Defaults to None.

  • title_fontsize (string or int, optional) – Matplotlib-style fontsizes. Use e.g. β€œsmall”, β€œmedium”, β€œlarge” or integer-values. Defaults to β€œlarge”.

  • text_fontsize (string or int, optional) – Matplotlib-style fontsizes. Use e.g. β€œsmall”, β€œmedium”, β€œlarge” or integer-values. Defaults to β€œmedium”.

  • ax (matplotlib Axes, optional) – Axes object to draw the plot onto, otherwise uses current Axes

Returns

ax – Axes containing the plot

Return type

matplotlib Axes

Examples

Plot the Cumulative Gains of a binary classifier:

from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_breast_cancer as load_data
from sklearn_evaluation.plot import cumulative_gain

# load data
X, y = load_data(return_X_y=True)

# split data into train and test
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=123
)

model = LogisticRegression(random_state=101)
model = model.fit(X_train, y_train)
y_probas = model.predict_proba(X_test)

# plot cumulative gain curve
cumulative_gain(y_test, y_probas)
../_images/cumulative_gain.png

Notes

New in version 0.8.4.

elbow_curve#

sklearn_evaluation.plot.elbow_curve(X, clf, range_n_clusters=None, n_jobs=1, show_cluster_time=True, ax=None)#

Plots elbow curve of different values of K of a clustering algorithm.

Parameters
  • X (array-like, shape = [n_samples, n_features]:) – Data to cluster, where n_samples is the number of samples and n_features is the number of features. Refer https://numpy.org/doc/stable/glossary.html#term-array-like

  • clf – Clusterer instance that implements fit,``fit_predict``, and score methods, and an range_n_clusters hyperparameter. e.g. sklearn.cluster.KMeans instance

  • range_n_clusters (None or list of int, optional) – List of n_clusters for which to plot the explained variances. Defaults to [1, 3, 5, 7, 9, 11].

  • n_jobs (int, optional) – Number of jobs to run in parallel. Defaults to 1.

  • show_cluster_time (bool, optional) – Include plot of time it took to cluster for a particular K.

  • ax (matplotlib.axes.Axes, optional) – The axes upon which to plot the curve. If None, the plot is drawn on the current Axes

Returns

ax – Axes containing the plot

Return type

matplotlib Axes

Examples

Plot the Elbow Curve:

from sklearn.cluster import KMeans
from sklearn.datasets import make_blobs

from sklearn_evaluation import plot

# generate data
X, _ = make_blobs(n_samples=100, centers=3, n_features=5, random_state=0)
kmeans = KMeans(random_state=1, n_init=5)

# plot elbow curve
plot.elbow_curve(X, kmeans, range_n_clusters=range(1, 30))
../_images/elbow_curve.png

elbow_curve_from_results#

sklearn_evaluation.plot.elbow_curve_from_results(n_clusters, sum_of_squares, times, ax=None)#

Same as elbow_curve, but it takes the number of clusters and sum of squares as inputs. Useful if you want to train the models yourself.

Examples

Plot the Elbow Curve from the results:

import time
import numpy as np

from sklearn.cluster import KMeans
from sklearn.datasets import make_blobs

from sklearn_evaluation import plot

# generate data
X, _ = make_blobs(n_samples=100, centers=3, n_features=5, random_state=0)

n_clusters = range(1, 30)
sum_of_squares = []
cluster_times = []
for i in n_clusters:
    start = time.time()
    kmeans = KMeans(n_clusters=i, n_init=5)
    sum_of_squares.append(kmeans.fit(X).score(X))
    cluster_times.append(time.time() - start)

sum_of_squares = np.absolute(sum_of_squares)

# plot the elbow curve from the results
plot.elbow_curve_from_results(n_clusters, sum_of_squares, cluster_times)
../_images/elbow_curve_from_results.png

feature_importances#

sklearn_evaluation.plot.feature_importances(data, top_n=None, feature_names=None, orientation='horizontal', ax=None)#

Get and order feature importances from a scikit-learn model or from an array-like structure. If data is a scikit-learn model with sub-estimators (e.g. RandomForest, AdaBoost) the function will compute the standard deviation of each feature.

Parameters
  • data (sklearn model or array-like structure) – Object to get the data from.

  • top_n (int) – Only get results for the top_n features.

  • feature_names (array-like) – Feature names

  • orientation (('horizontal', 'vertical')) – Bar plot orientation

  • ax (matplotlib Axes) – Axes object to draw the plot onto, otherwise uses current Axes

Returns

ax – Axes containing the plot

Return type

matplotlib Axes

Examples

Plot Feature Importances:

from sklearn import datasets
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

from sklearn_evaluation import plot

# generate data
X, y = datasets.make_classification(200, 20, n_informative=5, class_sep=0.65)

# split data into train and test
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)

model = RandomForestClassifier(n_estimators=1)
model.fit(X_train, y_train)

# plot all features
ax = plot.feature_importances(model)

# only top 5
plot.feature_importances(model, top_n=5)
../_images/feature_importances.png

ks_statistic#

sklearn_evaluation.plot.ks_statistic(y_true, y_score, figsize=None, title_fontsize='large', text_fontsize='medium', ax=None)#

Generates the KS Statistic plot from labels and scores/probabilities

Parameters
  • y_true (array-like, shape=[n_samples,]) – Ground truth target values. Refer: https://numpy.org/doc/stable/glossary.html#term-array-like

  • y_score (array-like, shape=[n_samples, n_classes]) – Prediction probabilities for each class returned by a classifier. Refer: https://numpy.org/doc/stable/glossary.html#term-array-like

  • figsize (2-tuple, optional) – Tuple denoting figure size of the plot e.g. (6, 6). Defaults to None.

  • title_fontsize (string or int, optional) – Matplotlib-style fontsizes. Use e.g. β€œsmall”, β€œmedium”, β€œlarge” or integer-values. Defaults to β€œlarge”.

  • text_fontsize (string or int, optional) – Matplotlib-style fontsizes. Use e.g. β€œsmall”, β€œmedium”, β€œlarge” or integer-values. Defaults to β€œmedium”.

  • ax (matplotlib Axes, optional) – Axes object to draw the plot onto, otherwise uses current Axes

Returns

ax – Axes containing the plot

Return type

matplotlib Axes

Examples

Plot a KS Statistic plot:

from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_breast_cancer as load_data

from sklearn_evaluation.plot import ks_statistic

# load data
X, y = load_data(return_X_y=True)

# split data into train and test
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=123
)
model = LogisticRegression(random_state=101)

model = model.fit(X_train, y_train)
y_probas = model.predict_proba(X_test)

# plot the KS statistic
ks_statistic(y_test, y_probas)
../_images/ks_statistics.png

Notes

New in version 0.8.4.

learning_curve#

sklearn_evaluation.plot.learning_curve(train_scores, test_scores, train_sizes, ax=None)#

Plot a learning curve

Plot a metric vs number of examples for the training and test set

Parameters
  • train_scores (array-like) – Scores for the training set

  • test_scores (array-like) – Scores for the test set

  • train_sizes (array-like) – Relative or absolute numbers of training examples used to generate the learning curve

  • ax (matplotlib Axes) – Axes object to draw the plot onto, otherwise uses current Axes

Returns

ax – Axes containing the plot

Return type

matplotlib Axes

Examples

Plot a Learning Curve:

from sklearn.model_selection import learning_curve
from sklearn import model_selection
from sklearn.naive_bayes import GaussianNB
from sklearn.svm import SVC
from sklearn.datasets import load_digits
import numpy as np

from sklearn_evaluation import plot

# load data
digits = load_digits()
X, y = digits.data, digits.target

# Cross validation with 100 iterations to get smoother mean test and train
# score curves, each time with 20% data randomly selected as a validation set.
cv = model_selection.ShuffleSplit(digits.data.shape[0], test_size=0.2, random_state=0)
cv = 5
estimator = GaussianNB()
train_sizes = np.linspace(0.1, 1.0, 5)
train_sizes, train_scores, test_scores = learning_curve(
    estimator, X, y, cv=cv, n_jobs=1, train_sizes=train_sizes
)
plot.learning_curve(train_scores, test_scores, train_sizes)

# SVC is more expensive so we do a lower number of CV iterations:
cv = model_selection.ShuffleSplit(digits.data.shape[0], test_size=0.2, random_state=0)
cv = 5
estimator = SVC(gamma=0.001)
train_sizes = np.linspace(0.1, 1.0, 5)
train_sizes, train_scores, test_scores = learning_curve(
    estimator, X, y, cv=cv, n_jobs=1, train_sizes=train_sizes
)

# plot learning curve
plot.learning_curve(train_scores, test_scores, train_sizes)
../_images/learning_curve_00.png
../_images/learning_curve_01.png

lift_curve#

sklearn_evaluation.plot.lift_curve(y_true, y_score, ax=None, figsize=None, title_fontsize='large', text_fontsize='medium')#

Generates the Lift Curve from labels and scores/probabilities The lift curve is used to determine the effectiveness of a binary classifier. A detailed explanation can be found at http://www2.cs.uregina.ca/~dbd/cs831/notes/lift_chart/lift_chart.html. The implementation here works only for binary classification.

Parameters
  • y_true (array-like, shape=[n_samples,]) – Ground truth (correct) target values. Refer: https://numpy.org/doc/stable/glossary.html#term-array-like

  • y_score (array-like, shape=[n_samples, n_classes]) – Prediction probabilities for each class returned by a classifier. Refer: https://numpy.org/doc/stable/glossary.html#term-array-like

  • figsize (2-tuple, optional) – Tuple denoting figure size of the plot e.g. (6, 6). Defaults to None.

  • title_fontsize (string or int, optional) – Matplotlib-style fontsizes. Use e.g. β€œsmall”, β€œmedium”, β€œlarge” or integer-values. Defaults to β€œlarge”.

  • text_fontsize (string or int, optional) – Matplotlib-style fontsizes. Use e.g. β€œsmall”, β€œmedium”, β€œlarge” or integer-values. Defaults to β€œmedium”.

  • ax (matplotlib Axes, optional) – Axes object to draw the plot onto, otherwise uses current Axes

Returns

ax – Axes containing the plot

Return type

matplotlib Axes

Examples

Plot the Lift Curve for a binary classifier:

from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_breast_cancer as load_data
from sklearn_evaluation.plot import lift_curve

# load data
X, y = load_data(return_X_y=True)

# split data into train and test
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=123
)
model = LogisticRegression(random_state=101)

model = model.fit(X_train, y_train)
y_probas = model.predict_proba(X_test)

# plot lift curve
lift_curve(y_test, y_probas)
../_images/lift_curve.png

Notes

New in version 0.8.4.

metrics_at_thresholds#

sklearn_evaluation.plot.metrics_at_thresholds(fn, y_true, y_score, n_thresholds=10, start=0.0, ax=None)#

Plot metrics at increasing thresholds

pca#

sklearn_evaluation.plot.pca(X, y=None, target_names=None, n_components=2, colors=None, ax=None)#

Plot principle component analysis curve.

Parameters
  • X (array-like, shape = [n_samples, n_features]) – Training data, where n_samples is the number of samples and n_features is the number of features

  • y (array-like or list or None) – set None if ignored otherwise, pass the targets here

  • target_names (list, optional) – list of target variable names

  • n_components (int, float or 'mle', default=2) – Number of components to keep. If 2, it generates the plot of first component vs second component. If >=3, all pairwise comparisons are generated.

  • colors (list, optional) – colors to be used for the scatter plots for each target. If not passed random colors are generated.

  • ax (list of matplotlib Axes, optional) – Axes object to draw the plot onto, otherwise uses current Axes. If passed the list should have (n_components * n_components-1)/2 Axes objects

Notes

New in version 0.10.1.

Returns

ax – Axes containing the plot

Return type

matplotlib Axes

Examples

Plot PCA of 3 components:

from sklearn_evaluation import plot
from sklearn.datasets import load_iris as load_data

# Load data
X, y = load_data(return_X_y=True)

# plot pca of 3 components
plot.pca(X, y, n_components=3, target_names=["Setosa", "Versicolor", "Virginica"])
../_images/pca_00.png
../_images/pca_01.png
../_images/pca_02.png

precision_at_proportions#

sklearn_evaluation.plot.precision_at_proportions(y_true, y_score, ax=None)#

Plot precision values at different proportions.

Parameters
  • y_true (array-like) – Correct target values (ground truth).

  • y_score (array-like) – Target scores (estimator predictions).

  • ax (matplotlib Axes) – Axes object to draw the plot onto, otherwise uses current Axes

Returns

ax – Axes containing the plot

Return type

matplotlib Axes

precision_recall#

sklearn_evaluation.plot.precision_recall(y_true, y_score, ax=None)#

Plot precision-recall curve.

Parameters
  • y_true (array-like, shape = [n_samples]) – Correct target values (ground truth).

  • y_score (array-like, shape = [n_samples] or [n_samples, 2] for binary) – classification or [n_samples, n_classes] for multiclass Target scores (estimator predictions).

  • ax (matplotlib Axes) – Axes object to draw the plot onto, otherwise uses current Axes

Notes

It is assumed that the y_score parameter columns are in order. For example, if y_true = [2, 2, 1, 0, 0, 1, 2], then the first column in y_score must contain the scores for class 0, second column for class 1 and so on.

Returns

ax – Axes containing the plot

Return type

matplotlib Axes

Examples

Plot a Precision-Recall Curve for binary classification:

import matplotlib.pyplot as plt
from sklearn import datasets
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

from sklearn_evaluation import plot

# generate data
data = datasets.make_classification(200, 10, n_informative=5, class_sep=0.65)
X = data[0]
y = data[1]

# split data into train and test
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)

est = RandomForestClassifier()
est.fit(X_train, y_train)

y_score = est.predict_proba(X_test)
y_true = y_test

# plot precision recall curve
plot.precision_recall(y_true, y_score)
plt.show()
../_images/precision_recall.png

Plot a Precision-Recall Curve for multi-class classification:

import matplotlib.pyplot as plt
from sklearn import datasets
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

from sklearn_evaluation import plot

# generate data for multiclass classification
data = datasets.make_classification(
    n_samples=1000,
    n_features=10,
    n_informative=5,
    n_classes=5,
    n_redundant=0,
    n_clusters_per_class=1,
    random_state=0,
)
X = data[0]
y = data[1]

# split data into train and test
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)

est = RandomForestClassifier()
est.fit(X_train, y_train)

y_score = est.predict_proba(X_test)
y_true = y_test

# plot precision recall curve
plot.precision_recall(y_true, y_score)
plt.show()
../_images/precision_recall_multiclass.png

prediction_error#

sklearn_evaluation.plot.prediction_error(y_true, y_pred, ax=None)#

Plot the scatter plot of measured values v. predicted values, with an identity line and a best fitted line to show the prediction difference.

Parameters
  • y_true (array-like, shape = [n_samples]) – Measured target values (ground truth).

  • y_pred (array-like, shape = [n_samples]) – Predicted target values.

  • ax (matplotlib Axes) – Axes object to draw the plot onto, otherwise uses current Axes

Returns

ax – Axes containing the plot

Return type

matplotlib Axes

Examples

Plot a Prediction Error Scatter plot:

from sklearn.datasets import load_diabetes
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn_evaluation import plot

# load data
X, y = load_diabetes(return_X_y=True)

# split data into train and test
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)

reg = LinearRegression()
reg.fit(X_train, y_train)

y_pred = reg.predict(X_test)

# plot prediction error
plot.prediction_error(y_test, y_pred)
../_images/prediction_error.png

residuals#

sklearn_evaluation.plot.residuals(y_true, y_pred, ax=None)#

Plot the residuals between measured and predicted values.

Parameters
  • y_true (array-like, shape = [n_samples]) – Measured target values (ground truth).

  • y_pred (array-like, shape = [n_samples]) – Predicted target values.

  • ax (matplotlib Axes) – Axes object to draw the plot onto, otherwise uses current Axes

Returns

ax – Axes containing the plot

Return type

matplotlib Axes

Examples

Plot a Residuals Scatter plot:

from sklearn.datasets import load_diabetes
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn_evaluation import plot

# load data
X, y = load_diabetes(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)

reg = LinearRegression()
reg.fit(X_train, y_train)

y_pred = reg.predict(X_test)

# plot residuals
plot.residuals(y_test, y_pred)
../_images/residuals.png

roc#

sklearn_evaluation.plot.roc(y_true, y_score, ax=None)#

Plot ROC curve

Parameters
  • y_true (array-like, shape = [n_samples]) –

    Correct target values (ground truth).

    e.g

    ”classes” format : [0, 1, 2, 0, 1, …] or [β€˜virginica’, β€˜versicolor’, β€˜virginica’, β€˜setosa’, …]

    one-hot encoded classes[[0, 0, 1],

    [1, 0, 0]]

  • y_score (array-like, shape = [n_samples] or [n_samples, 2] for binary) –

    classification or [n_samples, n_classes] for multiclass Target scores (estimator predictions).

    e.g

    ”scores” format[[0.1, 0.1, 0.8],

    [0.7, 0.15, 0.15]]

  • ax (matplotlib Axes, default: None) – Axes object to draw the plot onto, otherwise uses current Axes

Notes

It is assumed that the y_score parameter columns are in order. For example, if y_true = [2, 2, 1, 0, 0, 1, 2], then the first column in y_score must contain the scores for class 0, second column for class 1 and so on.

See also

ROC

Examples

Plot a ROC Curve for binary classification:

from sklearn import datasets
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

from sklearn_evaluation import plot

# generate data
X, y = datasets.make_classification(200, 10, n_informative=5, class_sep=0.65)

# split data into train and test
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)

est = RandomForestClassifier()
est.fit(X_train, y_train)

# y_pred = est.predict(X_test)
y_score = est.predict_proba(X_test)

# plot roc curve
plot.roc(y_test, y_score)
../_images/roc.png

scores_distribution#

sklearn_evaluation.plot.scores_distribution(y_scores, n_bins=5, title='Predictions distribution', color='b', ax=None)#

Generate a histogram from model’s predictions

Parameters
  • y_scores (array-like, shape=(n_samples, )) – Scores produced by a trained model for a single class

  • n_bins (int, default=5) – Number of histogram bins

  • title (title of the plot. Defaults to Predictions Distribution) –

  • color (color of the histogram. Defaults to blue.) –

  • ax (matplotlib Axes, default=None) – Axes object to draw the plot onto, otherwise uses current Axes

Returns

ax – Axes containing the plot

Return type

matplotlib Axes

Examples

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression

from sklearn_evaluation import plot

# generate data
X, y = make_classification(
    n_samples=10000, n_features=2, n_informative=2, n_redundant=0, random_state=0
)

# split data into train and test
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.33, random_state=0
)

clf = LogisticRegression()

y_scores = clf.fit(X_train, y_train).predict_proba(X_test)

# plot scores distribution
plot.scores_distribution(y_scores[:, 1], n_bins=10)
../_images/scores_distribution.png

silhouette_analysis#

sklearn_evaluation.plot.silhouette_analysis(X, clf, range_n_clusters=None, metric='euclidean', figsize=None, cmap='nipy_spectral', text_fontsize='medium', ax=None)#

Plots silhouette analysis of clusters provided.

Parameters
  • X (array-like, shape = [n_samples, n_features]:) – Cluster data, where n_samples is the number of samples and n_features is the number of features. Refer https://numpy.org/doc/stable/glossary.html#term-array-like

  • clf – Clusterer instance that implements fit,``fit_predict``, and score methods, and an n_clusters hyperparameter. e.g. sklearn.cluster.KMeans instance

  • range_n_clusters (None or list of int, optional) – List of n_clusters for which to plot the silhouette scores. Defaults to [2, 3, 4, 5, 6].

  • metric (string or callable, optional:) – The metric to use when calculating distance between instances in a feature array. If metric is a string, it must be one of the options allowed by sklearn.metrics.pairwise.pairwise_distances. If X is the distance array itself, use β€œprecomputed” as the metric.

  • figsize (2-tuple, optional:) – Tuple denoting figure size of the plot e.g. (6, 6). Defaults to None.

  • cmap (string or matplotlib.colors.Colormap instance, optional:) – Colormap used for plotting the projection. View Matplotlib Colormap documentation for available options. https://matplotlib.org/users/colormaps.html

  • text_fontsize (string or int, optional:) – Matplotlib-style fontsizes. Use e.g. β€œsmall”, β€œmedium”, β€œlarge” or integer-values. Defaults to β€œmedium”.

  • ax (matplotlib.axes.Axes, optional:) – The axes upon which to plot the curve. If None, the plot is drawn on a new set of axes.

Returns

ax – Axes containing the plot

Return type

matplotlib Axes

Examples

Plot the Silhouette Analysis:

from sklearn.cluster import KMeans
from sklearn.datasets import make_blobs

from sklearn_evaluation import plot

# generate data
X, y = make_blobs(
    n_samples=500,
    n_features=2,
    centers=4,
    cluster_std=1,
    center_box=(-10.0, 10.0),
    shuffle=True,
    random_state=1,
)

kmeans = KMeans(random_state=1, n_init=5)

# plot silhouette analysis of provided clusters
plot.silhouette_analysis(X, kmeans, range_n_clusters=[3])
../_images/silhouette_plot_basic.png

Notes

New in version 0.8.3.

silhouette_analysis_from_results#

sklearn_evaluation.plot.silhouette_analysis_from_results(X, cluster_labels, metric='euclidean', figsize=None, cmap='nipy_spectral', text_fontsize='medium', ax=None)#

Same as silhouette_plot but takes list of cluster_labels as input. Useful if you want to train the model yourself

Examples

Plot the Silhouette Analysis from the results:

from sklearn.cluster import KMeans
from sklearn.datasets import make_blobs

from sklearn_evaluation import plot

# generate data
X, y = make_blobs(
    n_samples=500,
    n_features=2,
    centers=4,
    cluster_std=1,
    center_box=(-10.0, 10.0),
    shuffle=True,
    random_state=1,
)

cluster_labels = []

# Cluster labels for four clusters
kmeans = KMeans(n_clusters=4, n_init=5)
cluster_labels.append(kmeans.fit_predict(X))

# Cluster labels for five clusters
kmeans = KMeans(n_clusters=5, n_init=5)
cluster_labels.append(kmeans.fit_predict(X))

# plot silhouette analysis from provided list of cluster labels
plot.silhouette_analysis_from_results(X, cluster_labels)
../_images/silhouette_plot_from_results_00.png
../_images/silhouette_plot_from_results_01.png

target_analysis#

sklearn_evaluation.plot.target_analysis(y_train, y_test=None, labels=None, colors=None, ax=None)#

Target analysis plot for visualising class imbalance.

There are two modes:

  1. Balance mode: if only y_train is specified

  2. Compare mode: if both train and test are specified

In balance mode, the bar chart is displayed with each class as its own color. In compare mode, a side-by-side bar chart is displayed colored by train or test respectively.

Parameters
  • y_train (array-like) – Array or list of shape (n,) that contains discrete data. Refer https://numpy.org/doc/stable/glossary.html#term-array-like

  • y_test (array-like, optional) – Array or list of shape (m,) that contains discrete data. If specified, the bar chart will be drawn in compare mode. Refer https://numpy.org/doc/stable/glossary.html#term-array-like

  • labels (list, optional) – A list of class names for the x-axis if the target is already encoded. Ensure that the labels are ordered lexicographically with respect to the values in the target. A common use case is to pass LabelEncoder.classes_ as this parameter. If not specified, the labels in the data will be used.

  • colors (list of strings) – Specify colors for the barchart.

  • ax (matplotlib.axes.Axes, optional) – The axes upon which to plot the curve. If None, the plot is drawn on the current Axes

Returns

ax – Axes containing the plot

Return type

matplotlib Axes

Examples

Plot the Target Analysis:

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split

from sklearn_evaluation import plot


# load a dataset
iris = load_iris()
X = iris.data
y = iris.target

# split data into train and test
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5)

# plot target analysis
plot.target_analysis(y_train, y_test)
../_images/target_analysis.png

Notes

New in version 0.8.3.

validation_curve#

sklearn_evaluation.plot.validation_curve(train_scores, test_scores, param_range, param_name=None, semilogx=False, ax=None)#

Plot a validation curve

Plot a metric vs hyperparameter values for the training and test set

Parameters
  • train_scores (array-like) – Scores for the training set

  • test_scores (array-like) – Scores for the test set

  • param_range (str) – Hyperparameter values used to generate the curve

  • param_range – Hyperparameter name

  • semilgo (bool) – Sets a log scale on the x axis

  • ax (matplotlib Axes) – Axes object to draw the plot onto, otherwise uses current Axes

Returns

ax – Axes containing the plot

Return type

matplotlib Axes

Examples

Plot a Validation Curve:

import matplotlib.pyplot as plt
import numpy as np
from sklearn.datasets import load_digits
from sklearn.svm import SVC
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import validation_curve

from sklearn_evaluation import plot

# load data
digits = load_digits()
X, y = digits.data, digits.target

param_range = np.logspace(-6, -1, 5)
param_name = "gamma"
train_scores, test_scores = validation_curve(
    SVC(),
    X,
    y,
    param_name=param_name,
    param_range=param_range,
    cv=5,
    scoring="accuracy",
    n_jobs=1,
)

# plot validation curve for "gamma" hyperparameter
plot.validation_curve(train_scores, test_scores, param_range, param_name, semilogx=True)
plt.show()
../_images/validation_curve_00_00.png
param_range = np.array([1, 10, 100])
param_name = "n_estimators"
train_scores, test_scores = validation_curve(
    RandomForestClassifier(),
    X,
    y,
    param_name=param_name,
    param_range=param_range,
    cv=10,
    scoring="accuracy",
    n_jobs=1,
)

# plot validation curve for "n_estimators" hyperparameter
plot.validation_curve(
    train_scores, test_scores, param_range, param_name, semilogx=False
)
../_images/validation_curve_01_00.png

cooks_distance#

sklearn_evaluation.plot.cooks_distance(X, y, ax=None)#

Plots cooks distance.

Parameters
Returns

ax – Axes containing the plot

Return type

matplotlib Axes

Examples

Plot a Cook’s Distance Bar plot:

from sklearn.datasets import make_regression
from sklearn_evaluation import plot

# generate data
X, y = make_regression(
    n_samples=100,
    n_features=14,
    n_informative=6,
    bias=1.2,
    noise=49.8,
    tail_strength=0.6,
    random_state=637,
)

# plot cooks distance
plot.cooks_distance(X, y)
../_images/cooks_distance.png

Notes

New in version 0.8.4.

report_evaluation#

sklearn_evaluation.report.evaluate_model(model, y_true, y_pred, X_test=None, y_score=None, report_title=None)#

Evaluates a given model and generates an HTML report

Parameters
  • model (estimator) – An estimator to evaluate.

  • y_true (array-like) – Correct target values (ground truth).

  • y_pred (array-like) – Target predicted classes (estimator predictions).

  • y_score (array-like, default None) – Target scores (estimator predictions).

  • report_title (str, default "Model evaluation - {model_name}") –

Examples

Generate evaluation report for RandomForestClassifier

from sklearn.ensemble import RandomForestClassifier
import urllib.request
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn_evaluation.report import evaluate_model

urllib.request.urlretrieve(
    "https://raw.githubusercontent.com/sharmaroshan/"
    + "Heart-UCI-Dataset/master/heart.csv",
    filename="heart.csv",
)

data = pd.read_csv("heart.csv")


column = "fbs"
X = data.drop(column, axis=1)
y = data[column]

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=2023
)


model = RandomForestClassifier()
model.fit(X_train, y_train)

y_pred = model.predict(X_test)
y_score = model.predict_proba(X_test)

report = evaluate_model(model, y_test, y_pred, y_score=y_score)
../_images/report_evaluation_00.png
../_images/report_evaluation_01.png
../_images/report_evaluation_02.png
../_images/report_evaluation_03.png
../_images/report_evaluation_04.png
../_images/report_evaluation_05.png

Notes

New in version 0.11.4.

report_comparison#

sklearn_evaluation.report.compare_models(model_a, model_b, X_test, y_true, report_title=None)#

Compares two models and generates an HTML report

Parameters
  • model_a (estimator) – An estimator to compare.

  • model_b (estimator) – An estimator to compare.

  • X_test (array-like of shape (n_samples, n_features)) – Training data, where n_samples is the number of samples and n_features is the number of features.

  • y_true (array-like) – Correct target values (ground truth).

  • report_title (str, default "Compare models - {model_a} vs {model_b}") –

Examples

Compare DecisionTreeClassifier and RandomForestClassifier

import pandas as pd
import urllib.request
from sklearn.model_selection import train_test_split
from sklearn_evaluation.report import compare_models
from sklearn.ensemble import RandomForestClassifier
from sklearn.tree import DecisionTreeClassifier

urllib.request.urlretrieve(
    "https://raw.githubusercontent.com/sharmaroshan/"
    + "Heart-UCI-Dataset/master/heart.csv",
    filename="heart.csv",
)

data = pd.read_csv("heart.csv")

column = "target"
X = data.drop(column, axis=1)
y = data[column]

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=2023
)


model_a = RandomForestClassifier()
model_a.fit(X_train, y_train)


model_b = DecisionTreeClassifier()
model_b.fit(X_train, y_train)


report = compare_models(model_a, model_b, X_test, y_test)
../_images/report_comparison_00.png
../_images/report_comparison_01.png
../_images/report_comparison_02.png
../_images/report_comparison_03.png
../_images/report_comparison_04.png
../_images/report_comparison_05.png
../_images/report_comparison_06.png
../_images/report_comparison_07.png
../_images/report_comparison_08.png
../_images/report_comparison_09.png
../_images/report_comparison_10.png
../_images/report_comparison_11.png
../_images/report_comparison_12.png
../_images/report_comparison_13.png

Notes

New in version 0.11.4.