Optimal number of clusters

Learn how to easily evaluate clustering algorithms and determine the optimal number of clusters using the below methods:

• Elbow curve plots the sum of squared errors (squared errors summed across all points) for each value of k.

• Silhouette analysis determines if individual points are correctly assigned to their clusters.

import matplotlib
from sklearn import datasets
from sklearn.cluster import KMeans
from sklearn_evaluation import plot

matplotlib.rcParams["figure.figsize"] = (7, 7)
matplotlib.rcParams["font.size"] = 18

# get data for clustering
X, y = datasets.make_blobs(
    n_samples=500,
    n_features=2,
    centers=4,
    cluster_std=1,
    center_box=(-10.0, 10.0),
    shuffle=True,
    random_state=1,
)

# Fit kMeans on the data
kmeans = KMeans(random_state=10, n_init=5)

Elbow curve

Elbow curve helps to identify the point at which the plot starts to become parallel to the x-axis. The K value corresponding to this point is the optimal number of clusters. In the below plot one is likely to select k=4. Currently the kmeans argument input only accepts Kmeans, MiniBatchKMeans, and BisectingKMeans.

plot.elbow_curve(X, kmeans, range_n_clusters=range(1, 30))

<Axes: title={'center': 'Elbow Plot'}, xlabel='Number of clusters', ylabel='Sum of Squared Errors'>

Tip

If you want to train the models yourself, you can use elbow_curve_from_results to plot.

Silhouette plot

The below plot shows that n_clusters value of 3, 5 and 6 are a bad pick for the given data. One is likely to select between 2 and 4 n_clusters.

silhouette = plot.silhouette_analysis(X, kmeans)

Tip

If you want to train the models yourself, you can use silhouette_analysis_from_results to plot.