Notebooks as experiments#

The .ipynb format is capable of storing tables and charts in a standalone file. This makes it a great choice for model evaluation reports. NotebookCollection allows you to retrieve results from previously executed notebooks to compare them.

import urllib.request

from ploomber_engine import execute_notebook
import jupytext

from sklearn_evaluation import NotebookCollection

Let’s first generate a few notebooks, we have a train.py script that trains a single model, let’s convert it to a jupyter notebook:

# download script
url = "https://raw.githubusercontent.com/ploomber/sklearn-evaluation/master/docs-assets/nb-collection/train.py"  # noqa
urllib.request.urlretrieve(url, filename="train.py")

# convert
nb = jupytext.read("train.py")
jupytext.write(nb, "train.ipynb")

We use papermill to execute the notebook with different parameters, we’ll train 4 models: 2 random forest, a linear regression and a support vector regression:

# models with their corresponding parameters
params = [
    {"model": "sklearn.ensemble.RandomForestRegressor", "params": {"n_estimators": 50}},
    {
        "model": "sklearn.ensemble.RandomForestRegressor",
        "params": {"n_estimators": 100},
    },
    {"model": "sklearn.linear_model.LinearRegression", "params": {}},
    {"model": "sklearn.svm.LinearSVR", "params": {}},
]

# ids to identify each experiment
ids = [
    "random_forest_1",
    "random_forest_2",
    "linear_regression",
    "support_vector_regression",
]

# output files
files = [f"{i}.ipynb" for i in ids]

# execute notebooks using papermill
for f, p in zip(files, params):
    execute_notebook("train.ipynb", output_path=f, parameters=p, progress_bar=False)

To use NotebookCollection, we pass a a list of paths, and optionally, ids for each notebook (uses paths by default).

The only requirement is that cells whose output we want to extract must have tags, each tag then becomes a key in the notebook collection. For instructions on adding tags, see this.

Extracted tables add colors to certain cells to identify the best and worst metrics. By default, it assumes that metrics are errors (smaller is better). If you are using scores (larger is better), pass scores=True, if you have both, pass a list of scores:

nbs = NotebookCollection(paths=files, ids=ids, scores=["r2"])

To get a list of tags available:

list(nbs)

['model_name', 'feature_names', 'model_params', 'plot', 'metrics', 'houseage']

model_params contains a dictionary with model parameters, let’s get them (click on the tabs to switch):

# pro-tip: then typing the tag, press the "Tab" key for autocompletion!
nbs["model_params"]

{
    'bootstrap': True,
    'ccp_alpha': 0.0,
    'criterion': 'squared_error',
    'max_depth': None,
    'max_features': 1.0,
    'max_leaf_nodes': None,
    'max_samples': None,
    'min_impurity_decrease': 0.0,
    'min_samples_leaf': 1,
    'min_samples_split': 2,
    'min_weight_fraction_leaf': 0.0,
    'n_estimators': 50,
    'n_jobs': None,
    'oob_score': False,
    'random_state': None,
    'verbose': 0,
    'warm_start': False,
}

{
    'bootstrap': True,
    'ccp_alpha': 0.0,
    'criterion': 'squared_error',
    'max_depth': None,
    'max_features': 1.0,
    'max_leaf_nodes': None,
    'max_samples': None,
    'min_impurity_decrease': 0.0,
    'min_samples_leaf': 1,
    'min_samples_split': 2,
    'min_weight_fraction_leaf': 0.0,
    'n_estimators': 100,
    'n_jobs': None,
    'oob_score': False,
    'random_state': None,
    'verbose': 0,
    'warm_start': False,
}

{
    'copy_X': True,
    'fit_intercept': True,
    'n_jobs': None,
    'positive': False,
}

{
    'C': 1.0,
    'dual': True,
    'epsilon': 0.0,
    'fit_intercept': True,
    'intercept_scaling': 1.0,
    'loss': 'epsilon_insensitive',
    'max_iter': 1000,
    'random_state': None,
    'tol': 0.0001,
    'verbose': 0,
}

plot has a y_true vs y_pred chart:

nbs["plot"]

On each notebook, metrics outputs a data frame with a single row with mean absolute error (mae) and mean squared error (mse) as columns.

For single-row tables, a “Compare” tab shows all results at once:

nbs["metrics"]

	random_forest_1	random_forest_2	linear_regression	support_vector_regression
mae	0.335750	0.334612	0.529571	5.526620
mse	0.262070	0.259918	0.536969	47.566589
r2	0.803338	0.804953	0.597049	-34.694793

	mae	mse	r2
0	0.33575	0.26207	0.803338

	mae	mse	r2
0	0.334612	0.259918	0.804953

	mae	mse	r2
0	0.529571	0.536969	0.597049

	mae	mse	r2
0	5.52662	47.566589	-34.694793

We can see that the second random forest is performing the best in both metrics.

houseage contains a multi-row table where with error metrics broken down by the HouseAge indicator feature. Multi-row tables do not display the “Compare” tab:

nbs["houseage"]

	mae	mse	r2
HouseAge
1.0	1.009761	1.019616	NaN
2.0	0.598993	0.641670	0.634842
3.0	0.501940	0.363853	0.441709
4.0	0.436660	0.401188	0.616589
5.0	0.417035	0.346325	0.552242
6.0	0.443272	0.698438	0.367883
7.0	0.357205	0.273171	0.333679
8.0	0.334108	0.228172	0.822341
9.0	0.290534	0.202969	0.745687
10.0	0.358639	0.282350	0.711650
11.0	0.394470	0.337448	0.641282
12.0	0.340859	0.262916	0.702657
13.0	0.360210	0.290545	0.715474
14.0	0.301151	0.182300	0.759122
15.0	0.339142	0.255157	0.757542
16.0	0.317741	0.187201	0.766880
17.0	0.349564	0.285747	0.695871
18.0	0.323012	0.233499	0.786160
19.0	0.343581	0.237219	0.784883
20.0	0.347235	0.262215	0.776864
21.0	0.359447	0.289481	0.785371
22.0	0.372129	0.351125	0.776726
23.0	0.300444	0.196655	0.864385
24.0	0.380160	0.303163	0.711934
25.0	0.371293	0.272823	0.801622
26.0	0.320122	0.211223	0.843578
27.0	0.357114	0.262620	0.804592
28.0	0.329497	0.294314	0.835548
29.0	0.289599	0.187176	0.849010
30.0	0.317346	0.275647	0.791591
31.0	0.330459	0.262560	0.807319
32.0	0.295509	0.238956	0.839666
33.0	0.333155	0.277300	0.772775
34.0	0.301338	0.260504	0.831491
35.0	0.294385	0.211963	0.846018
36.0	0.247906	0.136633	0.871464
37.0	0.303643	0.187501	0.866432
38.0	0.296352	0.193372	0.877469
39.0	0.349294	0.392790	0.764602
40.0	0.314531	0.269793	0.847896
41.0	0.292108	0.165369	0.891573
42.0	0.286787	0.199562	0.822431
43.0	0.305273	0.208920	0.829376
44.0	0.304194	0.203960	0.846741
45.0	0.279978	0.150665	0.869406
46.0	0.329021	0.258158	0.752839
47.0	0.270375	0.149788	0.872072
48.0	0.381506	0.333945	0.819137
49.0	0.388076	0.313536	0.834210
50.0	0.380818	0.308684	0.848639
51.0	0.322959	0.144724	0.922431
52.0	0.450975	0.449177	0.758457

	mae	mse	r2
HouseAge
1.0	0.869530	0.756082	NaN
2.0	0.590529	0.659306	0.624806
3.0	0.489360	0.346051	0.469025
4.0	0.420820	0.376601	0.640086
5.0	0.399285	0.329446	0.574065
6.0	0.464970	0.745339	0.325436
7.0	0.355783	0.265904	0.351404
8.0	0.341585	0.216583	0.831364
9.0	0.299191	0.206683	0.741033
10.0	0.357131	0.274048	0.720128
11.0	0.372309	0.300059	0.681027
12.0	0.352665	0.245282	0.722599
13.0	0.342166	0.282988	0.722875
14.0	0.296604	0.170435	0.774798
15.0	0.345793	0.261148	0.751850
16.0	0.318011	0.192191	0.760667
17.0	0.342608	0.272571	0.709894
18.0	0.320770	0.224992	0.793951
19.0	0.341195	0.230527	0.790952
20.0	0.349723	0.277426	0.763921
21.0	0.364587	0.306233	0.772951
22.0	0.375933	0.351871	0.776252
23.0	0.302956	0.201751	0.860870
24.0	0.373564	0.286468	0.727797
25.0	0.372185	0.275150	0.799930
26.0	0.325505	0.222724	0.835061
27.0	0.345674	0.256267	0.809319
28.0	0.340026	0.313757	0.824684
29.0	0.284437	0.190219	0.846555
30.0	0.317126	0.275848	0.791439
31.0	0.328082	0.260363	0.808932
32.0	0.301940	0.235945	0.841686
33.0	0.329437	0.269727	0.778980
34.0	0.299616	0.257438	0.833475
35.0	0.281850	0.200212	0.854554
36.0	0.248989	0.137471	0.870676
37.0	0.304156	0.187860	0.866177
38.0	0.288820	0.179444	0.886294
39.0	0.351208	0.374575	0.775518
40.0	0.317201	0.259683	0.853596
41.0	0.279981	0.147506	0.903285
42.0	0.270931	0.182838	0.837312
43.0	0.314691	0.211704	0.827102
44.0	0.318111	0.212518	0.840310
45.0	0.269538	0.141167	0.877639
46.0	0.331738	0.281499	0.730492
47.0	0.282319	0.148419	0.873241
48.0	0.396676	0.349063	0.810949
49.0	0.387618	0.302891	0.839839
50.0	0.377749	0.312382	0.846825
51.0	0.283568	0.127094	0.931880
52.0	0.454897	0.459108	0.753116

	mae	mse	r2
HouseAge
1.0	0.077045	0.005936	NaN
2.0	0.621786	0.704731	0.598956
3.0	0.394869	0.309883	0.524520
4.0	0.502120	0.526965	0.496384
5.0	0.402867	0.349914	0.547602
6.0	0.535902	0.955226	0.135479
7.0	0.471769	0.422926	-0.031604
8.0	0.435266	0.300705	0.765865
9.0	0.395658	0.334463	0.580930
10.0	0.548596	0.445740	0.544787
11.0	0.497246	0.449512	0.522155
12.0	0.436461	0.347855	0.606596
13.0	0.432717	0.332560	0.674330
14.0	0.412923	0.298274	0.605881
15.0	0.454044	0.359019	0.658850
16.0	0.451385	0.339579	0.577125
17.0	0.450863	0.388885	0.586097
18.0	0.422341	0.361656	0.668794
19.0	0.440418	0.385330	0.650572
20.0	0.480448	0.449165	0.617777
21.0	0.481173	0.437860	0.675358
22.0	0.557506	0.622022	0.604467
23.0	0.437108	0.361490	0.750713
24.0	0.480626	0.422856	0.598201
25.0	0.551525	0.541321	0.606389
26.0	0.507414	0.484308	0.641343
27.0	0.522484	0.532184	0.604017
28.0	0.574381	0.744957	0.583745
29.0	0.484700	0.541898	0.562865
30.0	0.538348	0.534287	0.596041
31.0	0.538239	0.521302	0.617441
32.0	0.515134	0.528955	0.645083
33.0	0.536822	0.510495	0.581690
34.0	0.608867	1.018926	0.340902
35.0	0.571631	0.534728	0.611543
36.0	0.503451	0.421068	0.603886
37.0	0.537139	0.475265	0.661442
38.0	0.567287	0.519219	0.670993
39.0	0.601552	0.741179	0.555814
40.0	0.610118	0.653960	0.631311
41.0	0.581788	0.574146	0.623550
42.0	0.503874	0.438613	0.609726
43.0	0.565345	0.495386	0.595420
44.0	0.553312	0.508499	0.617904
45.0	0.547449	0.493563	0.572186
46.0	0.613946	0.646647	0.380899
47.0	0.580874	0.482213	0.588160
48.0	0.765080	0.945197	0.488086
49.0	0.753427	0.834546	0.558714
50.0	0.593333	0.654756	0.678945
51.0	0.641995	0.661623	0.645383
52.0	0.736178	1.016155	0.453567

	mae	mse	r2
HouseAge
1.0	0.930033	0.864961	NaN
2.0	9.322419	168.378674	-94.819857
3.0	11.945956	305.227908	-467.336763
4.0	12.345843	297.444139	-283.264578
5.0	9.261336	154.188025	-198.346917
6.0	7.841133	81.646329	-72.893468
7.0	8.956448	122.344421	-297.423520
8.0	7.812335	91.259590	-70.056500
9.0	6.924770	70.481508	-87.310778
10.0	6.131887	60.508519	-60.794385
11.0	7.082020	71.660333	-75.177237
12.0	7.960372	108.852453	-122.106053
13.0	6.580463	54.817169	-52.681465
14.0	6.517646	64.191350	-83.818100
15.0	6.523527	62.058316	-57.969510
16.0	5.845502	50.085833	-61.371436
17.0	6.562966	65.651099	-68.874462
18.0	6.711008	63.053593	-56.744841
19.0	6.208104	55.306065	-49.153140
20.0	5.724718	50.765151	-42.199321
21.0	6.841046	65.565442	-47.612064
22.0	6.580509	66.400794	-41.223098
23.0	5.875658	47.119800	-31.494261
24.0	6.334337	56.804336	-52.975669
25.0	6.178044	55.552114	-39.393588
26.0	5.919405	49.557399	-35.699951
27.0	5.139975	33.694358	-24.071032
28.0	5.253580	38.707437	-20.628339
29.0	6.027190	57.472790	-45.361771
30.0	5.477935	37.556648	-27.395475
31.0	5.487792	38.149921	-26.996454
32.0	5.226146	34.626894	-22.233881
33.0	5.267401	35.107639	-27.767952
34.0	4.923614	31.292451	-19.241693
35.0	4.656281	27.499228	-18.977042
36.0	4.718963	28.595674	-25.901050
37.0	4.037945	20.693349	-13.741062
38.0	4.628752	27.261603	-16.274497
39.0	4.297735	23.923050	-13.337012
40.0	4.282328	24.824898	-12.995744
41.0	4.115536	21.197708	-12.898670
42.0	4.347792	24.072539	-20.419554
43.0	4.292008	22.977190	-17.765400
44.0	3.888998	18.914428	-13.212658
45.0	4.046095	20.273032	-16.572425
46.0	4.076637	22.940559	-20.963348
47.0	4.357417	23.796387	-19.323577
48.0	3.591272	15.788608	-7.551028
49.0	3.423189	14.383290	-6.605501
50.0	3.356563	13.936890	-5.833867
51.0	3.473428	15.150976	-7.120624
52.0	3.239542	14.041574	-6.550805

If we only compare two notebooks, the output is a bit different:

# only compare two notebooks
nbs_two = NotebookCollection(paths=files[:2], ids=ids[:2], scores=["r2"])

Comparing single-row tables includes a diff column with the error difference between experiments. Error reductions are showed in green, increments in red:

nbs_two["metrics"]

	random_forest_1	random_forest_2	diff	diff_relative	ratio
mae	0.335750	0.334612	-0.001138	-0.34%	0.996611
mse	0.262070	0.259918	-0.002152	-0.83%	0.991788
r2	0.803338	0.804953	0.001615	0.20%	1.002010

	mae	mse	r2
0	0.33575	0.26207	0.803338

	mae	mse	r2
0	0.334612	0.259918	0.804953

When comparing multi-row tables, the “Compare” tab appears, showing the difference between the tables:

nbs_two["houseage"]

	mae	mse	r2
HouseAge
1.000000	-0.140231	-0.263534	nan
2.000000	-0.008464	0.017636	-0.010036
3.000000	-0.012580	-0.017802	0.027316
4.000000	-0.015840	-0.024587	0.023497
5.000000	-0.017750	-0.016879	0.021823
6.000000	0.021698	0.046901	-0.042447
7.000000	-0.001422	-0.007267	0.017725
8.000000	0.007477	-0.011589	0.009023
9.000000	0.008657	0.003714	-0.004654
10.000000	-0.001508	-0.008302	0.008478
11.000000	-0.022161	-0.037389	0.039745
12.000000	0.011806	-0.017634	0.019942
13.000000	-0.018044	-0.007557	0.007401
14.000000	-0.004547	-0.011865	0.015676
15.000000	0.006651	0.005991	-0.005692
16.000000	0.000270	0.004990	-0.006213
17.000000	-0.006956	-0.013176	0.014023
18.000000	-0.002242	-0.008507	0.007791
19.000000	-0.002386	-0.006692	0.006069
20.000000	0.002488	0.015211	-0.012943
21.000000	0.005140	0.016752	-0.012420
22.000000	0.003804	0.000746	-0.000474
23.000000	0.002512	0.005096	-0.003515
24.000000	-0.006596	-0.016695	0.015863
25.000000	0.000892	0.002327	-0.001692
26.000000	0.005383	0.011501	-0.008517
27.000000	-0.011440	-0.006353	0.004727
28.000000	0.010529	0.019443	-0.010864
29.000000	-0.005162	0.003043	-0.002455
30.000000	-0.000220	0.000201	-0.000152
31.000000	-0.002377	-0.002197	0.001613
32.000000	0.006431	-0.003011	0.002020
33.000000	-0.003718	-0.007573	0.006205
34.000000	-0.001722	-0.003066	0.001984
35.000000	-0.012535	-0.011751	0.008536
36.000000	0.001083	0.000838	-0.000788
37.000000	0.000513	0.000359	-0.000255
38.000000	-0.007532	-0.013928	0.008825
39.000000	0.001914	-0.018215	0.010916
40.000000	0.002670	-0.010110	0.005700
41.000000	-0.012127	-0.017863	0.011712
42.000000	-0.015856	-0.016724	0.014881
43.000000	0.009418	0.002784	-0.002274
44.000000	0.013917	0.008558	-0.006431
45.000000	-0.010440	-0.009498	0.008233
46.000000	0.002717	0.023341	-0.022347
47.000000	0.011944	-0.001369	0.001169
48.000000	0.015170	0.015118	-0.008188
49.000000	-0.000458	-0.010645	0.005629
50.000000	-0.003069	0.003698	-0.001814
51.000000	-0.039391	-0.017630	0.009449
52.000000	0.003922	0.009931	-0.005341

	mae	mse	r2
HouseAge
1.0	1.009761	1.019616	NaN
2.0	0.598993	0.641670	0.634842
3.0	0.501940	0.363853	0.441709
4.0	0.436660	0.401188	0.616589
5.0	0.417035	0.346325	0.552242
6.0	0.443272	0.698438	0.367883
7.0	0.357205	0.273171	0.333679
8.0	0.334108	0.228172	0.822341
9.0	0.290534	0.202969	0.745687
10.0	0.358639	0.282350	0.711650
11.0	0.394470	0.337448	0.641282
12.0	0.340859	0.262916	0.702657
13.0	0.360210	0.290545	0.715474
14.0	0.301151	0.182300	0.759122
15.0	0.339142	0.255157	0.757542
16.0	0.317741	0.187201	0.766880
17.0	0.349564	0.285747	0.695871
18.0	0.323012	0.233499	0.786160
19.0	0.343581	0.237219	0.784883
20.0	0.347235	0.262215	0.776864
21.0	0.359447	0.289481	0.785371
22.0	0.372129	0.351125	0.776726
23.0	0.300444	0.196655	0.864385
24.0	0.380160	0.303163	0.711934
25.0	0.371293	0.272823	0.801622
26.0	0.320122	0.211223	0.843578
27.0	0.357114	0.262620	0.804592
28.0	0.329497	0.294314	0.835548
29.0	0.289599	0.187176	0.849010
30.0	0.317346	0.275647	0.791591
31.0	0.330459	0.262560	0.807319
32.0	0.295509	0.238956	0.839666
33.0	0.333155	0.277300	0.772775
34.0	0.301338	0.260504	0.831491
35.0	0.294385	0.211963	0.846018
36.0	0.247906	0.136633	0.871464
37.0	0.303643	0.187501	0.866432
38.0	0.296352	0.193372	0.877469
39.0	0.349294	0.392790	0.764602
40.0	0.314531	0.269793	0.847896
41.0	0.292108	0.165369	0.891573
42.0	0.286787	0.199562	0.822431
43.0	0.305273	0.208920	0.829376
44.0	0.304194	0.203960	0.846741
45.0	0.279978	0.150665	0.869406
46.0	0.329021	0.258158	0.752839
47.0	0.270375	0.149788	0.872072
48.0	0.381506	0.333945	0.819137
49.0	0.388076	0.313536	0.834210
50.0	0.380818	0.308684	0.848639
51.0	0.322959	0.144724	0.922431
52.0	0.450975	0.449177	0.758457

	mae	mse	r2
HouseAge
1.0	0.869530	0.756082	NaN
2.0	0.590529	0.659306	0.624806
3.0	0.489360	0.346051	0.469025
4.0	0.420820	0.376601	0.640086
5.0	0.399285	0.329446	0.574065
6.0	0.464970	0.745339	0.325436
7.0	0.355783	0.265904	0.351404
8.0	0.341585	0.216583	0.831364
9.0	0.299191	0.206683	0.741033
10.0	0.357131	0.274048	0.720128
11.0	0.372309	0.300059	0.681027
12.0	0.352665	0.245282	0.722599
13.0	0.342166	0.282988	0.722875
14.0	0.296604	0.170435	0.774798
15.0	0.345793	0.261148	0.751850
16.0	0.318011	0.192191	0.760667
17.0	0.342608	0.272571	0.709894
18.0	0.320770	0.224992	0.793951
19.0	0.341195	0.230527	0.790952
20.0	0.349723	0.277426	0.763921
21.0	0.364587	0.306233	0.772951
22.0	0.375933	0.351871	0.776252
23.0	0.302956	0.201751	0.860870
24.0	0.373564	0.286468	0.727797
25.0	0.372185	0.275150	0.799930
26.0	0.325505	0.222724	0.835061
27.0	0.345674	0.256267	0.809319
28.0	0.340026	0.313757	0.824684
29.0	0.284437	0.190219	0.846555
30.0	0.317126	0.275848	0.791439
31.0	0.328082	0.260363	0.808932
32.0	0.301940	0.235945	0.841686
33.0	0.329437	0.269727	0.778980
34.0	0.299616	0.257438	0.833475
35.0	0.281850	0.200212	0.854554
36.0	0.248989	0.137471	0.870676
37.0	0.304156	0.187860	0.866177
38.0	0.288820	0.179444	0.886294
39.0	0.351208	0.374575	0.775518
40.0	0.317201	0.259683	0.853596
41.0	0.279981	0.147506	0.903285
42.0	0.270931	0.182838	0.837312
43.0	0.314691	0.211704	0.827102
44.0	0.318111	0.212518	0.840310
45.0	0.269538	0.141167	0.877639
46.0	0.331738	0.281499	0.730492
47.0	0.282319	0.148419	0.873241
48.0	0.396676	0.349063	0.810949
49.0	0.387618	0.302891	0.839839
50.0	0.377749	0.312382	0.846825
51.0	0.283568	0.127094	0.931880
52.0	0.454897	0.459108	0.753116

When displaying dictionaries, a “Compare” tab shows with a diff view:

nbs_two["model_params"]

Legends

Colors
Added
Changed
Deleted

Links
(f)irst change
(n)ext change
(t)op

{
    'bootstrap': True,
    'ccp_alpha': 0.0,
    'criterion': 'squared_error',
    'max_depth': None,
    'max_features': 1.0,
    'max_leaf_nodes': None,
    'max_samples': None,
    'min_impurity_decrease': 0.0,
    'min_samples_leaf': 1,
    'min_samples_split': 2,
    'min_weight_fraction_leaf': 0.0,
    'n_estimators': 50,
    'n_jobs': None,
    'oob_score': False,
    'random_state': None,
    'verbose': 0,
    'warm_start': False,
}

{
    'bootstrap': True,
    'ccp_alpha': 0.0,
    'criterion': 'squared_error',
    'max_depth': None,
    'max_features': 1.0,
    'max_leaf_nodes': None,
    'max_samples': None,
    'min_impurity_decrease': 0.0,
    'min_samples_leaf': 1,
    'min_samples_split': 2,
    'min_weight_fraction_leaf': 0.0,
    'n_estimators': 100,
    'n_jobs': None,
    'oob_score': False,
    'random_state': None,
    'verbose': 0,
    'warm_start': False,
}

Lists (and sets) are compared based on elements existence:

nbs_two["feature_names"]

Both	Only in random_forest_1	Only in random_forest_2
AveBedrms
AveOccup
AveRooms
HouseAge
Latitude
Longitude
MedInc
Population

['MedInc', 'HouseAge', 'AveRooms', 'AveBedrms', 'Population', 'AveOccup', 'Latitude', 'Longitude']

Using the mapping interface#

NotebookCollection has a dict-like interface, you can retrieve data from individual notebooks:

nbs["model_params"]["random_forest_1"]

{'bootstrap': True,
 'ccp_alpha': 0.0,
 'criterion': 'squared_error',
 'max_depth': None,
 'max_features': 1.0,
 'max_leaf_nodes': None,
 'max_samples': None,
 'min_impurity_decrease': 0.0,
 'min_samples_leaf': 1,
 'min_samples_split': 2,
 'min_weight_fraction_leaf': 0.0,
 'n_estimators': 50,
 'n_jobs': None,
 'oob_score': False,
 'random_state': None,
 'verbose': 0,
 'warm_start': False}

nbs["plot"]["random_forest_2"]

f	1	{	f	1	{
	2	'bootstrap': True,		2	'bootstrap': True,
	3	'ccp_alpha': 0.0,		3	'ccp_alpha': 0.0,
	4	'criterion': 'squared_error',		4	'criterion': 'squared_error',
	5	'max_depth': None,		5	'max_depth': None,
	6	'max_features': 1.0,		6	'max_features': 1.0,
	7	'max_leaf_nodes': None,		7	'max_leaf_nodes': None,
	8	'max_samples': None,		8	'max_samples': None,
	9	'min_impurity_decrease': 0.0,		9	'min_impurity_decrease': 0.0,
	10	'min_samples_leaf': 1,		10	'min_samples_leaf': 1,
	11	'min_samples_split': 2,		11	'min_samples_split': 2,
	12	'min_weight_fraction_leaf': 0.0,		12	'min_weight_fraction_leaf': 0.0,
t	13	'n_estimators': 50,	t	13	'n_estimators': 100,
	14	'n_jobs': None,		14	'n_jobs': None,
	15	'oob_score': False,		15	'oob_score': False,
	16	'random_state': None,		16	'random_state': None,
	17	'verbose': 0,		17	'verbose': 0,
	18	'warm_start': False,		18	'warm_start': False,
	19	}		19	}

sklearn-evaluation

Notebooks as experiments

Contents

Notebooks as experiments#

Using the mapping interface#