7.3 Creating Evaluations

The process to create an evaluation is different if you want to create a single evaluation or a cross-validation evaluation:

• To create a single evaluation, you need two resources: a testing dataset (different from the one used for training) and a model, an ensemble, a logistic regression, deepnet, or fusion. All three processes follow a similar logic. You can find a separate explanation of each one in the following subsections.

• To create a cross-validation evaluation, you just need a dataset. BigML allows you to create cross-validation for models, ensembles, logistic regressions, deepnets and fusions. This process is explained in subsection 7.3.6

7.3.1 Model Evaluations

To evaluate a model, you can use any of the following options from the BigML Dashboard:

• Click Evaluate a model in the 1-click action menu from the evaluation list view. (Figure 7.31 .)

  

  This option redirects you to the New Evaluation view where you need to select a model and a testing dataset. (See Figure 7.32 .) From this view you can also select an ensemble by clicking the ensemble icon above the model selector.

  

• Click Evaluate in the 1-click action menu from the model view. (See Figure 7.33 .)

  

  Alternatively, click Evaluate in the pop up menu from the model list view (see Figure 7.34 ).

  

  By using any of these options, you will be redirected to the New Evaluation view where the model will be pre-filled in the selector and you only have to choose the testing dataset. If you previously split your original dataset into two subsets (one for training and another for testing) using the 1-click menu option from your dataset view, BigML will automatically select the corresponding testing dataset. Finally, click the Evaluate green button to perform the evaluation. (See Figure 7.35 .)

  

7.3.2 Ensemble Evaluations

To evaluate an ensemble you can use the following options from the BigML Dashboard:

• Click Evaluate an ensemble in the 1-click action menu from the evaluation list view (Figure 7.36 ).

  

  This option takes you to the New Evaluation view where you need to select an ensemble and a testing dataset. (See Figure 7.37 .) From this view, you can also select a model by clicking the model icon above the ensemble selector.

  

• Click Evaluate in the 1-click action menu from the ensemble view- (Figure 7.38 .)

  

  Alternatively, click Evaluate in the pop up menu from the ensembles list view (Figure 7.39 ).

  

  By using any of these two options, you will be redirected to the New Evaluation view where the ensemble will be pre-filled and you only have to choose the testing dataset. If you previously split your original dataset into two subsets (one for training and another for testing) using the 1-click menu option from your dataset view, BigML will automatically select the corresponding testing dataset. Finally, click the Evaluate green button to perform the evaluation. (See Figure 7.40 .)

  

7.3.3 Logistic Regression Evaluations

To evaluate a logistic regression, you can use these options from the BigML Dashboard:

• Click Evaluate a logistic regression from the 1-click action menu from the evaluation list view. (See Figure 7.41 .)

  

  This option redirects you to the New Evaluation view where you need to select a logistic regression and a testing dataset. (See Figure 7.42 .)

  

• Click Evaluate from the logistic regression 1-click action menu. (Figure 7.43 .)

  

  Alternatively, click Evaluate in the pop up menu from the logistic regression list view. (Figure 7.44 .)

  

  By using any of these options, you will be redirected to the New Evaluation view where the logistic regression will be pre-filled in the selector and you only need to choose the testing dataset. If you previously split your original dataset into two subsets (one for training and another for testing) using the 1-click menu option from your dataset view, BigML will automatically select the corresponding testing dataset. Finally, click the Evaluate green button to perform the evaluation. (See Figure 7.45 .)

  

7.3.4 Deepnet Evaluations

To evaluate a deepnet, you can use these options from the BigML Dashboard:

• Click Evaluate a deepnet from the 1-click action menu from the evaluation list view. (See Figure 7.46 .)

  

  This option redirects you to the New Evaluation view where you need to select a deepnet and a testing dataset. (See Figure 7.47 .)

  

• Click Evaluate from the deepnet 1-click action menu. (See Figure 7.48 .)

  

  Alternatively, click Evaluate in the pop up menu from the deepnet list view. (Figure 7.49 .)

  

  By using any of these options, you will be redirected to the New Evaluation view where the deepnet will be pre-filled in the selector and you only need to choose the testing dataset. If you previously split your original dataset into two subsets (one for training and another for testing) using the 1-click menu option from your dataset view, BigML will automatically select the corresponding testing dataset. Finally, click the Evaluate green button to perform the evaluation. (See Figure 7.50 .)

  

7.3.5 Fusion Evaluations

To evaluate a fusion, you can use these options from the BigML Dashboard:

• Click Evaluate a fusion from the 1-click action menu from the evaluation list view. (See Figure 7.51 .)

  

  This option redirects you to the New Evaluation view where you need to select a fusion and a testing dataset. (See Figure 7.52 .)

  

• Click Evaluate from the fusion 1-click action menu. (See Figure 7.53 .)

  

  Alternatively, click Evaluate in the pop up menu from the fusion list view. (Figure 7.54 .)

  

  By using any of these options, you will be redirected to the New Evaluation view where the fusion will be pre-filled in the selector and you only need to choose the testing dataset. Finally, click the Evaluate green button to perform the evaluation. (See Figure 7.55 .)

  

7.3.6 Cross-Validation Evaluations

In BigML, you can use k-fold cross-validation to evaluate your models, ensembles, logistic regressions, and deepnets. Cross-validation evaluations are implementing in BigML as a WhizzML script and they can be found in BigML Gallery:

1. Go to the scripts Gallery where you will find five different scripts to perform cross-validation:

   • Basic 5-fold cross-validation: performs a 5-fold cross-validation for models with default model configuration options. Learn the default options for models in Model’s k-fold cross-validation . (See Figure 7.56 .)

     

   • Model’s k-fold cross-validation: performs cross-validation for models. You can configure the \(k\)-fold parameter and the model inputs. Learn about the configurable inputs for models in Model’s k-fold cross-validation . (See Figure 7.57 .)

     

   • Ensemble’s k-fold cross-validation: performs cross-validation for ensembles. You can configure the \(k\)-fold parameter and the ensemble inputs. Learn about the configurable inputs for ensembles in Ensemble’s k-fold cross-validation . (See Figure 7.58 .)

     

   • Logistic regresion’s k-fold cross-validation: performs cross-validation for logistic regression. You can configure the \(k\)-fold parameter and the logistic inputs. Learn about the configurable inputs for logistic regressions in section 4.4 . (See Figure 7.59 .)

     

   • Deepnet’s k-fold cross-validation: performs cross-validation for deepnets. You can configure the \(k\)-fold parameter and the deepnet inputs. Learn about the configurable inputs for deepnets in section 5.4 . (See Figure 7.60 .)

     

2. By clicking the script preview you can inspect script’s details such as the source-code, the script input and the output. (See Figure 7.57 .) You can find additional documentation about WhizzML scripts here.

   

3. Clone your preferred script for FREE. You can clone it from the script preview by clicking the FREE button. (See Figure 7.62 .)

   

   Alternatively, you can clone it from the script view by clicking the FREE or Buy buttons. (See Figure 7.63 .)

   

   A modal window will appear asking you for confirmation. (See Figure 7.64 .)

   

4. Once you clone the script, you will be redirected to the Execution view to set your inputs. You need to select a dataset and optionally, you can configure the rest of the inputs. If you do not configure them, they will take the default values. You can find an explanation of all your inputs in subsection 7.4.5 . (See Figure 7.65 .)

   

5. Once you have selected the dataset, click Execute . (See Figure 7.66 .)

   

6. Once you execute the script, you can check the progress of your script in the execution view where you will find the elapsed time, the total resources generated and the script log messages. (See Figure 7.67 .)

   

7. Finally, cross-validation yields \(k\) different models and \(k\) different evaluations. The results of the single evaluations are averaged to obtain the final model performance measures. Access the final cross-validation evaluation containing the averaged measures by clicking on the evaluation ID link in the Outputs section. Read more about cross-validation measures in subsection 7.2.3 . The \(k\) intermediary resources can be found in the same view under the Resources panel. (See Figure 7.68 .)

   

You can perform again any new cross-validation by clicking on your cloned script listed under the scripts tab in the BigML Dashboard. (See Figure 7.69 .)