5.6 Deepnet Predictions

5.6.1 Introduction

The ultimate goal in building a deepnet is being able to make predictions with it. In BigML, you can make predictions for single instances or for many instances in batch. Each prediction comes with a measure indicating the prediction confidence. For Regression problems, the expected error is provided along with the predicted value, while for Classification problems the vector of probabilities per class is returned (a percentage ranging from 0% up to 100%).

The predictions tab in the main menu of the BigML Dashboard is where all your saved predictions are listed. (See Figure 5.50 .) You can search your predictions by name clicking on the search option on the top menu. In the predictions list view, you can see, for each prediction, the deepnet icon used for the prediction, the Name of the prediction, the Objective (objective field name), the Prediction (the prediction result), and the Age (time since the prediction was created).

When you first create an account at BigML, or every time that you start a new Project, your list of predictions will be empty. (See Figure 5.51 .)

Deepnet predictions are saved under the Classification & Regression option in the menu (see Figure 5.52 .)

Select the list for your single instances predictions or your batch predictions by clicking on the corresponding icons. (See Figure 5.53 and Figure 5.54 .)

5.6.2 Creating Deepnet Predictions

BigML provides two options to predict with your deepnets explained in the following subsections:

• predict: to predict one single instance

• batch prediction to predict multiple instances in batch.

Predict

BigML allows you to quickly make predictions for single instances by providing a form containing the input fields used by the deepnet, so you can easily set the values and get an immediate response.

Follow these steps to create a single prediction:

1. Click predict in the deepnet 1-click action menu. (See Figure 5.55 .)

   

   Alternatively, click predict in the pop up menu in the list view. (See Figure 5.56 .)

   

2. You will be redirected to the prediction form where you will find all the fields used by the deepnet as input fields. (See Figure 5.57 .)

   

3. Select the fields to be used for your prediction, set the input values for your selected fields and click Predict (Figure 5.58 .) Non-selected fields will be considered as missing values during the prediction.

   

4. Get the prediction at the top of the view along with the rest of class probabilities. (See Figure 5.59 .) BigML predictions are synchronous, i.e., when you send the input data, you get an immediate response.

   

5. Hide or display the histogram view containing the rest of your class probabilities by clicking on the icon highlighted in Figure 5.60 . You can download all the probabilities in PNG format, in CSV or JSON file by clicking on the corresponding icons. (See Single Predictions .)

   

6. Your prediction is automatically saved and you can find it in the predictions list view.

For regression deepnets, the process is the same, but instead of the predicted classes and the probabilities you get a numeric value for the objective field along with an expected error as the certainty measure.

Note: this option is only available from the BigML Dashboard for deepnets with less than 100 fields. If you want to perform single instance predictions for a higher number of fields, use the BigML API.

Deepnet Prediction with Images

Deepnet created from a dataset containing images is a convolutionary neural network (CNN) (See subsection 5.2.1 ). Because BigML treats images as an input type just like any other data types such as numeric and text, creating predictions using CNN is the same as other deepnets. Everything stated in the previous section still applies. The only thing different is input fields of images.

Follow these steps to create a single prediction:

1. Click predict in the deepnet 1-click action menu. See Figure 5.61 .

   

   Alternatively, click predict in the pop-up menu in the deepnet list view. See Figure 5.62 .

   

2. You will be directed to the prediction form where you will find all the input fields used by the deepnet. One of them is the image field. As seen in Figure 5.63 , click on the input field box to select an image. Because this is a single prediction, an image is input by using a single image source. Clicking on the input box, single image sources available will be in the dropdown list. You can also use the search box to locate specific ones.

   

3. Oftentimes single image sources were used for creating a composite source, they become component sources of the composite source. Or an image was uploaded as a part of an archive file (zip/tar) which created a composite source. In those cases, the composite source will be shown in the dropdown list, along with an icon “List components”. In the example in Figure 5.64 , predict-images.zip is a composite source, click on the icon to show its component sources.

   

   After the component sources of the composite are listed, scroll the dropdown list to find the desired one, then click to select it, as shown in Figure 5.65 .

   

4. Once an image is selected, click on the green button Predict to create a prediction for the image.

   

   As shown in Figure 5.66 , after the new prediction is created, the predicted class is at the top of the form along with its probability.

   Below the prediction is a histogram representing all the classes of the objective field with their respective predicted probabilities. Show or hide this histogram by clicking on the icon on the right of the predicted class (Figure 5.66 ). Up to seven different classes can be shown in the histogram together. When there are more than seven classes, the view can be scrolled by clicking on the arrow icons on the right. See Figure 5.67 . If desired, click on the sort icon on the left to sort the classes in the histogram by probability.

   

   The histogram can be exported in PNG format, in a CSV file, or in a JSON file by clicking on the corresponding icons on the top right.

5. In addition to images, deepnets may use other fields, which will be in the prediction form too. As shown in Figure 5.68 , all the fields can be selected, and their input values be set by dragging the knobs on the sliders or by entering precise values in their input boxes.

   

   Non-selected fields will be considered as having missing values during the prediction.

Batch Predictions

BigML batch predictions allow you to make simultaneous predictions for multiple instances. All you need is the deepnet you want to use to make predictions and a dataset containing the instances you want to predict. BigML will create a prediction for each instance in the dataset.

Follow these steps to create a batch prediction:

1. Click on batch prediction option under the deepnet 1-click action menu (Figure 5.69 )

   

   Alternatively, click on create batch prediction in the pop up menu of the list view (Figure 5.70 ).

   

2. Select the dataset containing all the instances you want to predict. The instances should contain the input values for the fields used by the deepnet as input fields. From this view you can also select another deepnet from the selector. (See Figure 5.71 .)

   

3. After the deepnet and the dataset are selected, the batch prediction configuration options will appear along with a preview of the prediction output (a CSV file). (See Figure 5.72 .) The default output format includes all your prediction dataset fields and adds an extra column with the class predicted. See subsection 5.6.3 for a detailed explanation of all configuration options.

   

4. By default, BigML generates an output Dataset with your batch predictions that you can later find in your datasets section in the BigML Dashboard. This option is active by default but you can deactivate it by clicking on the icon shown in Figure 5.73 .

   

5. After you configure your batch prediction, click on the green button Predict to generate your batch prediction. (See Figure 5.74 .)

   

6. When the batch prediction is created, you will be able to download the CSV file containing all your dataset instances along with a prediction for each one of them. (See Figure 5.75 .)

   

7. If you did not disable the option to create a dataset explained in step 4, you will also be able to access the output dataset from the batch prediction view. (See Figure 5.76 .)

   

Batch Predictions with Images

Deepnet created from a dataset containing images is a convolutionary neural network (CNN) (See subsection 5.2.1 ). Because BigML treats images as an input type just like any other data types such as numeric and text, creating batch predictions using CNN is the same as other deepnets.

The input of a batch prediction is a dataset. The dataset contains mutiple images which was created by a composite source.

As shown in Figure 5.77 , the input for the deepnet batch prediction is selected as predict-images, which is a dataset consisting of six images.

Everything stated earlier in current section (Batch Predictions ) applies.

5.6.3 Configuring Deepnet Predictions

BigML provides several options to configure your batch predictions such as setting a probability threshold Probability threshold ), default values for your missing numeric values (see Default Numeric Value ), fields mapping (see Fields Mapping ), and output file settings (see Output Settings .)

Probability threshold

Probability thresholds usually makes sense when you want to minimize false positives at the cost of false negatives. The positive class will be predicted if its probability is greater than the given threshold, otherwise the following class with greater probability will be predicted instead. This option is only available for classification deepnets.

To configure a threshold for your batch prediction follow these steps:

1. Select the positive class, i.e., the class for which you want to apply the threshold (Figure 5.78 ).

   

2. Set a probability threshold using the slider shown in Figure 5.79 .

   

3. Click Predict . If the positive class probability is greater than the given threshold, it will be predicted, otherwise the following class with greater probability will be predicted instead.

   

You can also find the same options to set a threshold for batch predictions under the configure panel (see Figure 5.81 ).

Default Numeric Value

If the dataset used to make the batch prediction contains instances with missing values for the numeric fields, the prediction will not be computed for them, unless you built the deepnet enabling the Missing numerics parameter (see subsection 5.4.4 ).

By using the Default numeric value before creating your batch prediction, you can easily replace all the missing numeric values by the field’s Mean, Median, Maximum, Minimum or by Zero. (See Figure 5.82 .)

Fields Mapping

You can specify which input fields of the deepnet match with the fields in the dataset contaning the instances you want to predict. BigML automatically matches fields by name, but you can also set an automatic match by field ID by clicking on the green switcher. Additionally, you can manually search for fields or remove them from the Dataset fields column if you do not want them to be considered during the batch prediction. (See Figure 5.83 .)

Note: Fields mapping from the BigML Dashboard is limited to 200 fields. For batch predictions with a higher number of fields, map your fields using the BigML API.

Output Settings

Batch predictions return a CSV file containing all your instances and the final predictions. Tune the following settings to customize your prediction file (see Figure 5.84 ):

• Separator: this option allows you to choose the best separator for your output file columns. The default separator is the comma. You can also select the semicolon, the tab, or the space.

• New line: this option allows you to set the new line character to use as the line break in the generated csv file: “LF”, “CRLF”.

• Output fields: by clicking on the list icon next to the separator selector, you can include or exclude all your dataset fields from your output file. You can also individually select the fields you want to include or exclude using the multiple output fields selector. Note: a maximum of 100 fields can be displayed in this selector, but all your dataset fields will be included in the output file by default unless you exclude them.

• Headers: this option includes or excludes a first row in the output file (and in the output dataset) with the names of each column (input field names, prediction column name, probability column name, etc.). By default, BigML includes the headers.

• Prediction column name: customize the name for your predictions column. By default, BigML takes the name of the deepnet’s objective field.

• Probability: this option allows you to include an additional column with the probability for the predicted class. By default it is not included in your ouput file. For regression deepnets, you will find the expected error instead of the probability.

• Probability column name: customize the name for the probability column if you include it in the output file. BigML sets “probability” as the default name. For regression deepnets, you will find the expected error column name.

• All class probabilities: this includes all the probabilities of the objective field classes per instance. This option will add \(n\) extra columns, one by class in the objective field. This option does not exist for regression deepnets.

5.6.4 Visualizing Deepnet Predictions

Deepnet predictions visualization changes depending if you are predicting one single instance (Single Predictions ), or you are predicting multiple instances using the batch predictions option (Batch Prediction ).

Single Predictions

For single predictions, find the predicted class given the input fields values at the top of the form along with its probability. (See Figure 5.85 .)

Below the prediction, there’s a histogram representing the rest of the objective field class probabilities. All the class probabilities must sum 100%. Show or hide this view by clicking on the icon highlighted in Figure 5.86 . You can see up to seven different classes at the same time; if you have more than seven classes, you can see the others by clicking on the arrows icons. Export this view in PNG format, in a CSV file, or in a JSON file by clicking on the corresponding icons. (See Figure 5.86 .)

For regression deepnets, instead of the class probabilities you will get the predicted numeric value for the objective field.

Prediction explanation

Prediction explanation helps understand why a deepnet makes a certain prediction. This is very useful in many applications, and the reasons behind a deepnet’s prediction are often as important as the prediction itself.

BigML prediction explanation is based on Shapley values. For more information, please refer to this research paper: A Unified Approach to Interpreting Model Predictions [ 25 ] .

For any classification or regression deepnet, you can request the explanation for the prediction by clicking the prediction explanation icon and then click Predict (see Figure 5.87 ).

The prediction explanation represents the most important factors considered by the deepnet in a prediction given the input values. Each input value will yield an associated importance, as you can see Figure 5.88 . The importances across all input fields should sum 100%.

For some input fields you will see a “+” icon next to the importance. This is because the importance may not be directly associated with the input value, i.e., it can be explained by other reasons. In the Figure 5.89 below, the importance of 36.58% for the field “Rating” is not explained by this field being equal to 90.00. Rather, it is because this field value is not missing.

The prediction explanation for deepnets is calculated using the results of over a thousand distinct predictions using random perturbations of the input data. For this reason, the calculation of the explanation may take some time to be computed.

Note: the input field importances in the prediction explanation are different from the overall field importances of the deepnet. A field can be very important for the deepnet but insignificant for a given prediction.

Batch Prediction

After creating your batch prediction, you get a CSV file and, optionally, an output dataset. Both outputs are explained in the following subsections.

Output CSV file

The batch prediction generates a CSV file containing your predictions for each of your dataset instances in the last column. (See Figure 5.90 .)

You can configure several options to customize your CSV file. You can find a detailed explanation of those options in Output Settings .

See an output CSV file example in Figure 5.91 . The column class in this example contains the final prediction (it is named by default as your deepnet’s Objective Field). In this case we are predicting whether a person is a good or a bad candidate for holding a credit. This file has been configured to contain also the probability for each prediction.

duration,age,amount,purpose,class,probability
24,26,5433,used car,good,0.88785
36,42,8086,new car,bad,0.55526
24,28,1376,radio/tv,good,0.8385
48,31,6758,radio/tv,bad,0.73576
26,30,7966,used car,good,0.7201
12,42,2577,furniture/equipment,good,0.67644
36,30,4455,business,good,0.52227
18,32,1442,new car,bad,0.75488
9,22,276,new car,good,0.57819

Output Dataset

By default BigML automatically creates a dataset out of your batch prediction. You can disable this option by configuring your batch prediction. (See Output Settings .) You will find the output dataset in your batch prediction view as shown in Figure 5.92 .

In the output dataset, you can find an additional field (named by default as your deepnet’s objective field) containing the class predicted for each one of your instances. If you configured your batch prediction to include the prediction probabilities and all class probabilites you will be able to find them in the last fields of your output dataset. (See Figure 5.93 .)

5.6.5 Consuming Deepnet Predictions

You can fully used single and batch predictions via the BigML API and bindings. The following subsections explain both tools.

Using Deepnet Predictions via the BigML API

Deepnet predictions have full citizenship in the BigML API which allows you to programmatically create, configure, retrieve, list, update, and delete single and batch predictions.

In the example below, see how to create a single prediction using a deepnet and defining the input data once you have properly set the BIGML_AUTH environment variable to contain your authentication credentials:

curl "https://bigml.io/prediction?$BIGML_AUTH" \
    -X POST \
    -H 'content-type: application/json' \
    -d '{"deepnet": "deepnet/50650bdf3c19201b64000020",
         "input_data": {"000001": 3, "000002":4.5, "000003":5}}}'

For more information on using predictions through the BigML API, please refer to the documentation.

Using Deepnet Predictions via BigML Bindings

You can also create, configure, retrieve, list, update, and delete single and batch predictions via BigML bindings which are libraries aimed to make it easier to use the BigML API from your language of choice. BigML offers bindings in multiple languages including Python, Node.js, Java, Swift and Objective-C. See below an example to create a deepnet with the Python bindings.

from bigml.api import BigML
api = BigML()
prediction = api.create_prediction(
    "deepnet/50650bdf3c19201b64000020",
    {"credit_amount": 5, "duration": 2.5})

For more information on BigML bindings, please refer to the bindings page.

5.6.6 Descriptive Information

Each deepnet prediction has an associated name, description, category, and tags. You can find a brief description of each concept in the following subsections. The More info menu option displays a panel that provides editing options. (See Figure 5.94 .)

Name

If you do not specify a name for your predictions, BigML assigns a default name depending on the type of predictions:

• Single predictions: the name always follows the structure “<deepnet name>”.

• Batch predictions: BigML combines your prediction dataset name and the deepnet name: “<deepnet name> with <dataset name>”.

Prediction names are displayed on the list and also on the top bar of a prediction view. Prediction names are indexed to be used in searches. Rename your predictions any time from the More info menu.

The name of a prediction cannot be longer than 256 characters. More than one prediction can have the same name even within the same project, but they will always have different identifiers.

Description

Each prediction also has a description that it is very useful for documenting your Machine Learning projects. Predictions take their description from the deepnet used to create them.

Descriptions can be written using plain text and also markdown. BigML provides a simple markdown editor that accepts a subset of markdown syntax. (See Figure 5.95 .)

Descriptions cannot be longer than 8192 characters.

Category

A category taken from the deepnet used to create it is associated with each prediction. Categories are useful to classify predictions according to the domain which your data comes from. This is useful when you use BigML to solve problems across industries or multiple customers.

A prediction category must be one of the categories listed on table Table 5.1 .

Tags

A prediction can also have a number of tags associated with it. These tags help to retrieve the prediction via the BigML API or to provide predictions with some extra information. Your prediction inherits the tags from the deepnet used to create it. Each tag is limited to a maximum of 128 characters. Each prediction can have up to 32 different tags.

5.6.7 Deepnet Predictions Privacy

The link displayed in the Privacy panel is the private URL of your prediction, so only a user logged into your account is able to see it. Neither single predictions nor batch predictions can be shared by using a secret link. (See Figure 5.96 .)

5.6.8 Moving Deepnet Predictions to Another Project

When you create a prediction, it will be assigned to the same project where the original deepnet is located. You cannot move predictions between projects as you do with other resources.

5.6.9 Stopping Deepnet Predictions

Single predictions are synchronous resources, so you cannot cancel them during the creation since you get the result immediately.

By contrast, batch predictions are asynchronous resources, so you can stop their creation before the task is finished. Use the delete batch prediction option from the 1-click action menu (Figure 5.97 ) or from the pop up menu on the list view.

A modal window will be displayed asking you for confirmation. If you stop the prediction during its creation you won’t be able to resume the same task again, so if you want to create the same prediction you will have to start a new task.

5.6.10 Deleting Deepnet Predictions

You can delete your single or batch predictions from the predictions view, using the 1-click action menu (see Figure 5.99 ) or using the pop up menu on the predictions list view (see Figure 5.100 ).

A modal window will be displayed asking you for confirmation. Once a prediction is deleted, it is permanently deleted, and there is no way you (or even the IT folks at BigML) can retrieve it.