4.4 Logistic Regression Configuration Options

While the 1-click creation menu option (see section 4.3 ) provides a convenient and easy way to create a BigML logistic regression, you can also have more control over the logistic regression creation and configure a number of parameters that affect the way BigML creates logistic regressions. Click the configure logistic regression menu option in the configuration menu of your dataset view. (See Figure 4.12 .)

Note: for some datasets, the configuration option may be disabled. This can only be due to the fact that the dataset does not contain any categorical field to be selected as the Objective Field. In case the field to be predicted in your dataset is a numeric field, you can discretize it so it becomes categorical. Read the Datasets documentation [ 23 ] to learn how to discretize a numeric field.

4.4.1 Objective Field

The objective field, or “target field”, is the field you want to predict. Logistic regressions only support categorical fields as the Objective Field.

BigML takes the last categorical field in your dataset as the objective field by default. If you want to change the objective field, you have two options: you can select another field from the configuration panel to build the logistic regression, or you can change it permanently from your dataset view.

• Select the Objective field from the logistic regression configuration panel. This option will only affect the logistic regression you are building that time. (See Figure 4.13 .)

  

• Change the default objective field for the dataset. This option will save your objective field preference for any model you build. Click on the edition icon next to the field name when you mouse over it, a pop up window will be displayed. Then click on the Objective field icon and Save it. (See Figure 4.14 .)

  

4.4.2 Automatic optimization

You can turn on the Automatic optimization option so BigML will automatically tune the parameters of your logistic regression (see Figure 4.15 ).

The high number of possible combinations for parameter values makes it difficult to find the optimum configuration since the combinations that lead to a poor result outnumber the ones that result in a satisfying performance. Hand-tuning different configurations is a time-consuming process that requires a high level of expertise and intuition. To combat this problem, BigML offers first-class support for automatic logistic regression parameter optimization.

Behind the scenes, BigML uses the same technology for logistic regression parameter optimization as the one used for OptiML. If you want to know more about the technical details, please read the Chapter 2 of the document OptiML with the BigML Dashboard [ 15 ] .

When you turn on the Automatic optimization option, all the logistic regression parameters will be disabled (because they will be automatically optimized), except the Default numeric value, the Missing numerics, and the Weights parameters which you can manually configure (see Figure 4.16 ).

Since the optimization process can take some time, BigML offers two configurable parameters to limit the time to create the optimized logistic regression: a training duration (see Training duration ) and the logistic regression candidates (see Logistic regression candidates ).

Training duration

The scale parameter to regulate the logistic regression runtime. It’s set as an integer from 1 to 10. It indicates the user preference for the amount of time they wish the optimization to take. The higher the number, the more time that users are willing to wait for possibly better logistic regression performance. The lower the number, the faster that users wish the logistic regression training to finish. The default value is set to 5.

The training duration is set in a scale. The actual training time depends on the dataset size, among other factors.

Logistic regression candidates

The maximum number of different logistic regressions (i.e., logistic regressions using a unique configuration) to be trained and evaluated during the optimization process. The default number is 128 candidates which is usually enough to find the best logistic regression, but you can set it from 4 up to 200. Only the top-performing logistic regression will be returned. If training duration is very low (see Training duration ) given the dataset size, it is possible that not all the logistic regression candidates will be tried out.

4.4.3 Weights

It is not unusual for a dataset to have some categories that are common and others very rare. For example, in datasets used to predict fraud, usually fraudulent transactions are very scarce compared to regular ones. When this happens, logistic regression tends to predict the most frequent values simply because the overall model’s performance metrics improve with that approach. However, in cases such as fraud prediction, you may be more interested in predicting rare values rather than successfully predicting frequent ones. In that case, you may want to assign more weights to the scarce instances so they are equivalent to the abundant ones.

BigML provides three different options to assign specific weights to your instances, balance objective, objective weights, and weight field explained in the following sections (see Figure 4.19 ).

Balance Objective

When you set the balance objective weight (see Figure 2.21 ), BigML automatically balances the classes of the objective field by assigning a higher weight to the less frequent classes, with the most frequent class always having a weight of 1. For example, take the following frequencies for each class:

[False, 2000; True, 50]

By enabling the Balance objective option, BigML will automatically apply the following weights:

[False, 1; True, 40]

In this example, the class “True” is getting forty times more weight as it is forty times less frequent than the most abundant class.

Objective Weights

The Objective weights option allows you to manually set a specific weight for each class of the objective field. BigML oversamples your weighted instances replicating them as many times as the weight stablishes. If you do not list a class, it is assumed to have a weight of 1. Weights of 0 are also valid, but if all classes have a weight of 0, the logistic regression creation will produce an error.

This option can be combined with the Weight field (see Weight Field ). When combining it with the Weight field, both weights are multiplied. For example if you assign a weight of 3 for the “True” class and the weight field assigns a weight of 2 for a given instance labeled as “True”, that instance will have a total weight of 6.

Weight Field

The Weight field option allows you to assign individual weights to each instance by choosing a special weight field. The selected field must be numeric and it must not contain any negative or missing values. The weight field will be excluded from the input fields when building the logistic regression. You can select an existing field in your dataset or you may create a new one in order to assign customized weights.

The weights of your weight field will impact in the same way as the Objective weights. If an instance has a weight of 3 it will be replicated three times in the dataset to train the model.

4.4.4 Default Numeric Value

Logistic regressions can include missing values as valid values for any type of fields as explained in subsection 4.2.2 . However, there can be situations for which you don’t want to include them in your model. For those cases, the Default numeric value parameter is an easy way to replace missing numeric values by another valid value. You can select to replace them by the field’s Mean, Median, Maximum, Minimum or by Zero. (See Figure 4.20 .)

Note: if your dataset does not contain missing values for your numeric fields, this parameter will not have impact on your logistic regression. By contrast, if your dataset contains missing numeric values and you neither select a Default numeric value or enable the Missing numerics configuration option, instances with missing numeric values will be ignored to build the model. (See next subsection 4.4.5 .)

4.4.5 Missing Numerics

By default, missing values for your numeric fields are included as valid values to build your logistic regression. However, as explained in the previous subsection, there can be cases for which you don’t want them to be included in syour model. The Missing numerics option allows you to select if you want to include or exclude the missing numeric values to build your logistic regression. (See Figure 4.21 .)

Note: missing values are always included for categorical, text, and items fields. (See section 4.2 .)

As explained in subsection 4.2.2 , by including missing numeric values, you will get an additional coefficient for your numeric fields denoting the missing values. You will find them at the end of the coefficients table as you can see in Figure 4.22 . If a numeric field has no missing values, those coefficients will be zero.

Missing numerics and Default numeric value parameters are substitutes. (See subsection 4.4.4 .) Therefore, if you enable the default numeric value parameter, the Missing numerics icon will automatically be disabled. (See Figure 4.23 .)

Note: if your dataset contains missing numeric values and you do not either enable Missing numerics or select a Default numeric value, instances containing missing values will be ignored when building the logistic regression.

4.4.6 Eps

This parameter sets the stopping criteria for the solver. If the difference between the current results and the last iteration results is smaller than Eps, then the solver is finished. You can set positive float values greater than 0 and smaller than 1. The default value is 0.0001. Higher values make the model to be faster built but they may result in a worse predictive performance.

4.4.7 Stats

This parameter allows you to include or exclude the statistical tests to assess the quality of the model’s fit to the data. (See Figure 4.25 .) BigML does not include these statistics by default to speed up the model creation because the time required to compute them can be considerably high.

When stats are enabled, BigML automatically configures your categorical fields with dummy coding. BigML selects the dummy class in alphabetical order, therefore for fields with classes such us “0”/”1”, “yes”/”no” or “true/”false”, the dummy class will always be “0”, “no” or “false”. You can see the field codings configuration by clicking in the wheel next to your categorical fields. (See Figure 4.26 .) This automatic configuration is to avoid multicollinearity in your model, otherwise all the statistics computed will be null. You can configure the categorical fields with any of the other field codings, contrast or other, but you cannot configure one-hot coding again if stats are enabled. See One-hot Coding to read more about field codings.

BigML computes two kinds of statistical tests, a test that measures the quality of the coefficients as a whole, and a set of tests that measure the significance of the individual coefficient estimates:

• Statistical test to measure the quality of the whole model:

  • Likelihood ratio: tests whether the coefficients as a whole have any predictive power to predict a certain class of the objective field over an intercept-only model.

• Statistical tests to measure the significance of the individual coefficient estimates:

  • Standard error: the variance of the coefficient estimates.

  • Z scores: how many standard deviations are the coefficient estimates from the mean. Letting \(\beta \) be the coeffcient estimate and \(SE\) the standard error, then the formula for each \(i\) coefficient is:

    \[ \text{Z score}_i = \frac{\beta _i}{SE(\beta _i)} \]

  • p-value: determines the statistical significance of each coefficient in the logistic regression formula. The p-value is associated with the \(\chi ^2\) of the Wald test with one degree of freedom at the \(Z^2\) value. The Wald test indicates how far the estimated parameters are from zero to test their significance.

    The p-value needs to be compared against a significance level, usually 0.1, 0.05 or 0.01. A p-value higher than the significance level, indicates that the null hypothesis can be accepted, hence the coefficient is non-significant. A p-value lower than the significance level indicates that the null hypothesis can be rejected, so the coefficient estimate is significant. A good practice is to retrain the logistic regression removing the non-significant coefficients.

    Note: note that p-values are not extracted directly from Z score values, they are associated with the \(\chi ^2\) of the Wald test.

  • Confidence intervals: give the size of the 95% confidence interval for each coefficient estimate. That is, for a coefficient estimate \(\beta \), and an interval value \(n\), the value of the coefficient is \(\beta \pm n\) with a confidence of 95%.

Once your logistic regression has been created, you will be able to visualize the stats in the coefficient table. (See Coefficient Table with Stats Computation .)

To avoid lengthy computation times, stats from large input datasets will be computed from a subsample of the dataset such that the number of coefficients \(X\) rows is less than or equal to 1E+8.

It is possible that some statistic values contain null values. Wald test statistics can not be computed for zero-value coefficients, and so their corresponding entries are null. Moreover, if the coefficients’ information matrix is ill-conditioned, e.g. if there are fewer instances of the positive class than the number of coefficients, or if there is a perfect correlation between the input fields (multicollinearity), the standard error, Z score, p-value, and confidence intervals will also have a null value.

4.4.8 Bias and auto-scaling

You can include or exclude the Bias from the model, a.k.a. the intercept term of the logistic regression formula. (See formula in section 4.2 .) For most cases, including the bias results in a better model. By default it is included. (See Figure 4.27 .)

You can also scale the numeric fields of your dataset, to ensure each field will have equivalent influence despite differences in magnitudes, e.g., salary and age. By enabling the Auto-scale parameter, BigML automatically transforms your numeric fields so their standard deviations equal 1. Auto-scaling fields will allow you to compare the different coefficients learned by the model as explained in section 4.2 . Fields are auto-scaled by default. (See Figure 4.27 .)

4.4.9 Regularization

The main goal for having a regularization factor is to avoid Overfitting, i.e., tailoring the model to the training data at the expense of generalization.

In BigML you can choose between L1 or L2 Regularization. L1 norm causes more coefficients to be zero, while using the L2 norm forces the values of all coefficients towards zero. Usually L2 yields better results, so it is the default option to create your logistic regression.

You can also tune the Strength (c) parameter, which is the inverse of the regularization strength, so higher values indicate less regularization. It must be a positive integer greater than 0. Too high values for strength will make the algorithm perfectly fit the training data boundaries, so the logistic regression will perform poorly when trying to predict new instances not seen before by the model. Too low values for strength will result in vague decision boundaries not following the data patterns, hence resulting also in a bad predictive performance. The default value is 1 which usually works well for most cases. (See Figure 4.28 .)

4.4.10 Field Codings

Categorical fields must be converted to numeric values in order to train a logistic regression model. By default, they are One-hot encoded, but BigML allows you to configure three other types of coding for each one of your categorical fields: Dummy coding, Contrast coding, and Other coding. See the following subsections for a detail explanation of each option. (Learn more about input fields transformations in subsection 4.2.1 .)

Note: if stats are enabled, then the default coding is Dummy for all your categorical fields, selecting the dummy class by alphabetical order. See subsection 4.4.7 for more details.

One-hot Coding

Categorical fields are One-hot encoded by default. That is, a separate 0-1 numeric field is created for each category and an additional one for missing values. For a given instance, the variable corresponding to the instance’s categorical value has its value set to 1, while the other variables are set to 0. See an example of One-hot coding scheme for a field containing three classes in Table 4.1 :

Dummy Coding

The main goal of using dummy coding is to compare a class selected as the reference or control class with the rest of classes. The control class is assigned a value of 0 for each variable. The control class is called dummy class in BigML and it is usually a class with a representative number of instances compared to the other classes in the dataset. See an example of dummy coding schema for three different classes, being the “Class 1” the dummy class, in Table 4.2 :

To set Dummy coding for a field:

1. Click on the configuration icon next to the field name. (See Figure 4.29 .)

   

2. A modal window will be displayed so you can configure the field codings for that field. If the field has not a previous configuration for field codings, it will be disable. Enable field coding configuration by clicking on the green switcher shown in Figure 4.30 .

   

3. Select the class you want to set as the dummy class. (See Figure 4.31 .)

   

4. Click Save . Make sure you saved your configuration by looking at the bottom message “Configured Coding: DUMMY”. (See Figure 4.32 .)

   

   Note: you cannot select several field codings for the same field simultaneously.

5. Close the modal window by clicking outside or by clicking Cancel .

   

   Note: if the Cancel button is red, it indicates there are changes you have not saved yet so you will lose them by closing the modal window.

6. After configuring the field codings for a field, the configuration icon will become green. (See Figure 4.34 .)

   

7. To remove the field coding configuration for that field, click Disable from the switcher and click Save again. (See Figure 4.35 .)

   

After creating your logistic regression, your dummy class will be identified with the dummy icon in the coefficients table view (see subsection 4.5.2 ). (See Figure 4.36 .)

Contrast Coding

Contrast coding allows you to set different values for different classes. Instead of the 0-1 values of One-hot coding, you will be able to set any integer or float value for each of the classes, plus an additional one for missing values. The sum of all values must equal 0. The values of the classes need to be set based on certain hypothesis, e.g., higher values for a class assume this class has more influence on the objective field than the others. A positive value indicates a positive relationship between the class and the objective field while a negative value indicates a negative relationship. A coefficient of 0 will exclude the class from the model. In the Table 4.3 you can see an example of contrast coding schema for three different classes.

To set Contrast coding for a field, follow these steps:

1. Click on the configuration icon next to the field name. (See Figure 4.37 .)

   

2. A modal window will be displayed so you can configure the field codings for that field. If the field has not a previous configuration for field codings, it will be disable. Enable field coding configuration by clicking on the green switcher shown in Figure 4.38

   

3. Select the Contrast coding option. (See Figure 4.39 .)

   

4. Set the values you want for your classes based on your hypothesis. All classes values must sum 0. (See Figure 4.40 .) By using the BigML API, multiple contrast codings can be given for a field as long as all the codings are Orthogonal to ensure there are no co-dependent coefficients. Check the corresponding documentation.

   

   Note: you cannot select several field codings for the same field simultaneously.

5. Click Save . Make sure you saved your configuration by looking at the bottom message “Configured Coding: CONTRAST”. (See Figure 4.41 .)

   

6. Close the modal window by clicking outside or by clicking Cancel .

   

   Note: if the Cancel button is red, it indicates there are changes you have not saved yet so you will lose them by closing the modal window.

7. After configuring the field codings for a field, the configuration icon will become green. (See Figure 4.43 .)

   

8. To remove the field coding configuration for that field, click Disable from the switcher and click Save again. (See Figure 4.44 .)

   

After creating your logistic regression, you will be able to see your Contrast coding values in the coefficients table view (see subsection 4.5.2 ) by clicking on the icon. (See Figure 4.45 .)

A modal window will be displayed with your codings values and you can download them in CSV or JSON format by clicking on the corresponding icons. (See Figure 4.45 .)

Other Coding

Other coding allows you to set different values for different classes. It works the same way as contrast coding (see Contrast Coding ), but in this case the values do not need to sum 0. In the Table 4.4 you can see an example of other coding schema for three different classes.

To set Other coding for a field, follow these steps:

1. Click on the configuration icon next to the field name. (See Figure 4.47 .)

   

2. A modal window will be displayed so you can configure the field codings for that field. If the field has not a previous configuration for field codings, it will be disable. Enable field coding configuration by clicking on the green switcher shown in Figure 4.48

   

3. Select the Other coding option. (See Figure 4.39 .)

   

4. Set the values you want for your classes based on your hypothesis. You can set any float or integer value. (See Figure 4.50 .) By using the BigML API, multiple other codings can be given for a field. Check the corresponding documentation.

   

   Note: you cannot select several field codings for the same field simultaneously.

5. Click Save . Make sure you saved your configuration by looking at the bottom message “Configured Coding: OTHER”. (See Figure 4.51 .)

   

6. Close the modal window by clicking outside or by clicking Cancel .

   

   Note: if the Cancel button is red, it indicates there are changes you have not saved yet so you will lose them by closing the modal window.

7. After configuring the field codings for a field, the configuration icon will become green. (See Figure 4.53 .)

   

8. To remove the field coding configuration for that field, click Disable from the switcher and click Save again. (See Figure 4.54 .)

   

After creating your logistic regression, you will be able to see your Other coding values in the coefficients table view (see subsection 4.5.2 ) by clicking on the icon. (See Figure 4.55 .)

A modal window will be displayed with your coding values and you can download them in CSV or JSON format by clicking on the corresponding icons. (See Figure 4.55 .)

4.4.11 Sampling Options

Sometimes you do not need all the data contained in your dataset to build your logistic regression. If you have a very large dataset, sampling may be a good way of getting faster results. BigML allows you to sample your dataset before creating the logistic regression, so you do not need to create a separate dataset first. You can find a detailed explanation of the sampling parameters available in the following subsections. (See Figure 4.57 .)

Rate

The Rate is the proportion of instances to include in your sample. Set any value between 0% and 100%. Defaults to 100%.

Range

Specifies a subset of instances from which to sample, e.g., choose from instance 1 until 200. The Rate you set will be computed over the Range configured. This option may be useful when you have temporal data, and you want to train your logistic regression with historical data, and test it with the most recent one to check if it can predict based on time.

Sampling

By default, BigML selects your instances for the sample by using a random number generator, which means two samples from the same dataset will likely be different even when using the same rates and row ranges. If you choose deterministic sampling, the random-number generator will always use the same seed, thus producing repeatable results. This lets you work with identical samples from the same dataset.

Replacement

Sampling with replacement allows a single instance to be selected multiple times. Sampling without replacement ensures that each instance cannot be selected more than once. By default, BigML generates samples without replacement.

Out of bag

This argument will create a sample containing only out-of-bag instances for the currently defined rate, so the final total number of instances for your sample will be one minus the rate configured for your sample (when replacement is false). This can be useful for splitting a dataset into training and testing subsets. It is only selectable when a sample rate is less than 100%.

4.4.12 Creating Logistic Regressions with Configured Options

After finishing the configuration of your options, you can change the default logistic regression name in the editable text box. Then you can click on the Create logistic regression button to create the new logistic regression, or reset the configuration by clicking on the Reset button.

4.4.13 API Request Preview

The API Request Preview button is in the middle on the bottom of the configuration panel, next to the Reset button (See (Figure 4.58 )). This is to show how to create the logistic regression programmatically: the endpoint of the REST API call and the JSON that specifies the arguments configured in the panel. Please see (Figure 4.59 ) below:

There are options on the upper right to either export the JSON or copy it to clipboard. On the bottom there is a link to the API documentation for logistic regressions, in case you need to check any of the possible values or want to extend your knowledge in the use of the API to automate your workflows.

Please note: when a default value for an argument is used in the configuration, the argument won’t appear in the generated JSON. Because during API calls, default values are used when arguments are missing, there is no need to send them in the creation request.