Logistic Regression

The math

The references that is used

• Implementing logistic regression from scratch in Python
• Machine Learning using C++: A Beginner’s Guide to Linear and Logistic Regression

The logistic regression that LinearML performs is a Binary Logistic Regression. i.e we can classify two values

To perform a prediction, we use a bunch of values that include the weights(w), inputs(x) and bias(b). which can be written using neural-network like formula.

\[ z = \left(\sum_{i=1}^n w_i x_i \right) \]

We need an activation function to get our predictions, in case of binary logistic regression, it's called the sigmoid which is usually denoted by \(\sigma\) of \(\hat y\), and is defined as follows.

\[ \hat y = \sigma(z) = \begin{cases} \frac{1}{1 + exp(-z)}, & \text{if $z$ $\geq$ 0} \\[2ex] \frac{exp(z)}{1 + exp(z)}, & \text{otherwise} \end{cases} \]

In order to compute the loss we can use the following function.

\[ Loss(\hat y, y) = -\frac{1}{m} \sum_{i=1}^{m} y \log(\hat y) + (1 - y) \log(1 - \hat y) \]

Now, to calculate the gradients to optimize your weights using gradient descent, you must calculate the derivative of your loss function

\[ \frac{\partial Loss(\hat y, y)}{\partial w} = \frac{1}{m}(\hat y - y)x_i^T \]

\[ \frac{\partial Loss(\hat y, y)}{\partial b} = \frac{1}{m}(\hat y - y) \]

When you have the final values from your derivative calculation, you can use it in the gradient descent equation and update the weights and bias.

The code

The data used here is the Breast Cancer Wisconsin (Diagnostic) Data Set which has bee modified to look like this, where we don't have id's and M=0, and B=1

#define INCLUDE_MAT_CONVERSIONS
#include "ds/mat.h"
#include "ml/logisticregress.h"
#include "model/metrics.h"
#include "model/train_test_split.h"
#include "parsers/csv.h"

int main(void) {
    CSV *csv_reader = csv_init(569, 31, ',');
    csv_parse(csv_reader, "data/bcancer.csv");

    Mat *X = csv_get_mat_slice(csv_reader, (Slice){1, 31});
    Mat *Y = csv_get_mat_slice(csv_reader, (Slice){0, 1});
    Mat *X_train, *X_test, *Y_train, *Y_test;

    train_test_split(X, Y, &X_train, &X_test, &Y_train, &Y_test, 0.3, 101);

    logregress_set_max_iter(2000);
    LogisticRegressionModel *model = logregress_init();
    logregress_fit(model, X_train, Y_train);

    // printf("prediction: %lf\n", logregress_predict(model, (double[]){15.22, 30.62, 103.4, 716.9, ... , 0}, 30));
    Array *preds = logregress_predict_many(model, X_test);
    Array *true = mat_get_col_arr(Y_test, 0);

    logregress_print(model);

    printf("confusion matrix: \n");
    Mat *conf_mat = model_confusion_matrix(true, preds);
    mat_print(conf_mat);

    arr_free(true);
    arr_free(preds);
    logregress_free(model);
    mat_free_many(7, X, Y, X_test, X_train, Y_test, Y_train, conf_mat);
    csv_free(csv_reader);
}

LogisticRegressionModel(bias: 0.5159147, loss: -12.4263621, weights: 0x5556e8a732c0)
weights:
1546.6922009
1139.6829595
8552.1648900
2522.0044946
11.8724211
-19.3345598
-44.9646156
-18.4984994
23.8378678
10.1676564
0.2338315
103.3839701
-139.7864354
-4498.8563443
0.2662770
-6.5798244
-8.6158697
-1.6938180
1.6508702
-0.3857419
1650.7843571
1445.0283208
8312.7672485
-4024.9280673
13.2972726
-72.4527931
-111.8298475
-26.6204266
28.0612275
5.4099162
confusion matrix:
  57.00   10.00
  2.00   101.00

Now, what does the confusion matrix generated by sklean look like.

array([[ 59,   7],
       [  3, 102]])

we are pretty close... checkout the python implementation here