How to use fastai tabular with custom metric
First, we have to load our data (I used the kaggle API to do this, check out my previous blogpost)
from pathlib import Path
p = Path('/notebooks/storage/data/Titanic')
filename = Path('/notebooks/storage/data/Titanic/titanic.zip')
!unzip -q {str(filename)} -d {str(p/"train")}
import pandas as pd
from pandas.api.types import is_string_dtype, is_numeric_dtype, is_categorical_dtype
from fastai.tabular.all import *
from sklearn.metrics import confusion_matrix
df = pd.read_csv(f'{p}/train/train.csv', index_col=0)
df.tail(5)
df.shape
Let's check out the data type for each column.
df.dtypes
We see that "Pclass", "Sex", "Cabin" and "Embarked" are objects. When we think about these variables, they clearly are categorical variables. So let's change that. Also, this blogpost is not about how to train the best possible model, so we do not want any fancy feature engineering. I therefore will get rid of the variable "Name".
df['Pclass'] = df['Pclass'].astype('category')
df['Sex'] = df['Sex'].astype('category')
df['Cabin'] = df['Cabin'].astype('category')
df['Embarked'] = df['Embarked'].astype('category')
df = df.drop(columns=['Name'])
I want to build a validation set, based on the index. The easiest way is to create an array with the length of our data, randomly select a percentage of indeces we want in our training data, and use the leftover indeces for our validation set.
# Define percentage of train data, define length of training indeces and validation indeces
p = 0.9
len_df = len(df)
len_idx_tain = round(len_df*p)
len_idx_val = len_df-len_idx_tain
# build array of indeces for training and validation
idx_arr = range(0,len_df)
train_idx = np.random.choice(range(0,len_df), len_idx_tain, replace=False)
val_idx = [i for i in idx_arr if i not in train_idx]
val_idx = np.asarray(val_idx)
Let's put these arrays into a split variable, which fastai knows how to make use of to split the data into training data and validation data.
splits = (list(train_idx),list(val_idx))
Then we make use of the "cont_cat_split" function to easily get a list of column names for each categorical and continuous variable.
cont_nn, cat_nn = cont_cat_split(df, dep_var='Survived')
cont_nn, cat_nn
That looks great! How about our missing values?
df.isna().sum()
Looks like there are quite a few missing values for age and cabin. But we know a way to make use of that! Even "no-information" can be an information.
procs_nn = [Categorify, FillMissing, Normalize]
Fastai provides us with many super useful out-of-the-box functions. Three of them you can see here: Categorify will take all of the variables with type "category" will replace the category with a numerical number and write the mapping into a dictionary. FillMissing, well, does exactly this. It also provides us with a extra boolean variable, indicating whether the row has a missing value for a specific variable (e.g. age). Normalize then normalizes our continuous data and saves the standardzier.
df['Survived'] = df['Survived'].astype(np.float32)
This one is important: fastai needs the dependend variable to be of type float32, even though we have a binary classification! We need this, because the loss function will be mse-loss. This leads to possibly odd behavior: the numbers can get below 0 and even below -1 or above 2! If we use a provided metric like accuracy, this leads to odd results. So we will write our own custom metric! But first, let's use the TabularPandas class and put that in a dataloader (important: do not let the batchsize become bigger than your data, this will result in a error!).
to_nn = TabularPandas(df, procs_nn, cat_nn, cont_nn, splits=splits, y_names='Survived')
# length of data is 891, so let the batchsize be smaller than that!
dls = to_nn.dataloaders(256)
Now is the time to train our model. But wait! We first have to define a custom metric, because the accuracy provided by fastai will lead to odd results. I can show you what I mean:
learn = tabular_learner(dls, layers=[64, 8], n_out=1, metrics=[accuracy])
learn.fit_one_cycle(4,1e-3)
What is going on here? Is the accuracy not improving? The answer is no. Tabular learner in combination with accuracy does not give us the result we want. Why is that? First, our predictions will not be 0 or 1, but floating point values. That's good for our loss, so our model can learn from its gradient (checkout my blogpost about gradients if you do not know what that means). But this leads to a malfunctioning of the accuracy metric, which fastai uses from sklearn. What we need to do is to define our own custom metric. Let's do that!
import sklearn.metrics as skm
def _accumulate(self, learn):
#pred = learn.pred.argmax(dim=self.dim_argmax) if self.dim_argmax else learn.pred
m = nn.Sigmoid()
pred = learn.pred
pred = torch.round(m(pred))
targ = learn.y
pred,targ = to_detach(pred),to_detach(targ)
self.preds.append(pred)
self.targs.append(targ)
AccumMetric.accumulate = _accumulate
def BinAccu():
return skm_to_fastai(skm.accuracy_score)
So what's going on here? Next to Callbacks, fastai provides a clever way how to customize metrics. To be honest, it took me some time to figure it out, but here's how it works. Each learner has a .pred and a .y, which means this is how you can get its predictions and its targets. As I already mentioned, preds are floating point values, but we want them to be between 0 and 1 - luckily, this is exactly what the sigmoid function is doing. Then we want the values to be either 0 or 1, so let's just round these values (threshold here is .5, but you can fiddle around with that). We then append these "new" preds and targets.
"In order to provide a more flexible foundation to support metrics like this fastai provides a Metric abstract class which defines three methods: reset, accumulate, and value (which is a property). Reset is called at the start of training, accumulate is called after each batch, and then finally value is called to calculate the final check."
So with this function we directly sit on top of accumulate. The function skm_to_fastai let's you use sklearn metrics (in this case: accuracy_score) and uses the pred and targ we provided in our tiny function. Important: we have to instanciate the instance first!
binaccu = BinAccu()
learn = tabular_learner(dls, n_out=1, metrics=[binaccu])
Ok, we're good to go. Let's use fastai's awesome lr_find().
learn.lr_find()
Looks like we should set our learning rate to about 1e-3.
learn.fit_one_cycle(10,1e-3)
Our accuracy is improving until epoch 2. So we should restart our training. But first, let's have a look, whether our accuracy score is doing what we expect:
preds, targs = learn.get_preds()
m = nn.Sigmoid()
confusion_matrix(torch.round(m(preds.view(-1))).numpy(), targs.view(-1).numpy())
1-abs(torch.round(m(preds.view(-1))) - targs.view(-1)).sum()/len((torch.round(m(preds.view(-1))) - targs.view(-1)))
Awesome. That's exactly what we wanted (not the result though, which is pretty bad ;) )
Let's re-run the trainig.
binaccu = BinAccu()
learn = tabular_learner(dls, layers=[128,128], n_out=1, metrics=[binaccu])
learn.fit_one_cycle(5,1e-5)
preds, targs = learn.get_preds()
m = nn.Sigmoid()
confusion_matrix(torch.round(m(preds.view(-1))).numpy(), targs.view(-1).numpy())
To be honest, the results aren't quite that good. With a RandomForest you can easily get up to 85% accuracy. However, in this blogpost I wanted to show you how to leverage fastai's tabular_learner and add a custom metric to it.
I hope you stay tuned for the next blogpost!
Lasse