f2_score for Keras

From: https://www.kaggle.com/kenmatsu4/f2-score-for-keras

Author: kenmatsu4

fbeta score: $$ \frac{(1+\beta^2)pr}{\beta^2p+r} $$

Definition of fbeta score is in the following wikipedia site. In this competition $\beta = 2$.
https://en.wikipedia.org/wiki/F1_score

Checking the score of Keras function with Scikit learn's fbeta_score function. The score is little bit different, however I think it is just small error due to epsilon value etc.

In [1]:
import numpy as np

import keras
from keras import backend as K
from keras import metrics

from sklearn.metrics import fbeta_score
Using TensorFlow backend.
In [2]:
# ref: https://github.com/keras-team/keras/blob/ac1a09c787b3968b277e577a3709cd3b6c931aa5/tests/keras/test_metrics.py
def f2_score(y_true, y_pred):
    beta = 2
    true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)), axis=1)
    predicted_positives = K.sum(K.round(K.clip(y_pred, 0, 1)), axis=1)
    possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)), axis=1)
    
    precision = true_positives / (predicted_positives + K.epsilon())
    recall = true_positives / (possible_positives + K.epsilon())
    
    return K.mean(((1+beta**2)*precision*recall) / ((beta**2)*precision+recall+K.epsilon()))
In [3]:
# Test Data
y_true_np = np.array([[0, 1, 1, 1, 0],
                      [0, 1, 1, 1, 0],
                      [0, 0, 0, 1, 1],
                      [1, 0, 1, 0, 0],
                      [1, 1, 0, 0, 0],
                      [1, 1, 0, 0, 0],
                      [1, 1, 0, 0, 0]])

y_pred_np = np.array([[0.6, 0.6, 0.6, 0.1, 0.1],
                      [0.1, 0.6, 0.6, 0.6, 0.1],
                      [0.1, 0.1, 0.6, 0.1, 0.6],
                      [0.6, 0.1, 0.1, 0.1, 0.6],
                      [0.6, 0.6, 0.6, 0.1, 0.1],
                      [0.6, 0.6, 0.6, 0.6, 0.6],
                      [0.1, 0.1, 0.1, 0.1, 0.1]])

y_true = K.variable(y_true_np)
y_pred = K.variable(y_pred_np)
WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
In [4]:
K.eval(f2_score(y_true, y_pred),)
Out[4]:
0.62071264
In [5]:
scores = []
for yt, yp in zip(y_true_np, y_pred_np):
    scores.append(fbeta_score(yt, (yp>0.5).astype(int), beta=2))
print(f"mean scores: {np.mean(scores)}")
scores
mean scores: 0.6207126207126207
/opt/conda/lib/python3.6/site-packages/sklearn/metrics/classification.py:1143: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 due to no predicted samples.
  'precision', 'predicted', average, warn_for)
Out[5]:
[0.6666666666666666,
 1.0,
 0.5,
 0.5,
 0.9090909090909091,
 0.7692307692307692,
 0.0]