iMet - Keras pretrained model as feature extractor

From: https://www.kaggle.com/dimitreoliveira/imet-keras-pretrained-model-as-feature-extractor

Author: DimitreOliveira

Score: 0.474

Using the bottleneck features of a pre-trained network

I'm sharing this code since it can get a little tricky to beginners do this, especially finding the right methods, APIs and models, I hope this can help someone.

Using the bottleneck features of a pre-trained model is basically using the models as a preprocessing step on your data, so you get a model (VGG16 in this case) and pass your data through it, the output will be the representation of your data according to the model. Then take these features and use on any other model (another deep learning model or even SVM).

What you need to know:

  • Similar to fine-tuning you need to remove the top of the pre-trained model.
  • Essentially it works as a pipeline with two models.
  • To have good results with this approach your data need to be similar to the same data used to train the model (ImageNet).
  • This will make your training a lot faster since you only need to pass all data once through the big model (VGG 16) than just train a smaller one on a less complex data.

Dependencies

In [1]:
import os
import cv2
import math
import warnings
import numpy as np
import pandas as pd
import seaborn as sns
import tensorflow as tf
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.metrics import confusion_matrix, fbeta_score
from keras import optimizers
from keras import backend as K
from keras.models import Sequential
from keras.applications.vgg16 import VGG16
from keras.preprocessing.image import ImageDataGenerator
from keras.callbacks import LearningRateScheduler, EarlyStopping
from keras.layers import Dense, Dropout, Flatten, Conv2D, MaxPool2D, Activation, BatchNormalization

# Set seeds to make the experiment more reproducible.
from tensorflow import set_random_seed
from numpy.random import seed
set_random_seed(0)
seed(0)

%matplotlib inline
sns.set(style="whitegrid")
warnings.filterwarnings("ignore")
Using TensorFlow backend.

Load data

In [2]:
train = pd.read_csv('../input/imet-2019-fgvc6/train.csv')
labels = pd.read_csv('../input/imet-2019-fgvc6/labels.csv')
test = pd.read_csv('../input/imet-2019-fgvc6/sample_submission.csv')

train["attribute_ids"] = train["attribute_ids"].apply(lambda x:x.split(" "))
train["id"] = train["id"].apply(lambda x: x + ".png")
test["id"] = test["id"].apply(lambda x: x + ".png")

print('Number of train samples: ', train.shape[0])
print('Number of test samples: ', test.shape[0])
print('Number of labels: ', labels.shape[0])
display(train.head())
display(labels.head())
Number of train samples:  109237
Number of test samples:  7443
Number of labels:  1103
id attribute_ids
0 1000483014d91860.png [147, 616, 813]
1 1000fe2e667721fe.png [51, 616, 734, 813]
2 1001614cb89646ee.png [776]
3 10041eb49b297c08.png [51, 671, 698, 813, 1092]
4 100501c227f8beea.png [13, 404, 492, 903, 1093]
attribute_id attribute_name
0 0 culture::abruzzi
1 1 culture::achaemenid
2 2 culture::aegean
3 3 culture::afghan
4 4 culture::after british
In [3]:
# Parameters
BATCH_SIZE = 64
EPOCHS = 200
LEARNING_RATE = 0.0001
HEIGHT = 128
WIDTH = 128
CANAL = 3
N_CLASSES = labels.shape[0]
ES_PATIENCE = 5
DECAY_DROP = 0.5
DECAY_EPOCHS = 10
classes = list(map(str, range(N_CLASSES)))
In [4]:
def f2_score_thr(threshold=0.5):
    def f2_score(y_true, y_pred):
        beta = 2
        y_pred = K.cast(K.greater(K.clip(y_pred, 0, 1), threshold), K.floatx())

        true_positives = K.sum(K.clip(y_true * y_pred, 0, 1), axis=1)
        predicted_positives = K.sum(K.clip(y_pred, 0, 1), axis=1)
        possible_positives = K.sum(K.clip(y_true, 0, 1), axis=1)

        precision = true_positives / (predicted_positives + K.epsilon())
        recall = true_positives / (possible_positives + K.epsilon())

        return K.mean(((1+beta**2)*precision*recall) / ((beta**2)*precision+recall+K.epsilon()))
    return f2_score

def step_decay(epoch):
    initial_lrate = LEARNING_RATE
    drop = DECAY_DROP
    epochs_drop = DECAY_EPOCHS
    lrate = initial_lrate * math.pow(drop, math.floor((1+epoch)/epochs_drop))
    
    return lrate
In [5]:
train_datagen = ImageDataGenerator(rescale=1./255)

train_generator = train_datagen.flow_from_dataframe(
    dataframe=train,
    directory="../input/imet-2019-fgvc6/train",
    x_col="id",
    y_col="attribute_ids",
    batch_size=BATCH_SIZE,
    shuffle=False,
    class_mode=None,
    target_size=(HEIGHT, WIDTH))

test_datagen = ImageDataGenerator(rescale=1./255)

test_generator = test_datagen.flow_from_dataframe(  
        dataframe=test,
        directory = "../input/imet-2019-fgvc6/test",    
        x_col="id",
        target_size=(HEIGHT, WIDTH),
        batch_size=1,
        shuffle=False,
        class_mode=None)
Found 109237 images.
Found 7443 images.

Bottleneck model - VGG16 (feature extractor)

In [6]:
model_vgg = VGG16(weights=None, include_top=False)
model_vgg.load_weights('../input/vgg16/vgg16_weights_tf_dim_ordering_tf_kernels_notop.h5')
WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.

Pass all the train data through the pre-trained model to extract features

In [7]:
STEP_SIZE_TRAIN = train_generator.n // train_generator.batch_size
train_data = model_vgg.predict_generator(train_generator, STEP_SIZE_TRAIN)

Recreate the train labels

In [8]:
train_labels = []
for label in train['attribute_ids'][:train_data.shape[0]].values:
    zeros = np.zeros(N_CLASSES)
    for label_i in label:
        zeros[int(label_i)] = 1
    train_labels.append(zeros)
    
train_labels = np.asarray(train_labels)

Split the new train data into train and validation for the next model

In [9]:
X_train, X_val, Y_train, Y_val = train_test_split(train_data, train_labels, test_size=0.2, random_state=0)

Second model - Deep Learning MLP

In [10]:
model = Sequential()
model.add(Flatten(input_shape=train_data.shape[1:]))
model.add(Dense(2048, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(1024, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(N_CLASSES, activation="sigmoid"))

optimizer = optimizers.Adam(lr=LEARNING_RATE)
thresholds = [0.1, 0.15, 0.2, 0.25, 0.28, 0.3, 0.4, 0.5]
metrics = ["accuracy", "categorical_accuracy", f2_score_thr(0.1), f2_score_thr(0.15), f2_score_thr(0.2), 
           f2_score_thr(0.25), f2_score_thr(0.28), f2_score_thr(0.3), f2_score_thr(0.4), f2_score_thr(0.5)]
lrate = LearningRateScheduler(step_decay)
es = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=ES_PATIENCE)
callbacks = [es]
model.compile(optimizer=optimizer, loss="binary_crossentropy",  metrics=metrics)
WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:3445: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.
Instructions for updating:
Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.
In [11]:
history = model.fit(x=X_train, y=Y_train,
                    validation_data=(X_val, Y_val),
                    epochs=EPOCHS,
                    batch_size=BATCH_SIZE,
                    callbacks=callbacks,
                    verbose=2)
WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
Train on 87347 samples, validate on 21837 samples
Epoch 1/200
 - 18s - loss: 0.0253 - acc: 0.9925 - categorical_accuracy: 0.0686 - f2_score: 0.1695 - f2_score_1: 0.1414 - f2_score_2: 0.1162 - f2_score_3: 0.0953 - f2_score_4: 0.0845 - f2_score_5: 0.0781 - f2_score_6: 0.0519 - f2_score_7: 0.0337 - val_loss: 0.0128 - val_acc: 0.9972 - val_categorical_accuracy: 0.1243 - val_f2_score: 0.2746 - val_f2_score_1: 0.2165 - val_f2_score_2: 0.1681 - val_f2_score_3: 0.1316 - val_f2_score_4: 0.1134 - val_f2_score_5: 0.1031 - val_f2_score_6: 0.0615 - val_f2_score_7: 0.0353
Epoch 2/200
 - 17s - loss: 0.0135 - acc: 0.9971 - categorical_accuracy: 0.1220 - f2_score: 0.2752 - f2_score_1: 0.2369 - f2_score_2: 0.1994 - f2_score_3: 0.1672 - f2_score_4: 0.1498 - f2_score_5: 0.1390 - f2_score_6: 0.0950 - f2_score_7: 0.0644 - val_loss: 0.0120 - val_acc: 0.9972 - val_categorical_accuracy: 0.1556 - val_f2_score: 0.3363 - val_f2_score_1: 0.2918 - val_f2_score_2: 0.2458 - val_f2_score_3: 0.2041 - val_f2_score_4: 0.1824 - val_f2_score_5: 0.1696 - val_f2_score_6: 0.1181 - val_f2_score_7: 0.0810
Epoch 3/200
 - 17s - loss: 0.0125 - acc: 0.9972 - categorical_accuracy: 0.1467 - f2_score: 0.3189 - f2_score_1: 0.2828 - f2_score_2: 0.2443 - f2_score_3: 0.2086 - f2_score_4: 0.1892 - f2_score_5: 0.1769 - f2_score_6: 0.1263 - f2_score_7: 0.0898 - val_loss: 0.0115 - val_acc: 0.9973 - val_categorical_accuracy: 0.1723 - val_f2_score: 0.3667 - val_f2_score_1: 0.3330 - val_f2_score_2: 0.2931 - val_f2_score_3: 0.2522 - val_f2_score_4: 0.2280 - val_f2_score_5: 0.2130 - val_f2_score_6: 0.1500 - val_f2_score_7: 0.1070
Epoch 4/200
 - 17s - loss: 0.0120 - acc: 0.9972 - categorical_accuracy: 0.1584 - f2_score: 0.3464 - f2_score_1: 0.3128 - f2_score_2: 0.2747 - f2_score_3: 0.2379 - f2_score_4: 0.2176 - f2_score_5: 0.2048 - f2_score_6: 0.1506 - f2_score_7: 0.1097 - val_loss: 0.0111 - val_acc: 0.9973 - val_categorical_accuracy: 0.1669 - val_f2_score: 0.3882 - val_f2_score_1: 0.3608 - val_f2_score_2: 0.3245 - val_f2_score_3: 0.2883 - val_f2_score_4: 0.2683 - val_f2_score_5: 0.2540 - val_f2_score_6: 0.1934 - val_f2_score_7: 0.1418
Epoch 5/200
 - 17s - loss: 0.0115 - acc: 0.9973 - categorical_accuracy: 0.1705 - f2_score: 0.3663 - f2_score_1: 0.3339 - f2_score_2: 0.2967 - f2_score_3: 0.2591 - f2_score_4: 0.2383 - f2_score_5: 0.2254 - f2_score_6: 0.1701 - f2_score_7: 0.1263 - val_loss: 0.0110 - val_acc: 0.9973 - val_categorical_accuracy: 0.1583 - val_f2_score: 0.4003 - val_f2_score_1: 0.3840 - val_f2_score_2: 0.3526 - val_f2_score_3: 0.3206 - val_f2_score_4: 0.3007 - val_f2_score_5: 0.2875 - val_f2_score_6: 0.2241 - val_f2_score_7: 0.1721
Epoch 6/200
 - 16s - loss: 0.0112 - acc: 0.9973 - categorical_accuracy: 0.1812 - f2_score: 0.3809 - f2_score_1: 0.3510 - f2_score_2: 0.3145 - f2_score_3: 0.2777 - f2_score_4: 0.2565 - f2_score_5: 0.2433 - f2_score_6: 0.1850 - f2_score_7: 0.1402 - val_loss: 0.0108 - val_acc: 0.9974 - val_categorical_accuracy: 0.1715 - val_f2_score: 0.4131 - val_f2_score_1: 0.4002 - val_f2_score_2: 0.3743 - val_f2_score_3: 0.3411 - val_f2_score_4: 0.3208 - val_f2_score_5: 0.3073 - val_f2_score_6: 0.2423 - val_f2_score_7: 0.1861
Epoch 7/200
 - 17s - loss: 0.0110 - acc: 0.9973 - categorical_accuracy: 0.1925 - f2_score: 0.3943 - f2_score_1: 0.3657 - f2_score_2: 0.3295 - f2_score_3: 0.2931 - f2_score_4: 0.2723 - f2_score_5: 0.2589 - f2_score_6: 0.1997 - f2_score_7: 0.1520 - val_loss: 0.0108 - val_acc: 0.9974 - val_categorical_accuracy: 0.1848 - val_f2_score: 0.4200 - val_f2_score_1: 0.4106 - val_f2_score_2: 0.3839 - val_f2_score_3: 0.3511 - val_f2_score_4: 0.3319 - val_f2_score_5: 0.3189 - val_f2_score_6: 0.2555 - val_f2_score_7: 0.1998
Epoch 8/200
 - 16s - loss: 0.0107 - acc: 0.9974 - categorical_accuracy: 0.2018 - f2_score: 0.4059 - f2_score_1: 0.3782 - f2_score_2: 0.3432 - f2_score_3: 0.3065 - f2_score_4: 0.2857 - f2_score_5: 0.2723 - f2_score_6: 0.2121 - f2_score_7: 0.1640 - val_loss: 0.0107 - val_acc: 0.9974 - val_categorical_accuracy: 0.1937 - val_f2_score: 0.4242 - val_f2_score_1: 0.4147 - val_f2_score_2: 0.3909 - val_f2_score_3: 0.3619 - val_f2_score_4: 0.3425 - val_f2_score_5: 0.3298 - val_f2_score_6: 0.2672 - val_f2_score_7: 0.2118
Epoch 9/200
 - 17s - loss: 0.0105 - acc: 0.9974 - categorical_accuracy: 0.2117 - f2_score: 0.4161 - f2_score_1: 0.3899 - f2_score_2: 0.3555 - f2_score_3: 0.3193 - f2_score_4: 0.2981 - f2_score_5: 0.2845 - f2_score_6: 0.2231 - f2_score_7: 0.1741 - val_loss: 0.0105 - val_acc: 0.9974 - val_categorical_accuracy: 0.2069 - val_f2_score: 0.4335 - val_f2_score_1: 0.4219 - val_f2_score_2: 0.3980 - val_f2_score_3: 0.3671 - val_f2_score_4: 0.3488 - val_f2_score_5: 0.3364 - val_f2_score_6: 0.2738 - val_f2_score_7: 0.2171
Epoch 10/200
 - 17s - loss: 0.0103 - acc: 0.9974 - categorical_accuracy: 0.2200 - f2_score: 0.4251 - f2_score_1: 0.3999 - f2_score_2: 0.3656 - f2_score_3: 0.3308 - f2_score_4: 0.3105 - f2_score_5: 0.2972 - f2_score_6: 0.2361 - f2_score_7: 0.1855 - val_loss: 0.0104 - val_acc: 0.9974 - val_categorical_accuracy: 0.2063 - val_f2_score: 0.4357 - val_f2_score_1: 0.4267 - val_f2_score_2: 0.4026 - val_f2_score_3: 0.3732 - val_f2_score_4: 0.3550 - val_f2_score_5: 0.3420 - val_f2_score_6: 0.2808 - val_f2_score_7: 0.2236
Epoch 11/200
 - 17s - loss: 0.0102 - acc: 0.9974 - categorical_accuracy: 0.2272 - f2_score: 0.4334 - f2_score_1: 0.4086 - f2_score_2: 0.3759 - f2_score_3: 0.3412 - f2_score_4: 0.3208 - f2_score_5: 0.3073 - f2_score_6: 0.2451 - f2_score_7: 0.1934 - val_loss: 0.0103 - val_acc: 0.9974 - val_categorical_accuracy: 0.2105 - val_f2_score: 0.4404 - val_f2_score_1: 0.4334 - val_f2_score_2: 0.4095 - val_f2_score_3: 0.3805 - val_f2_score_4: 0.3627 - val_f2_score_5: 0.3504 - val_f2_score_6: 0.2907 - val_f2_score_7: 0.2344
Epoch 12/200
 - 17s - loss: 0.0100 - acc: 0.9974 - categorical_accuracy: 0.2348 - f2_score: 0.4414 - f2_score_1: 0.4176 - f2_score_2: 0.3851 - f2_score_3: 0.3509 - f2_score_4: 0.3304 - f2_score_5: 0.3173 - f2_score_6: 0.2549 - f2_score_7: 0.2022 - val_loss: 0.0104 - val_acc: 0.9974 - val_categorical_accuracy: 0.2266 - val_f2_score: 0.4441 - val_f2_score_1: 0.4402 - val_f2_score_2: 0.4212 - val_f2_score_3: 0.3935 - val_f2_score_4: 0.3758 - val_f2_score_5: 0.3636 - val_f2_score_6: 0.3039 - val_f2_score_7: 0.2472
Epoch 13/200
 - 17s - loss: 0.0099 - acc: 0.9975 - categorical_accuracy: 0.2429 - f2_score: 0.4493 - f2_score_1: 0.4265 - f2_score_2: 0.3943 - f2_score_3: 0.3604 - f2_score_4: 0.3403 - f2_score_5: 0.3272 - f2_score_6: 0.2651 - f2_score_7: 0.2118 - val_loss: 0.0103 - val_acc: 0.9974 - val_categorical_accuracy: 0.2230 - val_f2_score: 0.4456 - val_f2_score_1: 0.4415 - val_f2_score_2: 0.4214 - val_f2_score_3: 0.3935 - val_f2_score_4: 0.3755 - val_f2_score_5: 0.3640 - val_f2_score_6: 0.2995 - val_f2_score_7: 0.2426
Epoch 14/200
 - 17s - loss: 0.0097 - acc: 0.9975 - categorical_accuracy: 0.2490 - f2_score: 0.4563 - f2_score_1: 0.4337 - f2_score_2: 0.4026 - f2_score_3: 0.3691 - f2_score_4: 0.3492 - f2_score_5: 0.3363 - f2_score_6: 0.2740 - f2_score_7: 0.2205 - val_loss: 0.0101 - val_acc: 0.9975 - val_categorical_accuracy: 0.2335 - val_f2_score: 0.4496 - val_f2_score_1: 0.4453 - val_f2_score_2: 0.4262 - val_f2_score_3: 0.4006 - val_f2_score_4: 0.3835 - val_f2_score_5: 0.3723 - val_f2_score_6: 0.3120 - val_f2_score_7: 0.2571
Epoch 15/200
 - 17s - loss: 0.0096 - acc: 0.9975 - categorical_accuracy: 0.2582 - f2_score: 0.4635 - f2_score_1: 0.4421 - f2_score_2: 0.4110 - f2_score_3: 0.3776 - f2_score_4: 0.3577 - f2_score_5: 0.3443 - f2_score_6: 0.2830 - f2_score_7: 0.2293 - val_loss: 0.0101 - val_acc: 0.9975 - val_categorical_accuracy: 0.2300 - val_f2_score: 0.4532 - val_f2_score_1: 0.4490 - val_f2_score_2: 0.4269 - val_f2_score_3: 0.4007 - val_f2_score_4: 0.3826 - val_f2_score_5: 0.3709 - val_f2_score_6: 0.3132 - val_f2_score_7: 0.2578
Epoch 16/200
 - 17s - loss: 0.0095 - acc: 0.9975 - categorical_accuracy: 0.2661 - f2_score: 0.4710 - f2_score_1: 0.4502 - f2_score_2: 0.4198 - f2_score_3: 0.3868 - f2_score_4: 0.3675 - f2_score_5: 0.3545 - f2_score_6: 0.2924 - f2_score_7: 0.2385 - val_loss: 0.0102 - val_acc: 0.9975 - val_categorical_accuracy: 0.2378 - val_f2_score: 0.4534 - val_f2_score_1: 0.4517 - val_f2_score_2: 0.4328 - val_f2_score_3: 0.4057 - val_f2_score_4: 0.3879 - val_f2_score_5: 0.3766 - val_f2_score_6: 0.3202 - val_f2_score_7: 0.2675
Epoch 17/200
 - 17s - loss: 0.0094 - acc: 0.9975 - categorical_accuracy: 0.2723 - f2_score: 0.4775 - f2_score_1: 0.4568 - f2_score_2: 0.4275 - f2_score_3: 0.3947 - f2_score_4: 0.3752 - f2_score_5: 0.3624 - f2_score_6: 0.3008 - f2_score_7: 0.2463 - val_loss: 0.0101 - val_acc: 0.9975 - val_categorical_accuracy: 0.2466 - val_f2_score: 0.4551 - val_f2_score_1: 0.4541 - val_f2_score_2: 0.4380 - val_f2_score_3: 0.4144 - val_f2_score_4: 0.3989 - val_f2_score_5: 0.3875 - val_f2_score_6: 0.3326 - val_f2_score_7: 0.2778
Epoch 18/200
 - 17s - loss: 0.0093 - acc: 0.9975 - categorical_accuracy: 0.2780 - f2_score: 0.4830 - f2_score_1: 0.4630 - f2_score_2: 0.4341 - f2_score_3: 0.4018 - f2_score_4: 0.3822 - f2_score_5: 0.3691 - f2_score_6: 0.3078 - f2_score_7: 0.2528 - val_loss: 0.0101 - val_acc: 0.9975 - val_categorical_accuracy: 0.2310 - val_f2_score: 0.4563 - val_f2_score_1: 0.4546 - val_f2_score_2: 0.4383 - val_f2_score_3: 0.4145 - val_f2_score_4: 0.3983 - val_f2_score_5: 0.3883 - val_f2_score_6: 0.3326 - val_f2_score_7: 0.2776
Epoch 19/200
 - 17s - loss: 0.0091 - acc: 0.9976 - categorical_accuracy: 0.2829 - f2_score: 0.4890 - f2_score_1: 0.4699 - f2_score_2: 0.4413 - f2_score_3: 0.4093 - f2_score_4: 0.3899 - f2_score_5: 0.3765 - f2_score_6: 0.3165 - f2_score_7: 0.2607 - val_loss: 0.0101 - val_acc: 0.9975 - val_categorical_accuracy: 0.2475 - val_f2_score: 0.4574 - val_f2_score_1: 0.4561 - val_f2_score_2: 0.4354 - val_f2_score_3: 0.4093 - val_f2_score_4: 0.3926 - val_f2_score_5: 0.3815 - val_f2_score_6: 0.3224 - val_f2_score_7: 0.2702
Epoch 20/200
 - 16s - loss: 0.0090 - acc: 0.9976 - categorical_accuracy: 0.2920 - f2_score: 0.4952 - f2_score_1: 0.4766 - f2_score_2: 0.4479 - f2_score_3: 0.4174 - f2_score_4: 0.3984 - f2_score_5: 0.3857 - f2_score_6: 0.3239 - f2_score_7: 0.2687 - val_loss: 0.0100 - val_acc: 0.9975 - val_categorical_accuracy: 0.2439 - val_f2_score: 0.4612 - val_f2_score_1: 0.4573 - val_f2_score_2: 0.4405 - val_f2_score_3: 0.4163 - val_f2_score_4: 0.4012 - val_f2_score_5: 0.3896 - val_f2_score_6: 0.3348 - val_f2_score_7: 0.2826
Epoch 21/200
 - 17s - loss: 0.0089 - acc: 0.9976 - categorical_accuracy: 0.2973 - f2_score: 0.5002 - f2_score_1: 0.4822 - f2_score_2: 0.4548 - f2_score_3: 0.4243 - f2_score_4: 0.4055 - f2_score_5: 0.3930 - f2_score_6: 0.3326 - f2_score_7: 0.2766 - val_loss: 0.0101 - val_acc: 0.9975 - val_categorical_accuracy: 0.2424 - val_f2_score: 0.4594 - val_f2_score_1: 0.4613 - val_f2_score_2: 0.4468 - val_f2_score_3: 0.4250 - val_f2_score_4: 0.4109 - val_f2_score_5: 0.4006 - val_f2_score_6: 0.3469 - val_f2_score_7: 0.2945
Epoch 22/200
 - 17s - loss: 0.0088 - acc: 0.9976 - categorical_accuracy: 0.3040 - f2_score: 0.5066 - f2_score_1: 0.4899 - f2_score_2: 0.4625 - f2_score_3: 0.4321 - f2_score_4: 0.4137 - f2_score_5: 0.4014 - f2_score_6: 0.3405 - f2_score_7: 0.2847 - val_loss: 0.0099 - val_acc: 0.9975 - val_categorical_accuracy: 0.2460 - val_f2_score: 0.4629 - val_f2_score_1: 0.4616 - val_f2_score_2: 0.4459 - val_f2_score_3: 0.4230 - val_f2_score_4: 0.4070 - val_f2_score_5: 0.3971 - val_f2_score_6: 0.3442 - val_f2_score_7: 0.2920
Epoch 23/200
 - 17s - loss: 0.0087 - acc: 0.9976 - categorical_accuracy: 0.3117 - f2_score: 0.5126 - f2_score_1: 0.4956 - f2_score_2: 0.4691 - f2_score_3: 0.4391 - f2_score_4: 0.4210 - f2_score_5: 0.4087 - f2_score_6: 0.3491 - f2_score_7: 0.2930 - val_loss: 0.0100 - val_acc: 0.9975 - val_categorical_accuracy: 0.2508 - val_f2_score: 0.4640 - val_f2_score_1: 0.4641 - val_f2_score_2: 0.4477 - val_f2_score_3: 0.4246 - val_f2_score_4: 0.4093 - val_f2_score_5: 0.3992 - val_f2_score_6: 0.3467 - val_f2_score_7: 0.2958
Epoch 24/200
 - 17s - loss: 0.0086 - acc: 0.9976 - categorical_accuracy: 0.3161 - f2_score: 0.5183 - f2_score_1: 0.5029 - f2_score_2: 0.4763 - f2_score_3: 0.4464 - f2_score_4: 0.4275 - f2_score_5: 0.4152 - f2_score_6: 0.3555 - f2_score_7: 0.2997 - val_loss: 0.0100 - val_acc: 0.9975 - val_categorical_accuracy: 0.2521 - val_f2_score: 0.4653 - val_f2_score_1: 0.4656 - val_f2_score_2: 0.4495 - val_f2_score_3: 0.4303 - val_f2_score_4: 0.4165 - val_f2_score_5: 0.4073 - val_f2_score_6: 0.3561 - val_f2_score_7: 0.3048
Epoch 25/200
 - 17s - loss: 0.0085 - acc: 0.9977 - categorical_accuracy: 0.3219 - f2_score: 0.5227 - f2_score_1: 0.5073 - f2_score_2: 0.4813 - f2_score_3: 0.4520 - f2_score_4: 0.4340 - f2_score_5: 0.4216 - f2_score_6: 0.3627 - f2_score_7: 0.3069 - val_loss: 0.0099 - val_acc: 0.9975 - val_categorical_accuracy: 0.2553 - val_f2_score: 0.4662 - val_f2_score_1: 0.4648 - val_f2_score_2: 0.4481 - val_f2_score_3: 0.4253 - val_f2_score_4: 0.4107 - val_f2_score_5: 0.4010 - val_f2_score_6: 0.3496 - val_f2_score_7: 0.3003
Epoch 26/200
 - 17s - loss: 0.0084 - acc: 0.9977 - categorical_accuracy: 0.3297 - f2_score: 0.5288 - f2_score_1: 0.5132 - f2_score_2: 0.4880 - f2_score_3: 0.4594 - f2_score_4: 0.4417 - f2_score_5: 0.4296 - f2_score_6: 0.3710 - f2_score_7: 0.3151 - val_loss: 0.0100 - val_acc: 0.9975 - val_categorical_accuracy: 0.2532 - val_f2_score: 0.4658 - val_f2_score_1: 0.4669 - val_f2_score_2: 0.4537 - val_f2_score_3: 0.4336 - val_f2_score_4: 0.4191 - val_f2_score_5: 0.4094 - val_f2_score_6: 0.3592 - val_f2_score_7: 0.3074
Epoch 27/200
 - 17s - loss: 0.0083 - acc: 0.9977 - categorical_accuracy: 0.3356 - f2_score: 0.5332 - f2_score_1: 0.5184 - f2_score_2: 0.4931 - f2_score_3: 0.4654 - f2_score_4: 0.4481 - f2_score_5: 0.4358 - f2_score_6: 0.3773 - f2_score_7: 0.3210 - val_loss: 0.0099 - val_acc: 0.9975 - val_categorical_accuracy: 0.2603 - val_f2_score: 0.4666 - val_f2_score_1: 0.4666 - val_f2_score_2: 0.4523 - val_f2_score_3: 0.4304 - val_f2_score_4: 0.4169 - val_f2_score_5: 0.4068 - val_f2_score_6: 0.3578 - val_f2_score_7: 0.3101
Epoch 28/200
 - 17s - loss: 0.0082 - acc: 0.9977 - categorical_accuracy: 0.3397 - f2_score: 0.5384 - f2_score_1: 0.5242 - f2_score_2: 0.4994 - f2_score_3: 0.4713 - f2_score_4: 0.4541 - f2_score_5: 0.4426 - f2_score_6: 0.3845 - f2_score_7: 0.3276 - val_loss: 0.0099 - val_acc: 0.9975 - val_categorical_accuracy: 0.2701 - val_f2_score: 0.4667 - val_f2_score_1: 0.4654 - val_f2_score_2: 0.4484 - val_f2_score_3: 0.4267 - val_f2_score_4: 0.4123 - val_f2_score_5: 0.4025 - val_f2_score_6: 0.3524 - val_f2_score_7: 0.3027
Epoch 29/200
 - 17s - loss: 0.0081 - acc: 0.9977 - categorical_accuracy: 0.3462 - f2_score: 0.5435 - f2_score_1: 0.5302 - f2_score_2: 0.5060 - f2_score_3: 0.4778 - f2_score_4: 0.4609 - f2_score_5: 0.4493 - f2_score_6: 0.3916 - f2_score_7: 0.3356 - val_loss: 0.0099 - val_acc: 0.9975 - val_categorical_accuracy: 0.2732 - val_f2_score: 0.4689 - val_f2_score_1: 0.4682 - val_f2_score_2: 0.4531 - val_f2_score_3: 0.4327 - val_f2_score_4: 0.4185 - val_f2_score_5: 0.4083 - val_f2_score_6: 0.3575 - val_f2_score_7: 0.3095
Epoch 30/200
 - 17s - loss: 0.0080 - acc: 0.9977 - categorical_accuracy: 0.3497 - f2_score: 0.5491 - f2_score_1: 0.5359 - f2_score_2: 0.5120 - f2_score_3: 0.4841 - f2_score_4: 0.4670 - f2_score_5: 0.4556 - f2_score_6: 0.3984 - f2_score_7: 0.3432 - val_loss: 0.0099 - val_acc: 0.9975 - val_categorical_accuracy: 0.2630 - val_f2_score: 0.4684 - val_f2_score_1: 0.4696 - val_f2_score_2: 0.4544 - val_f2_score_3: 0.4345 - val_f2_score_4: 0.4215 - val_f2_score_5: 0.4116 - val_f2_score_6: 0.3633 - val_f2_score_7: 0.3159
Epoch 31/200
 - 17s - loss: 0.0080 - acc: 0.9978 - categorical_accuracy: 0.3573 - f2_score: 0.5536 - f2_score_1: 0.5413 - f2_score_2: 0.5177 - f2_score_3: 0.4913 - f2_score_4: 0.4738 - f2_score_5: 0.4626 - f2_score_6: 0.4058 - f2_score_7: 0.3509 - val_loss: 0.0099 - val_acc: 0.9975 - val_categorical_accuracy: 0.2708 - val_f2_score: 0.4693 - val_f2_score_1: 0.4695 - val_f2_score_2: 0.4584 - val_f2_score_3: 0.4397 - val_f2_score_4: 0.4268 - val_f2_score_5: 0.4178 - val_f2_score_6: 0.3711 - val_f2_score_7: 0.3219
Epoch 32/200
 - 17s - loss: 0.0079 - acc: 0.9978 - categorical_accuracy: 0.3646 - f2_score: 0.5584 - f2_score_1: 0.5453 - f2_score_2: 0.5225 - f2_score_3: 0.4959 - f2_score_4: 0.4791 - f2_score_5: 0.4677 - f2_score_6: 0.4119 - f2_score_7: 0.3571 - val_loss: 0.0099 - val_acc: 0.9975 - val_categorical_accuracy: 0.2650 - val_f2_score: 0.4687 - val_f2_score_1: 0.4678 - val_f2_score_2: 0.4551 - val_f2_score_3: 0.4355 - val_f2_score_4: 0.4227 - val_f2_score_5: 0.4127 - val_f2_score_6: 0.3646 - val_f2_score_7: 0.3181
Epoch 33/200
 - 17s - loss: 0.0078 - acc: 0.9978 - categorical_accuracy: 0.3687 - f2_score: 0.5636 - f2_score_1: 0.5512 - f2_score_2: 0.5296 - f2_score_3: 0.5030 - f2_score_4: 0.4870 - f2_score_5: 0.4759 - f2_score_6: 0.4193 - f2_score_7: 0.3647 - val_loss: 0.0099 - val_acc: 0.9975 - val_categorical_accuracy: 0.2722 - val_f2_score: 0.4706 - val_f2_score_1: 0.4700 - val_f2_score_2: 0.4555 - val_f2_score_3: 0.4354 - val_f2_score_4: 0.4218 - val_f2_score_5: 0.4135 - val_f2_score_6: 0.3668 - val_f2_score_7: 0.3209
Epoch 34/200
 - 17s - loss: 0.0077 - acc: 0.9978 - categorical_accuracy: 0.3730 - f2_score: 0.5681 - f2_score_1: 0.5562 - f2_score_2: 0.5338 - f2_score_3: 0.5080 - f2_score_4: 0.4915 - f2_score_5: 0.4808 - f2_score_6: 0.4246 - f2_score_7: 0.3707 - val_loss: 0.0100 - val_acc: 0.9975 - val_categorical_accuracy: 0.2659 - val_f2_score: 0.4681 - val_f2_score_1: 0.4707 - val_f2_score_2: 0.4588 - val_f2_score_3: 0.4402 - val_f2_score_4: 0.4279 - val_f2_score_5: 0.4187 - val_f2_score_6: 0.3724 - val_f2_score_7: 0.3262
Epoch 35/200
 - 17s - loss: 0.0076 - acc: 0.9978 - categorical_accuracy: 0.3774 - f2_score: 0.5721 - f2_score_1: 0.5603 - f2_score_2: 0.5390 - f2_score_3: 0.5130 - f2_score_4: 0.4970 - f2_score_5: 0.4858 - f2_score_6: 0.4313 - f2_score_7: 0.3767 - val_loss: 0.0100 - val_acc: 0.9975 - val_categorical_accuracy: 0.2797 - val_f2_score: 0.4691 - val_f2_score_1: 0.4711 - val_f2_score_2: 0.4574 - val_f2_score_3: 0.4367 - val_f2_score_4: 0.4242 - val_f2_score_5: 0.4149 - val_f2_score_6: 0.3661 - val_f2_score_7: 0.3215
Epoch 00035: early stopping

Model graph loss

In [12]:
sns.set_style("whitegrid")
fig, (ax1, ax2, ax3) = plt.subplots(1, 3, sharex='col', figsize=(20,7))


ax1.plot(history.history['loss'], label='Train loss')
ax1.plot(history.history['val_loss'], label='Validation loss')
ax1.legend(loc='best')
ax1.set_title('Loss')

ax2.plot(history.history['acc'], label='Train Accuracy')
ax2.plot(history.history['val_acc'], label='Validation accuracy')
ax2.legend(loc='best')
ax2.set_title('Accuracy')

ax3.plot(history.history['categorical_accuracy'], label='Train Cat Accuracy')
ax3.plot(history.history['val_categorical_accuracy'], label='Validation Cat Accuracy')
ax3.legend(loc='best')
ax3.set_title('Cat Accuracy')

plt.xlabel('Epochs')
sns.despine()
plt.show()
In [13]:
fig, axes = plt.subplots(4, 2, sharex='col', figsize=(20,7))

axes[0][0].plot(history.history['f2_score'], label='Train F2 Score')
axes[0][0].plot(history.history['val_f2_score'], label='Validation F2 Score')
axes[0][0].legend(loc='best')
axes[0][0].set_title('F2 Score threshold 0.1')

axes[0][1].plot(history.history['f2_score_1'], label='Train F2 Score')
axes[0][1].plot(history.history['val_f2_score_1'], label='Validation F2 Score')
axes[0][1].legend(loc='best')
axes[0][1].set_title('F2 Score threshold 0.15')

axes[1][0].plot(history.history['f2_score_2'], label='Train F2 Score')
axes[1][0].plot(history.history['val_f2_score_2'], label='Validation F2 Score')
axes[1][0].legend(loc='best')
axes[1][0].set_title('F2 Score threshold 0.2')

axes[1][1].plot(history.history['f2_score_3'], label='Train F2 Score')
axes[1][1].plot(history.history['val_f2_score_3'], label='Validation F2 Score')
axes[1][1].legend(loc='best')
axes[1][1].set_title('F2 Score threshold 0.25')

axes[2][0].plot(history.history['f2_score_4'], label='Train F2 Score')
axes[2][0].plot(history.history['val_f2_score_4'], label='Validation F2 Score')
axes[2][0].legend(loc='best')
axes[2][0].set_title('F2 Score threshold 0.28')

axes[2][1].plot(history.history['f2_score_5'], label='Train F2 Score')
axes[2][1].plot(history.history['val_f2_score_5'], label='Validation F2 Score')
axes[2][1].legend(loc='best')
axes[2][1].set_title('F2 Score threshold 0.3')

axes[3][0].plot(history.history['f2_score_6'], label='Train F2 Score')
axes[3][0].plot(history.history['val_f2_score_6'], label='Validation F2 Score')
axes[3][0].legend(loc='best')
axes[3][0].set_title('F2 Score threshold 0.4')

axes[3][1].plot(history.history['f2_score_7'], label='Train F2 Score')
axes[3][1].plot(history.history['val_f2_score_7'], label='Validation F2 Score')
axes[3][1].legend(loc='best')
axes[3][1].set_title('F2 Score threshold 0.5')

plt.xlabel('Epochs')
sns.despine()
plt.show()

Find best threshold value

In [14]:
best_thr = 0
best_thr_val = history.history['val_f2_score'][-1]
for i in range(1, len(metrics)-2):
    if best_thr_val < history.history['val_f2_score_%s' % i][-1]:
        best_thr_val = history.history['val_f2_score_%s' % i][-1]
        best_thr = i

threshold = thresholds[best_thr]
print('Best threshold is: %s' % threshold)
Best threshold is: 0.15

Apply model to test set and output predictions

In [15]:
test_generator.reset()
STEP_SIZE_TEST = test_generator.n//test_generator.batch_size
# Pass the test data through the pre-trained model to extract features
bottleneck_preds = model_vgg.predict_generator(test_generator, steps=STEP_SIZE_TEST)
# Make prediction using the second model
preds = model.predict(bottleneck_preds)
In [16]:
predictions = []
for pred_ar in preds:
    valid = ''
    for idx, pred in enumerate(pred_ar):
        if pred > threshold:
            if len(valid) == 0:
                valid += str(idx)
            else:
                valid += (' %s' % idx)
    if len(valid) == 0:
        valid = str(np.argmax(pred_ar))
    predictions.append(valid)
In [17]:
filenames = test_generator.filenames
results = pd.DataFrame({'id':filenames, 'attribute_ids':predictions})
results['id'] = results['id'].map(lambda x: str(x)[:-4])
results.to_csv('submission.csv',index=False)
results.head(10)
Out[17]:
id attribute_ids
0 10023b2cc4ed5f68 121 223 343 344 369 766 1039 1059
1 100fbe75ed8fd887 121 1039 1059 1085
2 101b627524a04f19 79 304 482 498 703 718 813 961
3 10234480c41284c6 13 51 480 483 725 738 776 830 923 963 1046
4 1023b0e2636dcea8 147 156 283 322 584 813 903 954 1046 1092
5 1039cd6cf85845c 13 405 896 903 1092
6 103a5b3f83fbe88 194 744 813 1092
7 10413aaae8d6a9a2 51 147 813 1092
8 10423822b93a65ab 51 147
9 1052bf702cb099f7 188 597 612 671 723 780