Easy 71st 0.626 7fold Fastai incepresv2 + resnet50

From: https://www.kaggle.com/dilapsky/easy-71st-0-626-7fold-fastai-incepresv2-resnet50

Author: Dilapsky Lee

Score: 0.626

It's the first time I use fastai in Kaggle competition, I use fastai with 7fold cross validation, and inceptionresnetv2 + resnet50 ensemble. Pretty simple edition, and may seems to be stupid to some experienced Kaggler. This kernel can get both 0.626 in private & public LB, but I didn't select is for my another kernel get 0.627 in public LB and 0.623 in Private LB, which makes me get 71st place.
Well, fastai is easy to use, but it's difficult to get an outstanding result. BTW, thanks for [ods.ai] Alex Lekov for his kernel https://www.kaggle.com/itslek/fastai-resnet50-imet-v4-2 and thanks for Miguel Pinto for his kernel https://www.kaggle.com/mnpinto/imet-fastai-starter

The following part is similar to the kernels mentioned above, you can ignore it

In [1]:
#======================================From here to end just a demo to train the resnet50 models===========================================
#I also trained inceptionresnetv2, with the internel open and use !pip install pretrainedmodels to directly use inceptionresnetv2. 
#I only need the imet pretrained resnet50 models, internel open but cannot submit result is ok for me
import fastai
from fastai.vision import *
fastai.__version__

#Here just for example, use BATCH  = 72 and SIZE   = 320. It's appropriate for pre-trained at first SIZE= 256 and then 288, at last, 320.

BATCH  = 72
SIZE   = 320
path = Path('../input/imet-2019-fgvc6/') # iMet data path

from torch.utils import model_zoo
Path('models').mkdir(exist_ok=True)
!cp '../input/resnet50/resnet50.pth' 'models/'
def load_url(*args, **kwargs):
    model_dir = Path('models')
    filename  = 'resnet50.pth'
    if not (model_dir/filename).is_file(): raise FileNotFoundError
    return torch.load(model_dir/filename)
model_zoo.load_url = load_url

train_df = pd.read_csv(path/'train.csv')
train_df.head()

labels_df = pd.read_csv(path/'labels.csv')
labels_df.head()

test_df = pd.read_csv(path/'sample_submission.csv')
test_df.head()


tfms = get_transforms(do_flip=True, flip_vert=False, max_rotate=0.10, max_zoom=1.5, max_warp=0.2, max_lighting=0.2,
                     xtra_tfms=[(symmetric_warp(magnitude=(-0,0), p=0)),])

train, test = [ImageList.from_df(df, path=path, cols='id', folder=folder, suffix='.png') 
               for df, folder in zip([train_df, test_df], ['train', 'test'])]


class FocalLoss(nn.Module):
    def __init__(self, gamma=2):
        super().__init__()
        self.gamma = gamma

    def forward(self, logit, target):
        target = target.float()
        max_val = (-logit).clamp(min=0)
        loss = logit - logit * target + max_val + \
               ((-max_val).exp() + (-logit - max_val).exp()).log()

        invprobs = F.logsigmoid(-logit * (target * 2.0 - 1.0))
        loss = (invprobs * self.gamma).exp() * loss
        if len(loss.size())==2:
            loss = loss.sum(dim=1)
        return loss.mean()

Here I use 7 fold cross-validation. However, fastai automatically determined the number of class according to training dataset, which will lead to 1102 or 1101, 1100 classes rather than 1103 classes. Because the categories are imbalanced, some of categories only appears once: if it is divided into validation dataset, it will not be recoreded in training dataset. So I use n_splits=15, and find out the divisions that number of classes are 1103 in training dataset, they are: set0, set1, set2, set3, set7, set8 and set10.

In [2]:
from sklearn.model_selection import KFold
import numpy as np
kf = KFold(n_splits=15, random_state=43, shuffle=True)

kf

n = 0
for train_index, test_index in kf.split(train):
    print("TRAIN:", train_index, "TEST:", test_index)
# Change if n == 0 to n == 1,2,3,7,8,10 to use other divisions
    if n == 0:
        break
    n+=1

X_train, X_test = train[train_index], train[test_index]
TRAIN: [     0      1      2      3 ... 109233 109234 109235 109236] TEST: [    12     53     54    105 ... 109138 109151 109229 109231]

Here, for example, train resnet50 for 1 time. In fact, you can train here at least 15 times with resnet50 in 32400 seconds, save the model again, next, load it again and again by learn.load function.

In [3]:
data = (train.split_by_list(X_train,X_test)
        .label_from_df(cols='attribute_ids', label_delim=' ')
        .add_test(test)
        .transform(tfms, size=SIZE, resize_method=ResizeMethod.PAD, padding_mode='border',)
        .databunch(path=Path('.'), bs=BATCH).normalize(imagenet_stats)

       )
print('Data loaded')
learn = cnn_learner(data, base_arch=models.resnet50, loss_func=FocalLoss(), metrics=fbeta,pretrained=True)
#use learn.load function to use your previous trained model, move the model to correct file position by using !cp function
#learn.load('resnet50-0-v4-35')
learn.unfreeze()
learn.fit_one_cycle(1, slice(1e-5,1e-2))
learn.freeze()
learn.save('resnet50-0-v4-1', return_path=True)
#=================================This is the end of pre-trained resnet50 demo====================================================
Data loaded
epoch train_loss valid_loss fbeta time
0 3.554749 207.692154 0.377386 29:52
Out[3]:
PosixPath('models/resnet50-0-v4-1.pth')

The following code is actually what I used in the last submission result file. It's time saving by only loading models and do predictions. In public datasets, I only used less than 1500 seconds to do prediction in 7-fold inceptionresnetv2 and resnet50.

In [4]:
Incepres_BATCH  = 36
Incepres_SIZE   = 320
path = Path('../input/imet-2019-fgvc6/') # iMet data path
In [5]:
Res_BATCH  = 36
Res_SIZE   = 320
In [6]:
train, test = [ImageList.from_df(df, path=path, cols='id', folder=folder, suffix='.png') 
               for df, folder in zip([train_df, test_df], ['train', 'test'])]
Incepres_data = (train.split_by_rand_pct(0.1, seed=42)
        .label_from_df(cols='attribute_ids', label_delim=' ')
        .add_test(test)
        .transform(tfms, size=Incepres_SIZE, resize_method=ResizeMethod.PAD, padding_mode='border',)
        .databunch(path=Path('.'), bs=Incepres_BATCH).normalize(imagenet_stats))
Res_data = (train.split_by_rand_pct(0.1, seed=42)
        .label_from_df(cols='attribute_ids', label_delim=' ')
        .add_test(test)
        .transform(tfms, size=Res_SIZE, resize_method=ResizeMethod.PAD, padding_mode='border',)
        .databunch(path=Path('.'), bs=Res_BATCH).normalize(imagenet_stats))

Here you should copy the key source codes in github repositories in pytorch pretrained-models cadene https://github.com/Cadene/pretrained-models.pytorch and fastai https://github.com/fastai/fastai to avoid using internet

In [7]:
from fastai.vision import learner
In [8]:
# Copied from https://github.com/Cadene/pretrained-models.pytorch, and just use for inceptionresnetv2
from __future__ import print_function, division, absolute_import

import torch

import torch.nn as nn

import torch.utils.model_zoo as model_zoo

import os

import sys



__all__ = ['InceptionResNetV2', 'inceptionresnetv2']



pretrained_settings = {

    'inceptionresnetv2': {

        'imagenet': {

            'url': 'http://data.lip6.fr/cadene/pretrainedmodels/inceptionresnetv2-520b38e4.pth',

            'input_space': 'RGB',

            'input_size': [3, 299, 299],

            'input_range': [0, 1],

            'mean': [0.5, 0.5, 0.5],

            'std': [0.5, 0.5, 0.5],

            'num_classes': 1000

        },

        'imagenet+background': {

            'url': 'http://data.lip6.fr/cadene/pretrainedmodels/inceptionresnetv2-520b38e4.pth',

            'input_space': 'RGB',

            'input_size': [3, 299, 299],

            'input_range': [0, 1],

            'mean': [0.5, 0.5, 0.5],

            'std': [0.5, 0.5, 0.5],

            'num_classes': 1001

        }

    }

}





class BasicConv2d(nn.Module):



    def __init__(self, in_planes, out_planes, kernel_size, stride, padding=0):

        super(BasicConv2d, self).__init__()

        self.conv = nn.Conv2d(in_planes, out_planes,

                              kernel_size=kernel_size, stride=stride,

                              padding=padding, bias=False) # verify bias false

        self.bn = nn.BatchNorm2d(out_planes,

                                 eps=0.001, # value found in tensorflow

                                 momentum=0.1, # default pytorch value

                                 affine=True)

        self.relu = nn.ReLU(inplace=False)



    def forward(self, x):

        x = self.conv(x)

        x = self.bn(x)

        x = self.relu(x)

        return x





class Mixed_5b(nn.Module):



    def __init__(self):

        super(Mixed_5b, self).__init__()



        self.branch0 = BasicConv2d(192, 96, kernel_size=1, stride=1)



        self.branch1 = nn.Sequential(

            BasicConv2d(192, 48, kernel_size=1, stride=1),

            BasicConv2d(48, 64, kernel_size=5, stride=1, padding=2)

        )



        self.branch2 = nn.Sequential(

            BasicConv2d(192, 64, kernel_size=1, stride=1),

            BasicConv2d(64, 96, kernel_size=3, stride=1, padding=1),

            BasicConv2d(96, 96, kernel_size=3, stride=1, padding=1)

        )



        self.branch3 = nn.Sequential(

            nn.AvgPool2d(3, stride=1, padding=1, count_include_pad=False),

            BasicConv2d(192, 64, kernel_size=1, stride=1)

        )



    def forward(self, x):

        x0 = self.branch0(x)

        x1 = self.branch1(x)

        x2 = self.branch2(x)

        x3 = self.branch3(x)

        out = torch.cat((x0, x1, x2, x3), 1)

        return out





class Block35(nn.Module):



    def __init__(self, scale=1.0):

        super(Block35, self).__init__()



        self.scale = scale



        self.branch0 = BasicConv2d(320, 32, kernel_size=1, stride=1)



        self.branch1 = nn.Sequential(

            BasicConv2d(320, 32, kernel_size=1, stride=1),

            BasicConv2d(32, 32, kernel_size=3, stride=1, padding=1)

        )



        self.branch2 = nn.Sequential(

            BasicConv2d(320, 32, kernel_size=1, stride=1),

            BasicConv2d(32, 48, kernel_size=3, stride=1, padding=1),

            BasicConv2d(48, 64, kernel_size=3, stride=1, padding=1)

        )



        self.conv2d = nn.Conv2d(128, 320, kernel_size=1, stride=1)

        self.relu = nn.ReLU(inplace=False)



    def forward(self, x):

        x0 = self.branch0(x)

        x1 = self.branch1(x)

        x2 = self.branch2(x)

        out = torch.cat((x0, x1, x2), 1)

        out = self.conv2d(out)

        out = out * self.scale + x

        out = self.relu(out)

        return out





class Mixed_6a(nn.Module):



    def __init__(self):

        super(Mixed_6a, self).__init__()



        self.branch0 = BasicConv2d(320, 384, kernel_size=3, stride=2)



        self.branch1 = nn.Sequential(

            BasicConv2d(320, 256, kernel_size=1, stride=1),

            BasicConv2d(256, 256, kernel_size=3, stride=1, padding=1),

            BasicConv2d(256, 384, kernel_size=3, stride=2)

        )



        self.branch2 = nn.MaxPool2d(3, stride=2)



    def forward(self, x):

        x0 = self.branch0(x)

        x1 = self.branch1(x)

        x2 = self.branch2(x)

        out = torch.cat((x0, x1, x2), 1)

        return out





class Block17(nn.Module):



    def __init__(self, scale=1.0):

        super(Block17, self).__init__()



        self.scale = scale



        self.branch0 = BasicConv2d(1088, 192, kernel_size=1, stride=1)



        self.branch1 = nn.Sequential(

            BasicConv2d(1088, 128, kernel_size=1, stride=1),

            BasicConv2d(128, 160, kernel_size=(1,7), stride=1, padding=(0,3)),

            BasicConv2d(160, 192, kernel_size=(7,1), stride=1, padding=(3,0))

        )



        self.conv2d = nn.Conv2d(384, 1088, kernel_size=1, stride=1)

        self.relu = nn.ReLU(inplace=False)



    def forward(self, x):

        x0 = self.branch0(x)

        x1 = self.branch1(x)

        out = torch.cat((x0, x1), 1)

        out = self.conv2d(out)

        out = out * self.scale + x

        out = self.relu(out)

        return out





class Mixed_7a(nn.Module):



    def __init__(self):

        super(Mixed_7a, self).__init__()



        self.branch0 = nn.Sequential(

            BasicConv2d(1088, 256, kernel_size=1, stride=1),

            BasicConv2d(256, 384, kernel_size=3, stride=2)

        )



        self.branch1 = nn.Sequential(

            BasicConv2d(1088, 256, kernel_size=1, stride=1),

            BasicConv2d(256, 288, kernel_size=3, stride=2)

        )



        self.branch2 = nn.Sequential(

            BasicConv2d(1088, 256, kernel_size=1, stride=1),

            BasicConv2d(256, 288, kernel_size=3, stride=1, padding=1),

            BasicConv2d(288, 320, kernel_size=3, stride=2)

        )



        self.branch3 = nn.MaxPool2d(3, stride=2)



    def forward(self, x):

        x0 = self.branch0(x)

        x1 = self.branch1(x)

        x2 = self.branch2(x)

        x3 = self.branch3(x)

        out = torch.cat((x0, x1, x2, x3), 1)

        return out





class Block8(nn.Module):



    def __init__(self, scale=1.0, noReLU=False):

        super(Block8, self).__init__()



        self.scale = scale

        self.noReLU = noReLU



        self.branch0 = BasicConv2d(2080, 192, kernel_size=1, stride=1)



        self.branch1 = nn.Sequential(

            BasicConv2d(2080, 192, kernel_size=1, stride=1),

            BasicConv2d(192, 224, kernel_size=(1,3), stride=1, padding=(0,1)),

            BasicConv2d(224, 256, kernel_size=(3,1), stride=1, padding=(1,0))

        )



        self.conv2d = nn.Conv2d(448, 2080, kernel_size=1, stride=1)

        if not self.noReLU:

            self.relu = nn.ReLU(inplace=False)



    def forward(self, x):

        x0 = self.branch0(x)

        x1 = self.branch1(x)

        out = torch.cat((x0, x1), 1)

        out = self.conv2d(out)

        out = out * self.scale + x

        if not self.noReLU:

            out = self.relu(out)

        return out





class InceptionResNetV2(nn.Module):



    def __init__(self, num_classes=1001):

        super(InceptionResNetV2, self).__init__()

        # Special attributs

        self.input_space = None

        self.input_size = (299, 299, 3)

        self.mean = None

        self.std = None

        # Modules

        self.conv2d_1a = BasicConv2d(3, 32, kernel_size=3, stride=2)

        self.conv2d_2a = BasicConv2d(32, 32, kernel_size=3, stride=1)

        self.conv2d_2b = BasicConv2d(32, 64, kernel_size=3, stride=1, padding=1)

        self.maxpool_3a = nn.MaxPool2d(3, stride=2)

        self.conv2d_3b = BasicConv2d(64, 80, kernel_size=1, stride=1)

        self.conv2d_4a = BasicConv2d(80, 192, kernel_size=3, stride=1)

        self.maxpool_5a = nn.MaxPool2d(3, stride=2)

        self.mixed_5b = Mixed_5b()

        self.repeat = nn.Sequential(

            Block35(scale=0.17),

            Block35(scale=0.17),

            Block35(scale=0.17),

            Block35(scale=0.17),

            Block35(scale=0.17),

            Block35(scale=0.17),

            Block35(scale=0.17),

            Block35(scale=0.17),

            Block35(scale=0.17),

            Block35(scale=0.17)

        )

        self.mixed_6a = Mixed_6a()

        self.repeat_1 = nn.Sequential(

            Block17(scale=0.10),

            Block17(scale=0.10),

            Block17(scale=0.10),

            Block17(scale=0.10),

            Block17(scale=0.10),

            Block17(scale=0.10),

            Block17(scale=0.10),

            Block17(scale=0.10),

            Block17(scale=0.10),

            Block17(scale=0.10),

            Block17(scale=0.10),

            Block17(scale=0.10),

            Block17(scale=0.10),

            Block17(scale=0.10),

            Block17(scale=0.10),

            Block17(scale=0.10),

            Block17(scale=0.10),

            Block17(scale=0.10),

            Block17(scale=0.10),

            Block17(scale=0.10)

        )

        self.mixed_7a = Mixed_7a()

        self.repeat_2 = nn.Sequential(

            Block8(scale=0.20),

            Block8(scale=0.20),

            Block8(scale=0.20),

            Block8(scale=0.20),

            Block8(scale=0.20),

            Block8(scale=0.20),

            Block8(scale=0.20),

            Block8(scale=0.20),

            Block8(scale=0.20)

        )

        self.block8 = Block8(noReLU=True)

        self.conv2d_7b = BasicConv2d(2080, 1536, kernel_size=1, stride=1)

        self.avgpool_1a = nn.AvgPool2d(8, count_include_pad=False)

        self.last_linear = nn.Linear(1536, num_classes)



    def features(self, input):

        x = self.conv2d_1a(input)

        x = self.conv2d_2a(x)

        x = self.conv2d_2b(x)

        x = self.maxpool_3a(x)

        x = self.conv2d_3b(x)

        x = self.conv2d_4a(x)

        x = self.maxpool_5a(x)

        x = self.mixed_5b(x)

        x = self.repeat(x)

        x = self.mixed_6a(x)

        x = self.repeat_1(x)

        x = self.mixed_7a(x)

        x = self.repeat_2(x)

        x = self.block8(x)

        x = self.conv2d_7b(x)

        return x



    def logits(self, features):

        x = self.avgpool_1a(features)

        x = x.view(x.size(0), -1)

        x = self.last_linear(x)

        return x



    def forward(self, input):

        x = self.features(input)

        x = self.logits(x)

        return x



def inceptionresnetv2(num_classes=1000, pretrained='imagenet'):

    r"""InceptionResNetV2 model architecture from the

    `"InceptionV4, Inception-ResNet..." <https://arxiv.org/abs/1602.07261>`_ paper.

    """

    if pretrained:

        settings = pretrained_settings['inceptionresnetv2'][pretrained]

        assert num_classes == settings['num_classes'],"num_classes should be {}, but is {}".format(settings['num_classes'], num_classes)



        # both 'imagenet'&'imagenet+background' are loaded from same parameters

        model = InceptionResNetV2(num_classes=1001)

        model.load_state_dict(model_zoo.load_url(settings['url']))



        if pretrained == 'imagenet':

            new_last_linear = nn.Linear(1536, 1000)

            new_last_linear.weight.data = model.last_linear.weight.data[1:]

            new_last_linear.bias.data = model.last_linear.bias.data[1:]

            model.last_linear = new_last_linear



        model.input_space = settings['input_space']

        model.input_size = settings['input_size']

        model.input_range = settings['input_range']



        model.mean = settings['mean']

        model.std = settings['std']

    else:

        model = InceptionResNetV2(num_classes=num_classes)

    return model



'''

TEST

Run this code with:

```

cd $HOME/pretrained-models.pytorch

python -m pretrainedmodels.inceptionresnetv2

```

'''
# Comment these code, we will not use them

#if __name__ == '__main__':



 #   assert inceptionresnetv2(num_classes=10, pretrained=None)

  #  print('success')

   # assert inceptionresnetv2(num_classes=1000, pretrained='imagenet')

    #print('success')

   # assert inceptionresnetv2(num_classes=1001, pretrained='imagenet+background')

    #print('success')



    # fail

   # assert inceptionresnetv2(num_classes=1001, pretrained='imagenet')
Out[8]:
'\n\nTEST\n\nRun this code with:\n\n```\n\ncd $HOME/pretrained-models.pytorch\n\npython -m pretrainedmodels.inceptionresnetv2\n\n```\n\n'
In [9]:
# copied from https://github.com/fastai/fastai/blob/master/fastai/vision/models/cadene_models.py
def get_incepres_model(model_name:str, pretrained:bool, seq:bool=False, pname:str='imagenet', **kwargs):

    pretrained = pname if pretrained else None

    model = inceptionresnetv2(pretrained=pretrained, **kwargs)

    return nn.Sequential(*model.children()) if seq else model
In [10]:
def myinceptionresnetv2(pretrained:bool=False):  
    return get_incepres_model('inceptionresnetv2', pretrained, seq=True)


learner.model_meta[myinceptionresnetv2] = {'cut': -2, 'split': lambda m: (m[0][9],     m[1])}
In [11]:

In [11]:
#!ls ../input/15foldincepres/15foldincepres.pth
In [12]:
!ls ../input/incepres15fold320 
inceptionres-0-v4-52.pth   inceptionres-2-v4-52.pth    inceptionres-8-v4-60.pth
inceptionres-1-v4-54.pth   inceptionres-3-v4-49-n.pth
inceptionres-10-v4-48.pth  inceptionres-7-v4-60-n.pth

The following code just show the prediction matrix, in fastai, don't use learn.TTA for it required 8 times of running time!

In [13]:
#cv0.593
!cp ../input/incepres15fold320/inceptionres-0-v4-52.pth ./models/
Incepres_learn0 = cnn_learner(Incepres_data, base_arch=myinceptionresnetv2, loss_func=FocalLoss(), metrics=fbeta,pretrained=False)
Incepres_learn0.load('inceptionres-0-v4-52')
#test_preds = learn0.TTA(ds_type=DatasetType.Test)
Incepres_test_preds0 = Incepres_learn0.get_preds(ds_type=DatasetType.Test)
Incepres_test_preds0
Out[13]:
[tensor([[-4.2653, -3.4151, -4.6208,  ..., -3.5328, -3.0117, -3.9509],
         [-6.3921, -5.0922, -7.0856,  ..., -6.5807, -6.7583, -6.2641],
         [-5.0617, -4.7151, -5.1977,  ..., -4.5004, -4.3320, -2.7067],
         ...,
         [-5.0293, -4.5195, -5.8817,  ..., -5.1677, -5.0355, -3.2838],
         [-5.3900, -4.5209, -5.6121,  ..., -4.6059, -3.8802, -4.1440],
         [-5.6908, -3.6801, -5.7569,  ..., -5.2952, -4.7368, -4.5506]]),
 tensor([0, 0, 0,  ..., 0, 0, 0])]
In [14]:
#cv0.597
!cp ../input/incepres15fold320/inceptionres-1-v4-54.pth ./models/
Incepres_learn1 = cnn_learner(Incepres_data, base_arch=myinceptionresnetv2, loss_func=FocalLoss(), metrics=fbeta,pretrained=False)
Incepres_learn1.load('inceptionres-1-v4-54')
#test_preds = learn0.TTA(ds_type=DatasetType.Test)
Incepres_test_preds1 = Incepres_learn1.get_preds(ds_type=DatasetType.Test)
Incepres_test_preds1
Out[14]:
[tensor([[-4.3454, -3.9037, -4.7900,  ..., -4.1699, -3.2154, -4.2664],
         [-6.0941, -4.9309, -7.1140,  ..., -6.1242, -6.2596, -6.0142],
         [-3.9473, -3.4166, -3.9384,  ..., -3.4240, -2.7817, -1.7560],
         ...,
         [-5.5488, -4.8801, -6.7674,  ..., -6.2311, -5.4078, -5.0420],
         [-5.6256, -4.5541, -6.0245,  ..., -5.1311, -3.9012, -3.9199],
         [-5.3555, -3.8322, -5.7863,  ..., -5.0265, -4.3899, -4.1331]]),
 tensor([0, 0, 0,  ..., 0, 0, 0])]
In [15]:
#cv0.588
!cp ../input/incepres15fold320/inceptionres-2-v4-52.pth ./models/
Incepres_learn2 = cnn_learner(Incepres_data, base_arch=myinceptionresnetv2, loss_func=FocalLoss(), metrics=fbeta,pretrained=False)
Incepres_learn2.load('inceptionres-2-v4-52')
#test_preds = learn0.TTA(ds_type=DatasetType.Test)
Incepres_test_preds2 = Incepres_learn2.get_preds(ds_type=DatasetType.Test)
Incepres_test_preds2
Out[15]:
[tensor([[ -4.1277,  -3.6498,  -4.5712,  ...,  -3.4741,  -2.7750,  -3.6319],
         [ -5.8772,  -5.1386,  -6.5323,  ...,  -6.0434,  -6.1805,  -5.9441],
         [ -4.7086,  -4.0691,  -4.7429,  ...,  -4.1741,  -4.0316,  -2.4467],
         ...,
         [-15.2086, -12.0071,  -7.4604,  ...,   0.1399,  -8.6327,   2.0022],
         [ -4.8732,  -3.9241,  -4.5586,  ...,  -4.3891,  -3.4916,  -3.7974],
         [ -5.5192,  -5.0569,  -6.4683,  ...,  -5.7673,  -4.9811,  -4.2743]]),
 tensor([0, 0, 0,  ..., 0, 0, 0])]
In [16]:
#cv0.597
!cp ../input/incepres15fold320/inceptionres-8-v4-60.pth ./models/
Incepres_learn8 = cnn_learner(Incepres_data, base_arch=myinceptionresnetv2, loss_func=FocalLoss(), metrics=fbeta,pretrained=False)
Incepres_learn8.load('inceptionres-8-v4-60')
#test_preds = learn0.TTA(ds_type=DatasetType.Test)
Incepres_test_preds8 = Incepres_learn8.get_preds(ds_type=DatasetType.Test)
Incepres_test_preds8
Out[16]:
[tensor([[-4.4668, -4.0680, -4.4893,  ..., -3.5649, -2.8830, -3.5489],
         [-6.4138, -5.6419, -6.3109,  ..., -6.3615, -6.5848, -5.5644],
         [-3.7073, -3.3216, -3.6209,  ..., -2.9989, -2.8609, -1.9497],
         ...,
         [-4.0217, -2.5677, -5.2693,  ..., -4.6531, -4.0974, -4.2806],
         [-4.4224, -3.9210, -5.5461,  ..., -4.5090, -3.5095, -3.8008],
         [-5.3401, -4.8493, -6.2832,  ..., -5.1439, -5.1450, -5.0090]]),
 tensor([0, 0, 0,  ..., 0, 0, 0])]
In [17]:
#cv0.588
!cp ../input/incepres15fold320/inceptionres-3-v4-49-n.pth ./models/
Incepres_learn3 = cnn_learner(Incepres_data, base_arch=myinceptionresnetv2, loss_func=FocalLoss(), metrics=fbeta,pretrained=False)
Incepres_learn3.load('inceptionres-3-v4-49-n')
#test_preds = learn0.TTA(ds_type=DatasetType.Test)
Incepres_test_preds3 = Incepres_learn3.get_preds(ds_type=DatasetType.Test)
Incepres_test_preds3
Out[17]:
[tensor([[-4.5005, -4.4327, -5.8948,  ..., -4.3480, -3.7923, -3.9953],
         [-6.1613, -4.9657, -6.6781,  ..., -5.5273, -6.0323, -5.6275],
         [-4.4445, -4.4701, -5.4344,  ..., -4.8897, -4.3574, -2.7295],
         ...,
         [-5.2870, -5.1486, -6.3762,  ..., -6.0172, -6.1878, -4.9177],
         [-4.9963, -4.6179, -5.6261,  ..., -5.0932, -4.1407, -3.8624],
         [-5.3463, -4.3458, -5.8687,  ..., -4.9305, -4.3687, -3.7193]]),
 tensor([0, 0, 0,  ..., 0, 0, 0])]
In [18]:
#cv0.588
!cp ../input/incepres15fold320/inceptionres-7-v4-60-n.pth ./models/
Incepres_learn7 = cnn_learner(Incepres_data, base_arch=myinceptionresnetv2, loss_func=FocalLoss(), metrics=fbeta,pretrained=False)
Incepres_learn7.load('inceptionres-7-v4-60-n')
#test_preds = learn0.TTA(ds_type=DatasetType.Test)
Incepres_test_preds7 = Incepres_learn7.get_preds(ds_type=DatasetType.Test)
Incepres_test_preds7
Out[18]:
[tensor([[ -4.0239,  -3.7443,  -4.7036,  ...,  -3.5010,  -2.9975,  -3.7000],
         [ -6.3185,  -5.8634,  -6.5709,  ...,  -6.0478,  -5.9333,  -5.5359],
         [ -3.5450,  -3.7113,  -3.4057,  ...,  -3.3502,  -3.2449,  -2.4986],
         ...,
         [-14.0073,  -7.3866, -10.7455,  ..., -12.1249,  -7.0240,  -4.6256],
         [ -4.8225,  -4.0089,  -4.9977,  ...,  -4.7980,  -3.7673,  -3.8326],
         [ -5.3788,  -4.2490,  -5.1733,  ...,  -4.5405,  -4.4936,  -4.4176]]),
 tensor([0, 0, 0,  ..., 0, 0, 0])]
In [19]:
#Res_test_preds0
In [20]:
#cv0.594
!cp ../input/incepres15fold320/inceptionres-10-v4-48.pth ./models/
Incepres_learn10 = cnn_learner(Incepres_data, base_arch=myinceptionresnetv2, loss_func=FocalLoss(), metrics=fbeta,pretrained=False)
Incepres_learn10.load('inceptionres-10-v4-48')
#test_preds = learn0.TTA(ds_type=DatasetType.Test)
Incepres_test_preds10 = Incepres_learn10.get_preds(ds_type=DatasetType.Test)
Incepres_test_preds10
Out[20]:
[tensor([[-3.8995, -3.5324, -4.4256,  ..., -3.4903, -2.8345, -3.6886],
         [-5.6687, -4.2606, -6.5392,  ..., -5.9216, -5.6602, -5.2519],
         [-4.3911, -3.8050, -4.5347,  ..., -3.8015, -3.5835, -2.4617],
         ...,
         [-5.7678, -3.9385, -5.8303,  ..., -4.9946, -4.8488, -3.6600],
         [-4.4675, -3.5440, -5.0018,  ..., -4.0624, -3.2179, -3.3255],
         [-4.8368, -3.9792, -5.3259,  ..., -4.4560, -4.0509, -3.8415]]),
 tensor([0, 0, 0,  ..., 0, 0, 0])]
In [21]:
!ls ../input/resnet50imet15fold320
resnet50-0-v4-597.pth	resnet50-2-v4-597.pth  resnet50-8-v4-603.pth
resnet50-1-v4-603.pth	resnet50-3-v4-597.pth
resnet50-10-v4-599.pth	resnet50-7-v4-597.pth
In [22]:
#!ls ../input/resnet50-imet-15fold/resnet50-0-v4-35.pth
!cp ../input/resnet50imet15fold320/resnet50-0-v4-597.pth ./models/
Res_learn0 = cnn_learner(Res_data, base_arch=models.resnet50, loss_func=FocalLoss(), metrics=fbeta,pretrained=False)
Res_learn0.load('resnet50-0-v4-597')

Res_test_preds0 = Res_learn0.get_preds(ds_type=DatasetType.Test)
Res_test_preds0
Out[22]:
[tensor([[-4.5946, -4.5987, -5.3722,  ..., -4.5365, -3.4758, -4.6496],
         [-8.3546, -7.0035, -7.3729,  ..., -7.7271, -8.0781, -7.6059],
         [-3.3494, -3.6260, -4.6326,  ..., -4.1942, -3.8994, -1.5787],
         ...,
         [-6.0897, -5.3472, -7.4615,  ..., -5.5373, -5.8681, -5.2057],
         [-4.6906, -4.0183, -5.5341,  ..., -4.5959, -3.4451, -3.5317],
         [-5.9893, -5.1343, -7.0927,  ..., -6.4337, -5.6958, -4.6717]]),
 tensor([0, 0, 0,  ..., 0, 0, 0])]
In [23]:
!cp ../input/resnet50imet15fold320/resnet50-1-v4-603.pth ./models/
Res_learn1 = cnn_learner(Res_data, base_arch=models.resnet50, loss_func=FocalLoss(), metrics=fbeta,pretrained=False)
Res_learn1.load('resnet50-1-v4-603')

Res_test_preds1 = Res_learn1.get_preds(ds_type=DatasetType.Test)
Res_test_preds1
Out[23]:
[tensor([[-4.7588, -4.3074, -5.2228,  ..., -4.4323, -3.8511, -4.4131],
         [-6.6152, -6.2209, -7.6508,  ..., -6.4438, -6.9695, -6.5389],
         [-4.0976, -4.1697, -4.7361,  ..., -4.0613, -3.8613, -2.6070],
         ...,
         [-4.9224, -5.0643, -6.8925,  ..., -5.1667, -5.6924, -4.4804],
         [-4.6672, -3.9503, -4.6644,  ..., -4.0373, -3.1628, -3.3692],
         [-5.6984, -4.6895, -7.2736,  ..., -5.9359, -5.5464, -4.5327]]),
 tensor([0, 0, 0,  ..., 0, 0, 0])]
In [24]:
!cp ../input/resnet50imet15fold320/resnet50-2-v4-597.pth ./models/
Res_learn2 = cnn_learner(Res_data, base_arch=models.resnet50, loss_func=FocalLoss(), metrics=fbeta,pretrained=False)
Res_learn2.load('resnet50-2-v4-597')

Res_test_preds2 = Res_learn2.get_preds(ds_type=DatasetType.Test)
Res_test_preds2
Out[24]:
[tensor([[-5.1483, -4.5074, -5.9678,  ..., -5.0727, -4.1522, -4.1906],
         [-7.3275, -5.9644, -8.1587,  ..., -7.2860, -7.2181, -6.4895],
         [-4.8271, -4.2763, -5.6263,  ..., -5.2464, -4.4075, -2.0507],
         ...,
         [-5.9147, -4.6364, -7.4401,  ..., -5.8949, -6.2432, -5.1947],
         [-5.0750, -3.8342, -6.2827,  ..., -5.1197, -4.0181, -3.8616],
         [-5.7002, -4.7304, -6.3135,  ..., -5.8935, -5.0948, -3.8171]]),
 tensor([0, 0, 0,  ..., 0, 0, 0])]
In [25]:
!cp ../input/resnet50imet15fold320/resnet50-3-v4-597.pth ./models/
Res_learn3 = cnn_learner(Res_data, base_arch=models.resnet50, loss_func=FocalLoss(), metrics=fbeta,pretrained=False)
Res_learn3.load('resnet50-3-v4-597')

Res_test_preds3 = Res_learn3.get_preds(ds_type=DatasetType.Test)
Res_test_preds3
Out[25]:
[tensor([[-5.3466, -4.9569, -5.6028,  ..., -5.1744, -4.5988, -4.3220],
         [-6.6653, -5.4703, -7.5373,  ..., -6.3965, -6.7625, -5.3468],
         [-4.4287, -4.2409, -4.8829,  ..., -4.4367, -3.9345, -2.4024],
         ...,
         [-7.6790, -5.5326, -7.3628,  ..., -6.8827, -5.8634, -5.0110],
         [-5.3564, -4.8268, -5.4602,  ..., -5.0245, -3.8035, -3.7071],
         [-7.4734, -4.9104, -7.0449,  ..., -7.0032, -5.7598, -5.2106]]),
 tensor([0, 0, 0,  ..., 0, 0, 0])]
In [26]:
!cp ../input/resnet50imet15fold320/resnet50-7-v4-597.pth ./models/
Res_learn7 = cnn_learner(Res_data, base_arch=models.resnet50, loss_func=FocalLoss(), metrics=fbeta,pretrained=False)
Res_learn7.load('resnet50-7-v4-597')

Res_test_preds7 = Res_learn7.get_preds(ds_type=DatasetType.Test)
Res_test_preds7
Out[26]:
[tensor([[-4.5409, -4.4854, -5.8856,  ..., -5.3325, -4.1130, -4.2562],
         [-7.6093, -5.3958, -8.5099,  ..., -7.6233, -7.4503, -7.8359],
         [-4.0965, -3.7665, -4.6047,  ..., -4.5038, -4.0765, -1.9154],
         ...,
         [-7.3482, -4.9608, -8.1450,  ..., -6.7521, -6.9874, -5.5544],
         [-4.9209, -3.8275, -5.5425,  ..., -4.5357, -3.3131, -3.3121],
         [-5.6724, -4.1183, -6.0379,  ..., -5.9367, -5.4822, -4.7210]]),
 tensor([0, 0, 0,  ..., 0, 0, 0])]
In [27]:
!cp ../input/resnet50imet15fold320/resnet50-8-v4-603.pth ./models/
Res_learn8 = cnn_learner(Res_data, base_arch=models.resnet50, loss_func=FocalLoss(), metrics=fbeta,pretrained=False)
Res_learn8.load('resnet50-8-v4-603')

Res_test_preds8 = Res_learn8.get_preds(ds_type=DatasetType.Test)
Res_test_preds8
Out[27]:
[tensor([[-4.9691, -5.2152, -5.8360,  ..., -5.4668, -4.1740, -5.0228],
         [-7.4000, -8.1185, -8.7003,  ..., -8.2810, -8.4258, -7.6193],
         [-5.3930, -4.9971, -5.4428,  ..., -5.0778, -4.0536, -2.5628],
         ...,
         [-6.3091, -5.8759, -7.5705,  ..., -5.4380, -6.7409, -5.0636],
         [-5.1397, -4.5639, -5.4742,  ..., -5.3896, -3.9206, -3.9573],
         [-6.4764, -5.7999, -7.1607,  ..., -7.0257, -5.7520, -5.0876]]),
 tensor([0, 0, 0,  ..., 0, 0, 0])]
In [28]:
!cp ../input/resnet50imet15fold320/resnet50-10-v4-599.pth ./models/
Res_learn10 = cnn_learner(Res_data, base_arch=models.resnet50, loss_func=FocalLoss(), metrics=fbeta,pretrained=False)
Res_learn10.load('resnet50-10-v4-599')

Res_test_preds10 = Res_learn10.get_preds(ds_type=DatasetType.Test)
Res_test_preds10
Out[28]:
[tensor([[-4.8733, -4.6467, -6.1021,  ..., -5.0370, -4.2385, -4.2170],
         [-7.6408, -5.6151, -8.2164,  ..., -7.2314, -6.8521, -7.2054],
         [-4.5285, -3.4359, -4.7015,  ..., -4.3465, -3.7947, -2.6366],
         ...,
         [-4.8692, -3.3189, -6.8460,  ..., -5.6447, -4.4224, -4.8190],
         [-4.6620, -3.8545, -5.5461,  ..., -4.6140, -3.5286, -3.5007],
         [-5.2578, -4.1065, -5.2309,  ..., -5.2094, -4.6928, -4.2767]]),
 tensor([0, 0, 0,  ..., 0, 0, 0])]
In [29]:
#import numpy as np
#np.shape(np.array(Res_test_preds0[0]))
In [30]:
#Here use take average of everything, or you can appoint the weight, it's simple
test_preds = ((Incepres_test_preds0[0]*1+Incepres_test_preds1[0]*1.1
               +Incepres_test_preds2[0]*0.9+Incepres_test_preds3[0]*0.9
               +Incepres_test_preds7[0]*0.9+Incepres_test_preds8[0]*1.1
               +Incepres_test_preds10[0]*1.1)/14+(Res_test_preds0[0]*1+Res_test_preds1[0]*1
               +Res_test_preds2[0]*1+Res_test_preds3[0]*1
               +Res_test_preds7[0]*1+Res_test_preds8[0]*1
               +Res_test_preds10[0]*1)/14,Incepres_test_preds1[1])
#test_preds = ((test_preds0[0]+test_preds1[0]+test_preds2[0]+test_preds3[0]+test_preds7[0]+test_preds8[0]+test_preds10[0])/7,test_preds1[1])
In [31]:
i2c = np.array([[i, c] for c, i in Incepres_learn0.data.train_ds.y.c2i.items()]).astype(int)
def join_preds(preds, thr):
    return [' '.join(i2c[np.where(t==1)[0],1].astype(str)) for t in (preds[0].sigmoid()>thr).long()]
In [32]:
# Most of models get good result around 0.260, 0.270 is also an applicable threshold
test_df.attribute_ids = join_preds(test_preds, 0.260)
test_df.head()
Out[32]:
id attribute_ids
0 1002e7c8e9172e5c 1062 1092 584 671 738 79 813
1 10045628394b0c85 1092 51 718 813 903
2 1006c05aacbeaea6 1035 147 639 671 780 950
3 10073dbee72a9e88 121 464
4 10077d92747afb1a 1084 147 156 189 259 462 501 519 580 670 733 7...
In [33]:
test_df.to_csv('submission.csv', index=False)