7

I'm training a multi-label classifier that's supposed to be tested on underwater images. I'm wondering if feeding the model drawings of a certain class plus real images can affect the results badly. Was there a study on this? Or are there any past experiences anyone could share to help?

user
  • 101
  • 5
  • 1
    Intuitively, to minimize the biases introduced by the drawings you may want to have your drawings equally distributed among all the different classes that you have. Otherwise the system is likely to focus on the particular traits that constitute a drawing to predict the classes. Maybe applying some kind of style transfer as a preprocessing step to render all your input data alike could be an interesting idea to explore. – Jeanba Jan 10 '20 at 14:16
  • 1
    I think if the drawings are realistic drawings they maybe used for weight pretraining before actual training if the data size is large enough. If the drawings are not realistic (for example maybe some impressionism drawings) then it should not be used. I don't think the drawings should be trained along side the real images as it may disrupt the accuracy and training process – Clement Jan 11 '20 at 04:46
  • @ClementHui When you say weight pretraining, do you mean that I should unfreeze all layers?And by "data size large enough" how much is large enough? – user Jan 11 '20 at 08:11
  • When doing image detection, you should try using one of the available models like ResNet with imageNet weights. Then try training the imageNet model on the drawings but only unfreeze the last few layers. Then you should train on the main dataset. – Clement Jan 11 '20 at 08:17
  • By large enough I mean like 1000/class or more – Clement Jan 11 '20 at 08:18
  • Btw how large is your main dataset size? – Clement Jan 11 '20 at 08:18
  • @ClementHui I'm still working on the dataset, but I have 5000 images total for 20 labels. – user Jan 11 '20 at 08:24
  • @ClementHui Is 1000/class with/without data augmentation? – user Jan 11 '20 at 08:27
  • @ClementHui I should also probably mention that it's a multi-label classification. That probably has totally different requirements – user Jan 11 '20 at 08:29
  • I'm curious to know the results of your efforts, here are a few studies that appear to be in the realm of your need. [first academic link](https://www.researchgate.net/publication/337922291_Analysis_and_Recognition_of_Hand-Drawn_Images_with_Effective_Data_Handling) [second academic link](https://www.researchgate.net/publication/306081553_Classification_of_Photo_and_Sketch_Images_Using_Convolutional_Neural_Networks) [Journal link](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0183838) – CoffeeBaconAddict Dec 30 '20 at 02:34

2 Answers2

0

To my knowledge the deployment model (that you will test on underwater images) as inference will not have a negative effect. Yet drawings may even help differentiate some classes at training and inference. Provided that you won't use drawings in inference, adding them in training phase will not necessarily hurt the accuracy. Note that a drawing of a particular class should not be in the search domains of other classes, namely, a drawing of a particular class should not be the same with the other classes.

bit_scientist
  • 241
  • 1
  • 4
  • 15
  • Though if I were to do it I would use the drawings for pretraining the weights, if the drawings data is big enough and if the drawings are about underwater images as well. – Clement Jan 10 '20 at 02:09
  • @voo_doo I don't quite understand this bit: " a drawing of a particular class should not be the same with the other classes". – user Jan 10 '20 at 07:50
  • @bookfreak the author is saying that there are drawings of a certain class (say, a particular sea-grass). That drawing should not be similar in look to other real underwater images of other classes. If that makes sense... – bit_scientist Jan 10 '20 at 09:26
  • @voo_doo Do you have a link? It would be better to read the whole thing. – user Jan 10 '20 at 09:35
  • @bookfreak I guess you have the data set, post some of the drawings and other underwater images of other classes here as an edit. – bit_scientist Jan 10 '20 at 09:40
0

In general, training on any distribution other that what you are testing on could give worse performance. The model learns to fit the distribution you train it on.

Certainly more data of a sufficiently close distribution will help, e.g. training on Imagenet and testing on CIFAR, but hand-drawings vs natural images seem very different and unless that is what you're wanting to test on, I would imagine it will hurt performance. Sufficiently similar data might include augmentations of the original data, such as crops, flips, blurring etc, as commonly used to effectively enlarge the training set. This is ok as this extra data is effectively "in distribution".

Think of it like this: you are training the model, of finite capacity, to learn your intended distribution plus some info about the extra data that it will never see again, so part of it's modelling capacity is being used up on a pointless task.

Carl
  • 141
  • 3