Cooperative initialization based deep neural network training

Pravendra Singh, Munender Varshney, Vinay P. Namboodiri

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Researchers have proposed various activation functions. These activation functions help the deep network to learn non-linear behavior with a significant effect on training dynamics and task performance. The performance of these activations also depends on the initial state of the weight parameters, i.e., different initial state leads to a difference in the performance of a network. In this paper, we have proposed a cooperative initialization for training the deep network using ReLU activation function to improve the network performance. Our approach uses multiple activation functions in the initial few epochs for the update of all sets of weight parameters while training the network. These activation functions cooperate to overcome their drawbacks in the update of weight parameters, which in effect learn better "feature representation" and boost the network performance later. Cooperative initialization based training also helps in reducing the overfitting problem and does not increase the number of parameters, inference (test) time in the final model while improving the performance. Experiments show that our approach outperforms various baselines and, at the same time, performs well over various tasks such as classification and detection. The Top-1 classification accuracy of the model trained using our approach improves by 2.8% for VGG-16 and 2.1% for ResNet-56 on CIFAR-100 dataset.

Original languageEnglish
Title of host publicationProceedings - 2020 IEEE Winter Conference on Applications of Computer Vision, WACV 2020
PublisherIEEE
Pages1130-1139
Number of pages10
ISBN (Electronic)9781728165530
DOIs
Publication statusPublished - Mar 2020
Event2020 IEEE/CVF Winter Conference on Applications of Computer Vision, WACV 2020 - Snowmass Village, USA United States
Duration: 1 Mar 20205 Mar 2020

Publication series

NameProceedings - 2020 IEEE Winter Conference on Applications of Computer Vision, WACV 2020

Conference

Conference2020 IEEE/CVF Winter Conference on Applications of Computer Vision, WACV 2020
CountryUSA United States
CitySnowmass Village
Period1/03/205/03/20

ASJC Scopus subject areas

  • Computer Science Applications
  • Computer Vision and Pattern Recognition

Cite this

Singh, P., Varshney, M., & Namboodiri, V. P. (2020). Cooperative initialization based deep neural network training. In Proceedings - 2020 IEEE Winter Conference on Applications of Computer Vision, WACV 2020 (pp. 1130-1139). [9093378] (Proceedings - 2020 IEEE Winter Conference on Applications of Computer Vision, WACV 2020). IEEE. https://doi.org/10.1109/WACV45572.2020.9093378