Statistical machine learning using deep neural network (DNN) has demonstrated high classification accuracy on complex inputs in many application domains. Motivated by the tremendous success of DNNs in computer vision Krizhevsky2012 ; DBLP:journals/corr/HeZRS15 ; szegedy2015going there is a growing interest in employing DNNs in autonomous systems interacting with physical worlds, such as unmanned vehicles and robotics. However, these applications require DNNs to operate on “novel” conditions with perturbed inputs. For example, an autonomous vehicle needs to make reliable classifications even with noisy sensor data (stochastic perturbations) or under inclement weather conditions (structured perturbations).

For convolutional neural networks that depend on local feature extraction, perturbation of pixel level information can cause kernels to map features into incorrect vectors. When such error propagates along the depth of network, resulting classification accuracy is affected Nazar2017DeepCN . It is feasible to tolerate a certain type of perturbations by training the networks on equivalent perturbations, but it is practically intractable to anticipate all possible sources of perturbations during training. Moreover, training to improve accuracy under perturbed images reduces accuracy for clean images na2019mixture . Therefore, a new class of DNN architecture that are inherently resilient to input perturbations without prior training is necessary for autonomous applications.

Human brain, on the other hand, can provide accurate classification even under input perturbations. The neuro-inspired computing using Spiking neural network (SNN) has been actively researched for its potential to gain better understanding of biological neural systems and to achieve artificial neural network with learning ability similar to human brain Javed2010 . The event based operation principle of SNN maass1997networks ; izhikevich2008large differentiates itself from conventional deep learning which utilizes gradient descent. Biologically plausible neuron and synapse models used in SNN makes it possible to exploit temporal relationship between spiking events and optimize network parameters based on causality information Moreno-Bote2015 ; lansdell2019spiking , which can not be achieved with traditional DNNs. SNN with temporally dependent plastic learning rule has been developed to for image classification purposes. Works have shown network designs for classifying images with both simple Diehl2015 ; She2019FastAndLow ; Querlioz2013 ; Srinivasan2016 and complex  tavanaei2018deep ; lee2018deep ; tavanaei2016bio features. However, the classification accuracy using unsupervised learning with SNN alone is still far from what is achievable with state-of-the-art DNNs.

This paper presents a hybrid network architecture and learning methodology that couples causal learning of SNN with supervised training based classification using DNN. Our hypothesis is that input perturbation degrades feature quality locally, but at the global level, information from the entire input image is better preserved even under local perturbation. This global level information, if correctly extracted, can be used as contextual information to assist the classification task. We present an implementation of SNN for contextual information extraction and integrate that with a conventional DNN backbone creating a hybrid network. The entire architecture, called ScieNet, achieves more robust image classification under stochastic (noisy) and structured (rainy) perturbation of the input. Specifically, this paper makes the following key contributions:

We present a hybrid deep learning architecture with SNN based network components that pre-learn input contextual information without supervision and utilize it in DNN image classification.

We demonstrate that the proposed architecture is resilient to input perturbation while requiring no prior knowledge of the perturbation during training and inference. Moreover, as the network is never trained with perturbed inputs, the performance of the proposed network for clean images is not affected. We demonstrate resilience to stochastic (noisy) and structured (rain) input perturbations.

We show that ScieNet is a versatile design that can be easily integrated with different back-end deep learning architectures.

We demonstrate ScieNet with three different back-end DNN architectures, namely, MobileNetV2, ResNet101 and DenseNet, for image classification using the CIFAR10 dataset. The experiments are peformed considering Gaussian noise and rain (synthesized using liu2018erase ; Garg:2006:PRR:1141911.1141985 and practical) added to the images during inference. All versions of ScieNet shows signifcant improvement in accuracy under noise and rain when compared to the respective backbone DNNs in isolation.

The impact of noise and rain on image classification has received significant interest in recent years. Nazar Nazar2017DeepCN and Luo luo2014deep shows that noise in inference images causes degradation of image classification performance of DNN.

But majority of the past approaches focused on using either additional training with noisy/rainy images to improve accuracy. Solutions that have been proposed include training with dataset containing noise Nazar2017DeepCN and manually introducing noise to network parameters luo2014deep . Such approaches are able to improve accuracy when noise pattern used in training is similar to that in inference, but negatively affect network performance on clean dataset.

A parallel approach is to develop specialized pre-processing network, often using another DNN trained using noisy/rainy images, to de-noise or de-rain the input images before classification. For example, Na et. al. have demonstrated pre-processing network to improve classification accuracy under noise na2018noise . A recent paper by Liu liu2018erase shows an machine learning approach to remove rain drops from images.