基于超分辨率图像的卷积稀疏编码.docVIP

基于超分辨率图像的卷积稀疏编码 EXPLOITING SPARSENESS IN DEEP NEURAL NETWORKS FOR LARGE VOCABULARY SPEECH RECOGNITION Dong Yu 1, Frank Seide 2, Gang Li 2, Li Deng 1 2 Microsoft Research, Redmond, USA Microsoft Research Asia, Beijing, P.R.C 1 {dongyu,fseide,ganl,deng }@ ABSTRACT Recently, we developed context-dependent deep neural network (DNN)hidden Markov models for large vocabulary speech recogni-tion. While reducing errors by 33%compared to its discriminatively trained Gaussian-mixture counterpart on the switchboard benchmark task, DNN requires much more parameters. In this paper, we report our recent work on DNN for improved generalization, model size, and computation speed by exploiting parameter sparseness. We for-mulate the goal of enforcing sparseness as soft regularization and convex constraint optimization problems, and propose solutions un-der the stochastic gradient ascent setting. We also propose novel data structures to exploit the random sparseness patterns to reduce model size and computation time. The proposed solutions have been evaluated on the voice-search and switchboard datasets. They have decreased the number of nonzero connections to one third while re-ducing the error rate by 0.2-0.3%over the fully connected model on both datasets. The nonzero connections have been further reduced to only 12%and 19%on the two respective datasets without sacri?cingspeech recognition performance. Under these conditions we can re-duce the model size to 18%and 29%,and computation to 14%and 23%,respectively, on these two datasets. Index Terms —speech recognition, deep belief networks, deep neural networks, sparseness 1. INTRODUCTION Recently, we have witnessed the resurrection of arti?cialneural net-work (NN)hidden Markov model (HMM)hybrid systems for speech recognition. This mainly attributes to the discovery of the strong modeling ability of deep neural networks (DNNs1) and the availabil-ity of high-speed general purpose graphical processing units (GPG-PUs) for training D