SELU证明过程.pptxVIP

Self-Normalizing Neural NetworksAnd Scaled Exponential Linear UnitsLiu QixuanIntroductionDeep Learning has set new records at different benchmarks CNNs——vision and video tasks RNNs——speech and natural language processingBut FNNs——hard to find success stories, typically shallow, cannotexploit many levels of abstract representationsWhat is SNNs?SNN: Self-Normalizing Neural NetworksHelp to enable high-level abstract representationsThe activation function is “Scaled Exponential Linear Units”(SELUs)Activations close to zero mean and unit variance that are propagated through many network layers will converge towards zero mean and unit variance — even under the presence of noise and perturbations. What can SNNs do?This convergence property of SNNs allows to: train deep networks with many layers employ strong regularization schemes to make learning highly robust A core concept of SNNs?Consider two consecutive layers in a neural network with activation function f The activations x in the lower layer have mean μ and variance νThe activations y = f(wTx) in the higher layer has mean and variance we define n times the mean of the weight vector w Rn as and n times the second moment as There’s a mapping g that maps mean and variance of the activations from one layer the next layer:Normalization techniques ensure a mapping g that keeps (μ, ν) and (, ) close to predefined values, typically (0, 1). Scaled Exponential Linear Units(SELU)Only two design choices are available for adjusting the properties of the function g: (1) the activation function (2) the initialization of the weights. Scaled Exponential Linear Units(SELU)Only two design choices are available for adjusting the properties of the function g: (1) the activation function (2) the initialization of the weights. Scaled Exponential Linear Units(SELU)?For the activation function, we propose “scaled exponential linear units” (SELUs) to render a FNN as self-normalizing.The SELU activation function is given by Deriving the M