人工智能课件AI2014S_Chap04LinearModelsforRegression幻灯片.pptxVIP

Artificial Intelligence Linear Models for RegressionDonghui WangAI Institute@ZJU2014.03ContentsLinear basis function modelsBayesian linear regressionReferences:Bishop. “Pattern Recognition and Machine Learning”, Chapter 3. 2006.Linear basis function modelsLinear basis function modelsBias parameterBasis functionRegression:Linear regression:Linear basis function model:Linear combinations of fixed nonlinear functions of the input variablesTypical basis functionsPolynomial basis function:‘Gaussian’ basis function:sigmoid basis function:Fourier basis / wavelets basisMaximum likelihood and least squaresSSE: sum-of-squares error functionAssume:Thus:For data set X = {x1, . . . , xN} and target vector t = (t1, . . . , tN)T, the likelihood function:Maximum likelihood and least squaresSolving w by ML:N×M design matrixMoore-Penrose pseudo-inverseMaximum likelihood and least squaresAbout w0:Solving β by ML:Geometry of least squares Sequential learningGradient descentGradient descent is based on the observation that if the multivariable function J(w) is defined and differentiable in a neighborhood of a point w0, then J(w) decreases fastest if one goes from w0 in the direction of the negative gradient of J(.) at w0, -J(w0) .Sequential learningn = 1, 2, …, NLearning rate Error functionStochastic gradient descent (sequential gradient descent)least-mean-squares or the LMS algorithmSequential learningBatch gradient descent:Regularized least squaresSparse modellassoError function with regularization term:Weight decay:parameter shrinkage methodMultiple outputsM × K matrix of parametersOutput K-dimensional target vector y:Multiple outputsEstimate W by ML:Decision TheoryReferences:Bishop. “Pattern Recognition and Machine Learning”, Chapter 1.5. 2006.Decision TheoryHow to make optimal decisions in situations involving uncertainty?Discriminative models:Generative models:Training setD={(xi,Ci)}, i=1,…,Np(Ci | xi)Testing setX={xi}, i=1,…,MResponse Y={Ci}, i=1,…,MTraining setD={(xi,Ci)}, i=1,