信息论——浓缩打印版课件.ppt

Information Theory;History;What is Information?;Some Insights about Information;Shannon’s Definition;Shannon’s Definition;Communication System Model;A More Detailed Model;Information Theory Answers Two Fundamental Questions;Review and Outline;Review of Probability Theory;Discrete Probability Theory;Discrete Random Variable (R.V.);Probability Distribution;Joint Probability Distribution;Conditional Probability;Discrete and Continuous Prob. Distribution;Entropy;Definition of Entropy;Joint Entropy and Conditional Entropy;Conditional Entropy;Chain Rule of Entropy;Chain Rule of Entropy;Corollary of Chain Rule;Mutual Information;Relationship between Entropy and Mutual Information;Chain Rule for Mutual Information;Review and Outline;Entropy;Relative Entropy or Kullback Leibler Distance;Entropy and MIRepresented by RE;Intuition of Convexity and Concavity;Math: Concavity and Convexity;Math: Jensen’s Inequality;Information Inequality;Corollary of Information Ineq. Th.;Theorem;Math: LOG SUM Inequality;Convexity of Relative Entropy;Concavity of Entropy;Mutual Information and Input Distribution;Mutual Information and Transition Probability;Markov Chain;Data Processing Inequality;Corollary;Fano’s Inequality;Fano’s Inequality;Summary;Review and Outline;Relative Entropy or Kullback Leibler Distance;Review and Outline;Properties of Entropy and MI;Convexity of Relative Entropy;Math: LOG SUM Inequality (Re);Markov Chain;Data Processing Inequality;Corollary;Fano’s Inequality;Fano’s Inequality;Review and Outline;Asymptotic Equipartition Property Chapter 3;;Example;AEP;Consequence of AEP: Typical Set;The Reason of Calling it Typical;How big of The Typical Set?;Data Compression by Typical Set;Data Compression Procedure;The Total bits Needed;Theorem;Jointly Typical Sequence Ch8.6/p194;The Number of Jointly Typical Sequence;Ch 4 Entropy Rates of a Stochastic Process;Review of Markov Chain;Review of Markov Chain;Review of Markov Chain;Review of Markov Chain;Entropy Rate;ER of Stationary St