信息处理漫谈与案例.pptVIP

* * * 预处理去除停用词 虚词,助词出现频率高,对于表达意义的贡献却不大. 如:“着” 、“了” 、“过” 、“的” 、“地” 、“得” 统计词频时过滤掉这些停用词. * 特征选择 目标 表达力强 频率较高 区分度高 合理的特征评价函数 消除干扰,提高分类准确率 特征空间降维,减少运算量 * 特征选择(2) 文档频次 (DF) 根据预先设定的阈值去除那些文档频次特别低和特别高的特征项。 合理的阈值往往难以得到 ! 互信息(MI) 出现频率差异很大的特征项的互信息大小不具有可比性 !(即低频特征具有较高的MI) 同时,训练集中不同类别相差较大时,低频词也有较大MI. 实践证明,互信息方法是效果最差的特征选择方法! * χ2 Statistic χ2 统计量(念xi)：度量两者(term和类别)独立性的缺乏程度， χ2 越大，独立性越小，相关性越大(若ADBC,则类和词独立, N=A+B+C+D) A B C D t ~t c ~c * 特征选择性能比较: * 特征项权重计算 布尔权重 词频权重 TFIDF权重 * Similarity computing Two document D1, D2, they can be represented with VSM D1=(d1, d2,… dn) D2=(d1’, d2 ’,… dn ’) We could compute the similarity with Cos(D1,D2)=|D1*D2|/(|D1|* |D2|) =| d1* d1’+ d2* d2’+… dn * dn’| sqrt(D1*D1) * sqrt(D2*D2) * Outline Semantic computing Text representation and similarity computing Text clustering and categorization Information retrieval and search engine (FTP case) Information extraction and question answering (sample system demo) Keywords extraction and automatic abstraction One more case: Computer Aided Law Understanding Text Categorization Classifier Text Categorization * Introduction to Classifier Generative Classifier Maximize likelihood of training data Learn a model of joint probability p(x, y) e.g. Na?ve bayes Discriminative Classifier Minimize classification error rate Learn a direct mapping from input x to the class labels y or model the posterior p(y|x) directly e.g. SVM * Generative Classifier vs Discriminative Classifier Generative Classifier Build statistical model of each class Discriminative Classifier Build model of boundary between classes * A simple discriminative Classifier Given a space of word vectors Define a hyperplane ，which divides the data into two parts * ° * * * * * * * * ° ° ° ° ° ° ° ° * Text Categorization x1 x2…xn y x1 x2…xn y x1x2…xn y … x1x2…xn y Feature Selection x1x2 …xn Model Training x1 x2…xn x1 x2…xn ? Data Text * Components of Text Categorization Text