chapter 8. clssification basic conceptschapter 8. classification basic conceptschapter 8. classification basic conceptschapter 8. classification basic concepts.pptVIP

* * * * * * * * * * * * * * * * * * * * * * I : the expected information needed to classify a given sample E (entropy) : expected information based on the partitioning into subsets by A * * * * * * * * * * * * * * * * * * * * * * * * * * * * Boosting Analogy: Consult several doctors, based on a combination of weighted diagnoses—weight assigned based on the previous diagnosis accuracy How boosting works? Weights are assigned to each training tuple A series of k classifiers is iteratively learned After a classifier Mi is learned, the weights are updated to allow the subsequent classifier, Mi+1, to pay more attention to the training tuples that were misclassified by Mi The final M* combines the votes of each individual classifier, where the weight of each classifiers vote is a function of its accuracy Boosting algorithm can be extended for numeric prediction Comparing with bagging: Boosting tends to have greater accuracy, but it also risks overfitting the model to misclassified data * * Adaboost (Freund and Schapire, 1997) Given a set of d class-labeled tuples, (X1, y1), …, (Xd, yd) Initially, all the weights of tuples are set the same (1/d) Generate k classifiers in k rounds. At round i, Tuples from D are sampled (with replacement) to form a training set Di of the same size Each tuple’s chance of being selected is based on its weight A classification model Mi is derived from Di Its error rate is calculated using Di as a test set If a tuple is misclassified, its weight is increased, o.w. it is decreased Error rate: err(Xj) is the misclassification error of tuple Xj. Classifier Mi error rate is the sum of the weights of the misclassified tuples: The weight of classifier Mi’s vote is Random Forest (Breiman 2001) Random Forest: Each classifier in the ensemble is a decision tree classifier and is generated using a random selection of attributes at each node to determine the split During classification, each tree votes and the most popular class is returned Two Meth