TREND 烂ND 燛VENT 燚ETECTION .PDF

TREND AND EVENT DETECTION IN SOCIAL STREAMS Kostas Tsioutsiouliklis @kostas September 2012 Outline •  Trending topic detecon 1)  Simple counng 2)  Chi-square test 3)  Topic-specific models •  Event detecon 1)  Online clustering 2)  Online clustering using MinHash •  Natural language processing for tweets st 1 approach: Simple Counng •  400M tweets per day, or 4600 tweets per sec Tokenizaon, phrase extracon •  What is a topic? •  n-gram –  Simple, low precision/high recall, large space •  Diconary-based phrases (wikipedia entries, named enes) –  Simple, high precision/low recall, stale •  Noun-phrases (NP) extracted via part-of-speech tagging –  Difficult for short texts Term frequencies •  Tokenize text and count term frequencies •  Assume, for now, unigrams Term Count Obama 1 Detours 1 … … Hurricane 1 … … Term Count Obama 1 Detours 1 … … Hurricane 2