Diversifying Restricted Boltzmann Machine for Document Modeling.pdf

Diversifying Restricted Boltzmann Machine for Document Modeling Pengtao Xie School of Computer Science Carnegie Mellon University Pittsburgh, PA, 15213 pengtaox@ Yuntian Deng School of Computer Science Carnegie Mellon University Pittsburgh, PA, 15213 yuntiand@ Eric P. Xing School of Computer Science Carnegie Mellon University Pittsburgh, PA, 15213 epxing@ ABSTRACT Restricted Boltzmann Machine (RBM) has shown great ef- fectiveness in document modeling. It utilizes hidden units to discover the latent topics and can learn compact semantic representations for documents which greatly facilitate doc- ument retrieval, clustering and classification. The popular- ity (or frequency) of topics in text corpora usually follow a power-law distribution where a few dominant topics occur very frequently while most topics (in the long-tail region) have low probabilities. Due to this imbalance, RBM tends to learn multiple redundant hidden units to best represent dominant topics and ignore those in the long-tail region, which renders the learned representations to be redundant and non-informative. To solve this problem, we propose Di- versified RBM (DRBM) which diversifies the hidden units, to make them cover not only the dominant topics, but also those in the long-tail region. We define a diversity metric and use it as a regularizer to encourage the hidden units to be diverse. Since the diversity metric is hard to optimize directly, we instead optimize its lower bound and prove that maximizing the lower bound with projected gradient ascent can increase this diversity metric. Experiments on docu- ment retrieval and clustering demonstrate that with diver- sification, the document modeling power of DRBM can be greatly improved. Categories and Subject Descriptors H.2.8 [Database Management]: Database Applications, Data Mining General Terms Algorithms, Experiments Keywords Diversified Restricted Boltzmann Machine, Diversity, Power- law Distribution, Document Modeling, Topic Modeling Permis