Extension of Zipf’s law to words and phrases.pdfVIP

Extension of Zipf’s Law to Words and Phrases Le Quan Ha, E. I. Sicilia-Garcia, Ji Ming, F. J. Smith School Computer Science Queen’s University of Belfast Belfast BT7 1NN, Northern Ireland q.le@qub.ac.uk Abstract Zipf’s law states that the frequency of word tokens in a large corpus of natural language is inversely proportional to the rank. The law is investigated for two languages English and Mandarin and for n- gram word phrases as well as for single words. The law for single words is shown to be valid only for high frequency words. However, when single word and n-gram phrases are combined together in one list and put in order of frequency the combined list follows Zipf’s law accurately for all words and phrases, down to the lowest frequencies in both languages. The Zipf curves for the two languages are then almost identical. 1. Introduction The law discovered empirically by Zipf (1949) for word tokens in a corpus states that if f is the frequency of a word in the corpus and r is the rank, then: r k f = (1) where k is a constant for the corpus. When log(f) is drawn against log(r) in a graph (which is often called a Zipf curve), a straight line is obtained with a slope of –1. An example with a small corpus of 250,000 tokens is given in Figure 1. Zipf’s discovery was followed by a large body of literature reviewed in a series of papers edited by Guiter and Arapov (1982). It continues to stimulate interest today (Samuelson, 1996; Montermurro, 2002; Ferrer and Solé, 2002) and, for example, it has been applied to citations Silagadze (1997) and to DNA sequences (Yonezawa Motohasi, 1999; Li, 2001). 1 10 100 1000 10000 100000 1 10 100 1000 10000 100000 log rank Figure 1 Zipf curve for the unigrams extracted from a 250,000 word tokens corpus Zipf discovered the law by analysing manually the frequencies of words in the novel “Ulysses” by James Joyce. It contains a vocabulary of 29,899 different word types associate