THE DISTRIBUTION OF ENGLISH LANGUAGE （英语语言的分布）.pdfVIP

THE DISTRIBUTION OF ENGLISH LANGUAGE WORD PAIRS LINAS VEPSTAS ABSTRACT. This short note presents some empirical data on the distribution of word pairs obtained from English text. Particular attention is paid to the distribution of mutual infor- mation. 1. INTRODUCTION The mutual information of word pairs observed in text is commonly used in statistical corpus linguistics for a variety of reasons: to help identify idioms and collocations, to create dependency parsers, such as minimum spanning tree parsers, and many other uses. This is because the mutual information of a word-pair co-occurrence is a good indicator of how often two words ’go together’. However, little seems to have been written on the distribution of word pairs, or the distribution of mutual information. The current work is a short note reporting on the empirical distribution of word pairs seen in English text. Mutual information is a measure of the relatedness of two words. For example “North- ern Ireland” will have high mutual information, since the words “Northern” and “Ireland” are commonly used together. By contrast, “Ireland is” will have negative mutual informa- tion, mostly because the word “is” is used with many, many other words besides “Ireland”; there is no special relationship between these two words. High mutual information word pairs are typically noun phrases, often idioms and collocations, and almost always embody some concept (so, for example, “Northern Ireland” is the name of a place — the name of the conception of a particular country). 2. DEFINITIONS In what follows, a “pair” will always mean an “ordered word pair”; an ordered pair is simply a pair x y where both words x and y occur in the same sentence, and the word x occurs to the left of the word y in that senten