Deviation of Zipf’s and Heaps’ Laws inHuman Languages with LimitedDictionary Sizes Zipf’s law on word frequency and Heaps’ law on the growth of distinct words are observed in Indo-European language family, but it does not hold for languages like Chinese, Japanese and Korean. These languages consist of characters, and are of very limited dictionary sizes. Extensive experiments show that: (i) The character frequency distribution follows a power law with exponent close to one, at which the corresponding Zipf’s exponent diverges. Indeed, the character frequency decays exponentially in the Zipf’s plot. (ii) The number of distinct characters grows with the text length in three stages: It grows linearly in the beginning, then turns to a logarithmical form, and eventually saturates. A theoretical model for writing process is proposed, which embodies the rich-get-richer mechanism and the effects of limited dictionary size. Experiments, simulations and analytical solutions agree well with each other. This work refines the understanding about Zipf’s and Heaps’ laws in human language systems. Citation： Linyuan Lu，Zi-Ke Zhang， Tao Zhou，Deviation of Zipf’s and Heaps’ Laws inHuman Languages with LimitedDictionary Sizes，Scientific Reports 3，1082 （2013）. Download: srep01082.pdf 相关博文： http://blog.sciencenet.cn/home.php?mod=spaceuid=3075do=blogid=659858 最近有一文，较详细讨论了Heaps和Zipf之关系，感兴趣者可供参考~ 题目：Power-Law Connections:From Zipf to Heaps and Beyond 作者：Iddo I. Eliazar and Morrel H. Cohen Abstract In this paper we explore the asymptotic statistics of a general modelof rank distribution in the large-ensemble limit; the construction of thegeneral model is motivated by recent empirical studies of rank distributions.Applying Lorenzian, oligarchic, and Heapsian asymptotic analyseswe establish a comprehensive set of closed-form results linking togetherrank distributions, probability distributions, oligarchy sizes, and innovationrates. In particular, the general results reveal the fundamentalunderlying connections between Zipf’s law, Pareto’s law, and Heaps’ law– three elemental empirical power-laws that are ubiquitously observed inthe sciences. Keywords: rank distributions; power-laws; Zipf’s law; Pareto’s law;Heaps’ law; Lorenz curves; the distribution of wealth; oligarchy sizes;innovation rates; phase transitions; self-organized criticality. PACS: 02.50.-r (Probability theory, stochastic processes, and statistics); 89.65.-s (Social and economic systems).
标签: 选择动力学
