After I showed the benchmarking results of SyntaxNet and our rule system based on grammar engineering, many people seem to be surprised by the fact that the rule system beats the newest deep-learning based parser in data quality . I then got asked many questions, one question is: Q: We know that rules crafted by linguists are good at precision, how about recall? This question is worth a more in-depth discussion and serious answer because it touches the core of the viability of the forgotten school: why is it still there? what does it have to offer? The key is the excellent data quality as advantage of a hand-crafted system, not only for precision, but high recall is achievable as well. Before we elaborate, here was my quick answer to the above question: Unlike precision, recall is not rules' forte, but there are ways to enhance recall; To enhance recall without precision compromise, one needs to develop more rules and organize rules in a hierarch y, and organize grammars in a pipeline , so r ecall is a function of time; To enhance recall with limited compromise in precision, one can fine-tune the rules to loosen conditions. Let me address these points by presenting the scene of action for this linguistic art in its engineering craftsmanship. A rule system is based on compiled computational grammars. A grammar is a set of linguistic rules encoded in some formalism. What happens in grammar engineering is not much different from other software engineering projects. As knowledge engineer, a computational linguist codes a rule in a NLP-specific language, based on a development corpus. The development is data-driven, each line of rule code goes through rigid unit tests and then regression tests before it is submitted as part of the updated system. Depending on the design of the architect, there are all types of information available for the linguist developer to use in crafting a rule's conditions, e.g. a rule can check any elements of a pattern by enforcing conditions on (i) word or stem itself (i.e. string literal, in cases of capturing, say, idiomatic expressions), and/or (ii) POS (part-of-speech, such as noun, adjective, verb, preposition), (iii) and/or orthography features (e.g. initial upper case, mixed case, token with digits and dots), and/or (iv) morphology features (e.g. tense, aspect, person, number, case, etc. decoded by a previous morphology module), (v) and/or syntactic features (e.g. verb subcategory features such as intransitive, transitive, ditransitive), (vi) and/or lexical semantic features (e.g. human, animal, furniture, food, school, time, location, color, emotion). There are almost infinite combinations of such conditions that can be enforced in rules' patterns. A linguist's job is to use such conditions to maximize the benefits of capturing the target language phenomena, through a process of trial and error. Given the description of grammar engineering as above, what we expect to see in the initial stage of grammar development is a system precision-oriented by nature. Each rule developed is geared towards a target linguistic phenomenon based on the data observed in the development corpus: conditions can be as tight as one wants them to be, ensuring precision. But no single rule or a small set of rules can cover all the phenomena. So the recall is low in the beginning stage. Let us push things to extreme, if a rule system is based on only one grammar consisting of only one rule, it is not difficult to quickly develop a system with 100% precision but very poor recall. But what is good of a system that is precise but without coverage? So a linguist is trained to generalize. In fact, most linguists are over-trained in school for theorizing and generalization before they get involved in software industrial development. In my own experience in training new linguists into knowledge developers, I often have to de-train this aspect of their education by enforcing strict procedures of data-driven and regression-free development. As a result, the system will generalize only to the extent allowed to maintain a target precision, say 90% or above. It is a balancing art. Experienced linguists are better than new graduates. Out of explosive possibilities of conditions, one will only test some most likely combination of conditions based on linguistic knowledge and judgement in order to reach the desired precision with maximized recall of the target phenomena. For a given rule, it is always possible to increase recall at compromise of precision by dropping some conditions or replacing a strict condition by a loose condition (e.g. checking a feature instead of literal, or checking a general feature such as noun instead of a narrow feature such as human ). When a rule is fine-tuned with proper conditions for the desired balance of precision and recall, the linguist developer moves on to try to come up with another rule to cover more space of the target phenomena. So, as the development goes on, and more data from the development corpus are brought to the attention on the developer's radar, more rules are developed to cover more and more phenomena, much like silkworms eating mulberry leaves. This is incremental enhancement fairly typical of software development cycles for new releases. Most of the time, newly developed rules will overlap with existing rules, but their logical OR points to an enlarged conquered territory. It is hard work, but recall gradually, and naturally, picks up with time while maintaining precision until it hits long tail with diminishing returns. There are two caveats which are worth discussing for people who are curious about this seasoned school of grammar engineering. First, not all rules are equal. A non-toy rule system often provides mechanism to help organize rules in a hierarchy for better quality as well as easier maintenance: after all, a grammar hard to understand and difficult to maintain has little prospect for debugging and incremental enhancement. Typically, a grammar has some general rules at the top which serve as default and cover the majority of phenomena well but make mistakes in the exceptions which are not rare in natural language. As is known to all, naturally language is such a monster that almost no rules are without exceptions. Remember in high school grammar class, our teacher used to teach us grammar rules. For example, one rule says that a bare verb cannot be used as predicate with third person singular subject, which should agree with the predicate in person and number by adding -s to the verb: hence, She leaves instead of *S he leave . But soon we found exceptions in sentences like The teacher demanded that she leave. This exception to the original rule only occurs in object clauses following certain main clause verbs such as demand , theoretically labeled by linguists as subjunctive mood. This more restricted rule needs to work with the more general rule to result in a better formulated grammar. Likewise, in building a computational grammar for automatic parsing or other NLP tasks, we need to handle a spectrum of rules with different degrees of generalizations in achieving good data quality for a balanced precision and recall. Rather than adding more and more restrictions to make a general rule not to overkill the exceptions, it is more elegant and practical to organize the rules in a hierarchy so the general rules are only applied as default after more specific rules are tried, or equivalently, specific rules are applied to overturn or correct the results of general rules. Thus, most real life formalisms are equipped with hierarchy mechanism to help linguists develop computational grammars to model the human linguistic capability in language analysis and understanding. The second point that relates to the topic of recall of a rule system is so significant but often neglected that it cannot be over-emphasized and it calls for a separate writing in itself. I will only present a concise conclusion here. It relates to multiple levels of parsing that can significantly enhance recall for both parsing and parsing-supported NLP applications . In a multi-level rule system, each level is one module of the system, involving a grammar. Lower levels of grammars help build local structures (e.g. basic Noun Phrase), performing shallow parsing. System thus designed are not only good for modularized engineering, but also great for recall because shallow parsing shortens the distance of words that hold syntactic relations (including long distance relations) and lower level linguistic constructions clear the way for generalization by high level rules in covering linguistic phenomena. In summary, a parser based on grammar engineering can reach very high precision; in addition, there are proven effective ways of enhancing its recall too. High recall can be achieved if enough time and expertise are invested in its development. In case of parsing, as shown by test results , our seasoned English parser is good at both precision (96% vs. SyntaxNet 94%) and recall (94% vs. Syntax 95%, only 1 percentage point lower than SyntaxNet) in news genre, and with regards to social media, our parser is robust enough to beat SyntaxNet in both precision (89% vs. SyntaxNet 60%) and recall (72% vs. SyntaxNet 70%). Is Google SyntaxNet Really the World’s Most Accurate Parser? It is untrue that Google SyntaxNet is the “world’s most accurate parser” R. Srihari, W Li, C. Niu, T. Cornell. InfoXtract: A Customizable Intermediate Level Information Extraction Engine. Journal of Natural Language Engineering, 12(4), 1-37, 2006 K. Church: “A Pendulum Swung Too Far” , Linguistics issues in Language Technology, 2011; 6(5) Pros and Cons of Two Approaches: Machine Learning vs Grammar Engineering Introduction of Netbase NLP Core Engine Overview of Natural Language Processing Dr. Wei Li’s English Blog on NLP
Requirements: 1)the number of individuals in a population is set to be fixed at N across generations; 2) the number of carriers of the i -th genotype is set to be a variable N i ; 3)N i may change over generations depending on the fitness of the the i -th genotype while N remains fixed. Ways of fulfillment in NetLogo: An example is seen in Novak Wilensky's 'Bug Hunt Coevolution' model, where the way of embodying the fitness is to ask each individual to die if it fulfills a certain criterion (here it's lack of energy to a certain extent), and to randomly ask one of the rest individuals to reproduce +1 once an individual dies. Why not ask the individual with highest energy to reproduce? As I understand the selection force will become stronger in the model if so. Now the point is what's the real-world case. Actually we have a few alternative ways of manifesting the fitness in the model. 1) The way introduced by the authors (weak selection; the worst may be eliminated finally but the best may not win); 2) As said above, ask the individual with the lowest energy to die and the one with the highest to multiply (strong directed selection); 3) Ask the individual with the highest energy to multiply, meanwhile ask a randomly selected individual from the rest of the population to die (weak selection; the best may win finally but the worst may never be eliminated); 4) Create a table with fitness (or energy) of each genotype, and ask a planned number of individuals of each genotype to die or multiply based on a comprehensive calculation. I would say, the fourth way is actually not an agent-based model any more. However, our decision of which way to be used depends on characteristics of the studied organisms/people in real world.
Fornew readers and those who request to be “ 好友 good friends”please read my 公告 栏 first. An unending and unresolved discussion among concerned citizens and scientists is the support for basic research. A couple of readers of my recent blog articles also raised this issue with me. I don’t pretend that I have a perfect solution for this problem, but merely respond here with my personal observations. 1. One reason citizens and governments support basic research is the “Serendipity” factor. The history of science is full of examples where seemingly useless theory turns out to be crucial in applications later on. Einstein’s theory of relativity surely does not seem to affect any practical things we do, particularly at the time of its invention. But without it, our ubiquitous GPS system found in modern autos would not be possible (see http://www.sciencenet.cn/m/user_content.aspx?id=247269 ). Furthermore, the communication satellites used in the GPS was first proposed in science fiction literature by Arthur C. Clarke who also famously said “ New ideas pass through three periods: 1) It can't be done. 2) It probably can be done, but it's not worth doing. 3) I knew it was a good idea all along!” Basic research often passes through similar stages concerning whether or not it represents public money well spent. 2. Basic research can be problem-driven . Especially in academia and universities, basic research can be supported by national goals or society wishes. Note problem-driven is different from applied . Industrial research is applied and aimed at produce products and profits for the company. 3. Basic research can also be methodology oriented , e.g., the theory of automatic control and optimization. Such effort clearly can be justified on economic grounds 4. Society and government support Arts and Music. These efforts do not benefit people in a material way, nor do they benefit all people. But no one would argue against the notion that a civilization without Art and Music will be a lifeless and dull existence indeed. Basic research holds the same kind of inspiration and beauty. 5. Not every bit of basic research needs to be an invention or discovery. Much effort in basic research deal with the codification and simplification of knowledge. Without such efforts, human beings cannot possibly deal with the expansion and accumulation of knowledge through the ages. 6. Chinese culture is heavily experience based rather than knowledge based. Medicine is one example that comes to mind. It is more a matter of 知其然 and not so much 知其所以然 . Since experience must be based on reality, I submit in the Chinese collective conscience there is a preference for “applied” or applications. 7. Even accepting the desirability of basic research, there is the question of “How Much?”. After all, it is mostly public money we are spending to do it. This question of course does not have a simple answer. In a democratic society as opposed to a dictatorship, it is an issue of supply and demand, public policy, culture, and general education level of the populace. Take the example of Western (or Chinese) opera. This used to be a popular art form for the general public before the days of movies, radio, broadcast, cable, and satellite, television. But nowadays it is supported largely by rich patrons, foundations, and the educated elite. The amount of funds for and what kinds of basic research in general should also follow the complex interactions of the market and society. 8. Lastly, it is the responsibility of scientists themselves to communicate the needs for and benefits of basic research to the public. By and large, scientists do a relatively poor job in this regard. Some arrogant scientists even look down at any popularization effort and deem such endeavor to be beneath them. They are the equivalent of the high priests of old religion and cults (the less the public understand, the better. Mystery let me kept my position and power). My past blog articles clearly demonstrates that I do not agree with this view. The best defense for democracy is an educated public. One of the main purposes of platform such as ScienceNet is to educate and provide a forum for the civilized discussion of the history, administration, support, and understanding of science and technology but not a place for quarrels and shouting down others. Long may she live and prosper.
标签: based
