Towards Integrating Information and Control Theories: From Information-Theoretic Measures to System Performance Limitations 信息論與控制論之融合：從信息論測度到系統性能局限 链接： https://scholars.cityu.edu.hk/en/theses/towards-integrating-information-and-control-theories-from-informationtheoretic-measures-to-system-performance-limitations(8f21be24-127e-495d-a0cc-4a6e728450ee).html --- 其中引用了不少有意思的 quotes（也可参考： http://blog.sciencenet.cn/blog-286797-1022865.html 以及 http://blog.sciencenet.cn/blog-286797-1021452.html ）： Frontispiece When one submerges the gourd bowl in water, there floats the gourd ladle. — Chinese proverb Chapter 1 There is an obvious analogy between the problem of smoothing the data to eliminate or reduce the effect of tracking errors and the problem of separating a signal from interfering noise in communications systems. — R. B. Blackman, H. W. Bode, and C. E. Shannon, “Data Smoothing and Prediction in Fire-Control Systems,” 1946 (We) become aware of the essential unity of the set of problems centring about communication, control, and statistical mechanics, whether in the machine or living tissue... We have decided to call the entire field of control and communication theory, whether in the machine or the animal, by the same Cybernetics. — N. Wiener, “Cybernetics,” 1948, Fundamental limits are actually at the core of many fields of engineering, science and mathematics... Firstly, they evolve from basic axioms about the nature of the universe. Secondly, they describe inescapable performance bounds that act as benchmarks for practical systems. And thirdly, they are recognized as being central to the design of real systems. — M. M. Seron, J. H. Braslavsky, and G. C. Goodwin, “Fundamental Limitations in Filtering and Control,” 1997 Chapter 2 The sciences do not try to explain, they hardly even try to interpret, they mainly make models. By a model is meant a mathematical construct which, with the addition of certain verbal interpretations, describes observed phenomena. The justification of such a mathematical construct is solely and precisely that it is expected to work. — John von Neumann The only way of discovering the limits of the possible is to venture a little way past them into the impossible. — Arthur C. Clarke A theory is the more impressive the greater the simplicity of its premises, the more different kinds of things it relates, and the more generalized its area of applicability. Therefore the deep impression that classical thermodynamics made upon me. It is the only physical theory of universal content which I am convinced will never be overthrown, within the framework of applicability of its basic concepts. — Albert Einstein Chapter 3 The idea of a statistical message source is central to Shannon’s work. The study of random processes had entered into communication before his communication theory. There was a growing understanding of and ability to deal with problems of random noise... Wiener had dealt extensively with the extrapolation, interpolation, and smoothing of time series. Although Wiener’s book was published in 1949, it had been available earlier in a wartime version known as the Yellow Peril (the cover was yellow). Shannon and Bode took considerable pains to put Wiener’s work in a form more directly useful to them (and to many others). — J. R. Pierce, “The Early Days of Information Theory,” 1973 We said before: “It feeds upon negative entropy,” attracting, as it were, a stream of negative entropy upon itself, to compensate the entropy increase it produces by living and thus to maintain itself on a stationary and fairly low entropy level. — Erwin Schrodinger, “What is Life,” 1944 If one has really technically penetrated a subject, things that previously seemed in complete contrast, might be purely mathematical transformations of each other. — John von Neumann Chapter 4 However, by building an amplifier whose gain is deliberately made, say 40 decibels higher than necessary (10000 fold excess on energy basis), and then feeding the output back on the input in such a way as to throw away that excess gain, it has been found possible to effect extraordinary improvement in constancy of amplification and freedom from nonlinearity. — H. S. Black, “Stabilized Feedback Amplifiers,” 1934 In control and communication we are always fighting nature’s tendency to degrade the organized and to destroy the meaningful; the tendency, as Gibbs has shown us, for entropy to increase. — N. Wiener, “The Human Use of Human Beings,” 1950 All stable processes we shall predict. All unstable processes we shall control. — John von Neumann Chapter 5 What’s in a name? In the case of Shannon’s measure the naming was not accidental. In 1961 one of us (Tribus) asked Shannon what he had thought about when he had finally confirmed his famous measure. Shannon replied: “My greatest concern was what to call it. I thought of calling it ‘information’, but the word was overly used, so I decided to call it ‘uncertainty’. When I discussed it with John von Neumann, he had a better idea. Von Neumann told me, ‘You should call it entropy, for two reasons. In the first place you uncertainty function has been used in statistical mechanics under that name. In the second place, and more importantly, no one knows what entropy really is, so in a debate you will always have the advantage.’” — M. Tribus, E. C. McIrvine, “Energy and information,” 1971 The bottom line for mathematicians is that the architecture has to be right. In all the mathematics that I did, the essential point was to find the right architecture. It’s like building a bridge. Once the main lines of the structure are right, then the details miraculously fit. The problem is the overall design. — Freeman Dyson Far waters fail to quench near fires. — Chinese Proverb Chapter 6 I like to think of Bode’s integrals as conservation laws. They state precisely that a certain quantity—the integrated value of the log of the magnitude of the sensitivity function—is conserved under the action of feedback. The total amount of this quantity is always the same. It is equal to zero for stable plant/compensator pairs, and it is equal to some fixed positive amount for unstable ones... This applies to every controller, no matter how it was designed. Sensitivity improvements in one frequency range must be paid for with sensitivity deteriorations in another frequency range, and the price is higher if the plant is open-loop unstable. — G. Stein, “Respect the Unstable,” 2003 The average performance of any pair of algorithms across all possible problems is identical. This means in particular that if some algorithm’s performance is superior to that of another algorithm over some set of optimization problems, then the reverse must be true over the set of all other optimization problems. — D. H. Wolpert, W. G. Macready, “No Free Lunch Theorems for Optimization,” 1997 We know the past but cannot control it. We control the future but cannot know it. — Claude Shannon Chapter 7 Consider the case where you are the controller and you observe samples of the process output whose average has been satisfactorily close to set point and that suffers only from white noise disturbances. Should you make an adjustment to the control output upon observing a sample of the process output that is not on set point? If the average of the process output is indeed nearly at the set point then any deviation, if it is really white or unautocorrelated, will be completely independent of the previous value of the control output and it will have no impact on subsequent disturbances. Therefore, if you should react to such a deviation, you would be wasting your time because the next observation will contain another deviation that has nothing to do with the previous deviation on which you acted. You, in fact, may make things worse... A feedback controller cannot decrease the standard deviation of the white noise riding on the process output. At best it can keep the average on set point. — D. M. Koenig, “Practical Control Engineering,” 2009 In respect of military method, we have, firstly, Measurement; secondly, Estimation of quantity; thirdly, Calculation; fourthly, Balancing of chances; fifthly, Victory. — Sun Tzu, “The Art of War” All the evidence shows that God was actually quite a gambler, and the universe is a great casino, where dice are thrown, and roulette wheels spin on every occasion. Over a large number of bets, the odds even out and we can make predictions... But over a very small number of rolls of the dice, the uncertainty principle is very important. — Stephen Hawking Chapter 8 The world is continuous, but the mind is discrete. — David Mumford Time is defined so that motion looks simple. — John Wheeler If everything seems under control, you’re just not going fast enough. — Mario Andretti Chapter 9 Essentially, all models are wrong, but some are useful. — George. E. P. Box In theory, theory and practice are the same. In practice, they are not. — Albert Einstein Any one who considers arithmetical methods of producing random digits is, of course, in a state of sin. — John von Neumann Chapter 10 An understanding of fundamental limitations is an essential element in all engineering. Shannon’s early results on channel capacity have always had center court in signal processing. Strangely, the early results of Bode were not accorded the same attention in control. — K. J. Astrom, in G. Stein, “Respect the Unstable,” 2003 If I turn toward a science not for external reasons such as earning an income, or for ambition, and also not — at least not exclusively — for the mere sportive joy and the fun of brain-acrobatics, then I must ask myself the question: what is the final goal that the science I am devoted to will and can reach? To what extent are its general results “true”? What is essential and what is based only on accident in its development? — Albert Einstein As a mathematical discipline travels far from its empirical source, or still more, if it is a second and third generation only indirectly inspired by ideas coming from “reality” it is beset with very grave dangers. It becomes more and more purely aestheticizing, more and more purely I’art pour I’art. This need not be bad, if the field is surrounded by correlated subjects, which still have closer empirical connections, or if the discipline is under the influence of men with an exceptionally well-developed taste. But there is a grave danger that the subject will develop along the line of least resistance, that the stream, so far from its source, will separate into a multitude of insignificant branches, and that the discipline will become a disorganized mass of details and complexities. In other words, at a great distance from its empirical source, or after much “abstract” inbreeding, a mathematical subject is in danger of degeneration. At the inception the style is usually classical; when it shows signs of becoming baroque, then the danger signal is up... In any event, whenever this stage is reached, the only remedy seems to me to be the rejuvenating return to the source: the re-injection of more or less directly empirical ideas. I am convinced that this was a necessary condition to conserve the freshness and the vitality of the subject and that this will remain equally true in the future. — John von Neumann
Towards Integrating Control and Information Theories: From Information-T heoretic Measures to Control Performance Limitations 控制论与信息论融合之初探：从信息论测度到控制性能极限 Song Fang, Jie Chen, Hideaki Ishii http://www.springer.com/cn/book/9783319492889 Introduction 节选 : 1.1 A Word on Control versus Communication The attempt to merge control and communication theories dates back to the early works of Hendrik Bode, Claude Shannon, and Norbert Wiener. In , observations on similarities and relations between control and communication were made in many places, proclaiming that “the theory of control in engineering, whether human or animal or mechanical, is a chapter in the theory of messages,” and that “it is my book that the physical functioning of the living individual and the operation of some of the newer communication machines are precisely parallel in their analogous attempts to control entropy through feedback.” Indeed, the book is themed on the merging of “control and communication in the animal and the machine.” As Norbert Wiener put it, “(we) become aware of the essential unity of the set of problems centring about communication, control, and statistical mechanics, whether in the machine or living tissue... We have decided to call the entire field of control and communication theory, whether in the machine or the animal, by the same Cybernetics.” On the other hand, as early as in , it was observed that “there is an obvious analogy between the problem of smoothing the data to eliminate or reduce the effect of tracking errors and the problem of separating a signal from interfering noise in communications systems,” (see, e.g., ) while noting that “this analogy... must of course not be carried too far.” It was not carried too far. Since then, control and communication have grown into two almost independent fields (see, e.g., , respectively), with different focuses and even different mathematical tools (control/estimation theories and information/coding theories, respectively) for modeling, analysis, and design. In a way, control mainly concerns utilization of information, while communication is more about information transmission. In control, properties of systems are almost always of first priority and should be soundly modeled where possible, based on which stability and performance analysis can then be carried out. On the other hand, in communication, properties of signals are most emphasized, whereas models of communication systems are often not easily obtained, and certainly not describable in ways typical for dynamic processes. For instance, what is the input-output “dynamic model” of the source coding layer/process using Hamming code ? In addition, in control, the element of “time” is always explicit in the analysis of system dynamics, while in communication, at least in classical information theory, “time” is not directly involved. Time delays are tolerable in communication and information theory. Indeed, to achieve a tight bound of channel capacity usually requires coding with arbitrarily long block lengths . On the other hand, time delays severely deteriorate stability and performance as well as robustness of the closed-loop systems in control theory , and must be taken good care of in the analysis and design. In spite of these differences, and just as an old Chinese saying goes, “that which is long divided must unify; that which is long unified must divide,” the interplay of control and communication (information theory) is once again a heated topic in recent years. 参考文献详见 http://link.springer.com/book/10.1007%2F978-3-319-49289-6 的 Back Matter。
Towards Integrating Control and Information Theories: From Information-T heoretic Measures to Control Performance Limitations 控制论与信息论融合之初探：从信息论测度到控制性能极限 Song Fang, Jie Chen, Hideaki Ishii http://www.springer.com/cn/book/9783319492889 每一章的备选 quotes ： Frontispiece When one submerges the gourd bowl in water, there floats the gourd ladle. — Chinese proverb Chapter 1 There is an obvious analogy between the problem of smoothing the data to eliminate or reduce the effect of tracking errors and the problem of separating a signal from interfering noise in communications systems. — R. B. Blackman, H. W. Bode, and C. E. Shannon, “Data Smoothing and Prediction in Fire-Control Systems,” 1946 (We) become aware of the essential unity of the set of problems centring about communication, control, and statistical mechanics, whether in the machine or living tissue... We have decided to call the entire field of control and communication theory, whether in the machine or the animal, by the same Cybernetics. — N. Wiener, “Cybernetics,” 1948, Fundamental limits are actually at the core of many fields of engineering, science and mathematics... Firstly, they evolve from basic axioms about the nature of the universe. Secondly, they describe inescapable performance bounds that act as benchmarks for practical systems. And thirdly, they are recognized as being central to the design of real systems. — M. M. Seron, J. H. Braslavsky, and G. C. Goodwin, “Fundamental Limitations in Filtering and Control,” 1997 Chapter 2 The sciences do not try to explain, they hardly even try to interpret, they mainly make models. By a model is meant a mathematical construct which, with the addition of certain verbal interpretations, describes observed phenomena. The justification of such a mathematical construct is solely and precisely that it is expected to work. — John von Neumann The only way of discovering the limits of the possible is to venture a little way past them into the impossible. — Arthur C. Clarke A theory is the more impressive the greater the simplicity of its premises, the more different kinds of things it relates, and the more generalized its area of applicability. Therefore the deep impression that classical thermodynamics made upon me. It is the only physical theory of universal content which I am convinced will never be overthrown, within the framework of applicability of its basic concepts. — Albert Einstein Chapter 3 The idea of a statistical message source is central to Shannon’s work. The study of random processes had entered into communication before his communication theory. There was a growing understanding of and ability to deal with problems of random noise... Wiener had dealt extensively with the extrapolation, interpolation, and smoothing of time series. Although Wiener’s book was published in 1949, it had been available earlier in a wartime version known as the Yellow Peril (the cover was yellow). Shannon and Bode took considerable pains to put Wiener’s work in a form more directly useful to them (and to many others). — J. R. Pierce, “The Early Days of Information Theory,” 1973 We said before: “It feeds upon negative entropy,” attracting, as it were, a stream of negative entropy upon itself, to compensate the entropy increase it produces by living and thus to maintain itself on a stationary and fairly low entropy level. — Erwin Schrodinger, “What is Life,” 1944 If one has really technically penetrated a subject, things that previously seemed in complete contrast, might be purely mathematical transformations of each other. — John von Neumann Chapter 4 However, by building an amplifier whose gain is deliberately made, say 40 decibels higher than necessary (10000 fold excess on energy basis), and then feeding the output back on the input in such a way as to throw away that excess gain, it has been found possible to effect extraordinary improvement in constancy of amplification and freedom from nonlinearity. — H. S. Black, “Stabilized Feedback Amplifiers,” 1934 In control and communication we are always fighting nature’s tendency to degrade the organized and to destroy the meaningful; the tendency, as Gibbs has shown us, for entropy to increase. — N. Wiener, “The Human Use of Human Beings,” 1950 All stable processes we shall predict. All unstable processes we shall control. — John von Neumann Chapter 5 What’s in a name? In the case of Shannon’s measure the naming was not accidental. In 1961 one of us (Tribus) asked Shannon what he had thought about when he had finally confirmed his famous measure. Shannon replied: “My greatest concern was what to call it. I thought of calling it ‘information’, but the word was overly used, so I decided to call it ‘uncertainty’. When I discussed it with John von Neumann, he had a better idea. Von Neumann told me, ‘You should call it entropy, for two reasons. In the first place you uncertainty function has been used in statistical mechanics under that name. In the second place, and more importantly, no one knows what entropy really is, so in a debate you will always have the advantage.’” — M. Tribus, E. C. McIrvine, “Energy and information,” 1971 The bottom line for mathematicians is that the architecture has to be right. In all the mathematics that I did, the essential point was to find the right architecture. It’s like building a bridge. Once the main lines of the structure are right, then the details miraculously fit. The problem is the overall design. — Freeman Dyson Far waters fail to quench near fires. — Chinese Proverb Chapter 6 I like to think of Bode’s integrals as conservation laws. They state precisely that a certain quantity—the integrated value of the log of the magnitude of the sensitivity function—is conserved under the action of feedback. The total amount of this quantity is always the same. It is equal to zero for stable plant/compensator pairs, and it is equal to some fixed positive amount for unstable ones... This applies to every controller, no matter how it was designed. Sensitivity improvements in one frequency range must be paid for with sensitivity deteriorations in another frequency range, and the price is higher if the plant is open-loop unstable. — G. Stein, “Respect the Unstable,” 2003 The average performance of any pair of algorithms across all possible problems is identical. This means in particular that if some algorithm’s performance is superior to that of another algorithm over some set of optimization problems, then the reverse must be true over the set of all other optimization problems. — D. H. Wolpert, W. G. Macready, “No Free Lunch Theorems for Optimization,” 1997 We know the past but cannot control it. We control the future but cannot know it. — Claude Shannon Chapter 7 Consider the case where you are the controller and you observe samples of the process output whose average has been satisfactorily close to set point and that suffers only from white noise disturbances. Should you make an adjustment to the control output upon observing a sample of the process output that is not on set point? If the average of the process output is indeed nearly at the set point then any deviation, if it is really white or unautocorrelated, will be completely independent of the previous value of the control output and it will have no impact on subsequent disturbances. Therefore, if you should react to such a deviation, you would be wasting your time because the next observation will contain another deviation that has nothing to do with the previous deviation on which you acted. You, in fact, may make things worse... A feedback controller cannot decrease the standard deviation of the white noise riding on the process output. At best it can keep the average on set point. — D. M. Koenig, “Practical Control Engineering,” 2009 In respect of military method, we have, firstly, Measurement; secondly, Estimation of quantity; thirdly, Calculation; fourthly, Balancing of chances; fifthly, Victory. — Sun Tzu, “The Art of War” All the evidence shows that God was actually quite a gambler, and the universe is a great casino, where dice are thrown, and roulette wheels spin on every occasion. Over a large number of bets, the odds even out and we can make predictions... But over a very small number of rolls of the dice, the uncertainty principle is very important. — Stephen Hawking Chapter 8 The world is continuous, but the mind is discrete. — David Mumford Time is defined so that motion looks simple. — John Wheeler If everything seems under control, you’re just not going fast enough. — Mario Andretti Chapter 9 Essentially, all models are wrong, but some are useful. — George. E. P. Box In theory, theory and practice are the same. In practice, they are not. — Albert Einstein Any one who considers arithmetical methods of producing random digits is, of course, in a state of sin. — John von Neumann Chapter 10 An understanding of fundamental limitations is an essential element in all engineering. Shannon’s early results on channel capacity have always had center court in signal processing. Strangely, the early results of Bode were not accorded the same attention in control. — K. J. Astrom, in G. Stein, “Respect the Unstable,” 2003 If I turn toward a science not for external reasons such as earning an income, or for ambition, and also not — at least not exclusively — for the mere sportive joy and the fun of brain-acrobatics, then I must ask myself the question: what is the final goal that the science I am devoted to will and can reach? To what extent are its general results “true”? What is essential and what is based only on accident in its development? — Albert Einstein As a mathematical discipline travels far from its empirical source, or still more, if it is a second and third generation only indirectly inspired by ideas coming from “reality” it is beset with very grave dangers. It becomes more and more purely aestheticizing, more and more purely I’art pour I’art. This need not be bad, if the field is surrounded by correlated subjects, which still have closer empirical connections, or if the discipline is under the influence of men with an exceptionally well-developed taste. But there is a grave danger that the subject will develop along the line of least resistance, that the stream, so far from its source, will separate into a multitude of insignificant branches, and that the discipline will become a disorganized mass of details and complexities. In other words, at a great distance from its empirical source, or after much “abstract” inbreeding, a mathematical subject is in danger of degeneration. At the inception the style is usually classical; when it shows signs of becoming baroque, then the danger signal is up... In any event, whenever this stage is reached, the only remedy seems to me to be the rejuvenating return to the source: the re-injection of more or less directly empirical ideas. I am convinced that this was a necessary condition to conserve the freshness and the vitality of the subject and that this will remain equally true in the future. — John von Neumann
Towards Integrating Control and Information Theories: From Information-T heoretic Measures to Control Performance Limitations 控制论与信息论融合之初探：从信息论测度到控制性能极限 Song Fang, Jie Chen, Hideaki Ishii http://www.springer.com/cn/book/9783319492889 Chapter 1 There is an obvious analogy between the problem of smoothing the data to eliminate or reduce the effect of tracking errors and the problem of separating a signal from interfering noise in communications systems. — R. B. Blackman, H. W. Bode, and C. E. Shannon, “Data Smoothing and Prediction in Fire-Control Systems,” 1946 Chapter 2 The idea of a statistical message source is central to Shannon’s work. The study of random processes had entered into communication before his communication theory. There was a growing understanding of and ability to deal with problems of random noise. — J. R. Pierce, “The Early Days of Information Theory,” 1973 Chapter 3 All stable processes we shall predict. All unstable processes we shall control. — John von Neumann Chapter 4 As the signal power is increased from zero, we allot the power to the channels with the lowest noise. When the available power is increased still further, some of the power is put into noisier channels. The process by which the power is distributed among the various bins is identical to the way in which water distributes itself in a vessel, hence this process is sometimes referred to as “water-filling”. — T. M. Cover, J. A. Thomas, “Elements of Information Theory,” 1991 Chapter 5 I like to think of Bode’s integrals as conservation laws. They state precisely that a certain quantity—the integrated value of the log of the magnitude of the sensitivity function—is conserved under the action of feedback. The total amount of this quantity is always the same. It is equal to zero for stable plant/compensator pairs, and it is equal to some fixed positive amount for unstable ones... This applies to every controller, no matter how it was designed. Sensitivity improvements in one frequency range must be paid for with sensitivity deteriorations in another frequency range, and the price is higher if the plant is open-loop unstable. — G. Stein, “Respect the Unstable,” 2003 Chapter 6 In control and communication we are always fighting nature’s tendency to degrade the organized and to destroy the meaningful; the tendency, as Gibbs has shown us, for entropy to increase. — N. Wiener, “The Human Use of Human Beings,” 1950 Chapter 7 We know the past but cannot control it. We control the future but cannot know it. — Claude Shannon Chapter 8 It can also be shown that the Kalman filter extracts the maximum possible information about output data. If we form the residual between the measured output and the estimated output, we can show that for the Kalman filter the error (residual) is a white noise process, so there is no remaining dynamic information content in the error. — K. J. Astrom, R. M. Murray, “Feedback Systems: An Introduction for Scientists and Engineers,” 2010 Chapter 9 If everything seems under control, you’re just not going fast enough. — Mario Andretti
标签: 维纳
