Information theory
The information theory is a mathematical theory in the field of probability theory and statistics , which the US-American mathematician Claude Shannon back. It deals with terms such as information and entropy , information transfer , data compression and coding, and related topics.
In addition to mathematics, computer science and telecommunications , the theoretical consideration of communication through information theory is also used to describe communication systems in other areas (e.g. media in journalism , the nervous system in neurology , DNA and protein sequences in molecular biology , knowledge in information science and documentation ).
Shannon's theory uses the term entropy to characterize the information content (also called information density) of messages. The more irregular a message is, the higher its entropy. In addition to the concept of entropy, the Shannon-Hartley law according to Shannon and Ralph Hartley is fundamental for information theory . It describes the theoretical upper limit of the channel capacity , i.e. the maximum data transmission rate that a transmission channel can achieve without transmission errors depending on the bandwidth and the signal-to-noise ratio .
history
Claude Shannon in particular made significant contributions to the theory of data transmission and probability theory in the 1940s to 1950s .
He wondered how one can ensure loss-free data transmission via electronic (now also optical) channels . It is particularly important to separate the data signals from the background noise.
In addition, attempts are made to identify and correct errors that have occurred during transmission. For this it is necessary to send additional redundant data (i.e. no additional information carrying) data in order to enable the data receiver to verify or correct data.
It is doubtful, and also not claimed by Shannon, that his study A Mathematical Theory of Communication ("Information Theory ") published in 1948 is of substantial importance for questions outside of communications engineering. The concept of entropy that he uses and is connected to thermodynamics is a formal analogy for a mathematical expression. In general, information theory can be defined as engineering theory on a high level of abstraction. It shows the trend towards the scientification of technology, which led to the development of engineering.
The reference point of Shannon's theory is the accelerated development of electrical communications technology with its forms of telegraphy, telephony, radio and television in the first half of the 20th century. Before and next to Shannon, Harry Nyquist , Ralph Hartley and Karl Küpfmüller also made significant contributions to the theory of communications engineering. Mathematical clarifications of relevance to information theory were provided by Norbert Wiener , who also helped it to gain considerable publicity in the context of his reflections on cybernetics.
An overarching question for communications engineers was how economically efficient and interference-free communications can be achieved. The advantages of modulation have been recognized; H. changing the form of the message by technical means. In the technical context, two basic forms of messages - continuous and discrete - can be distinguished. These can be assigned the common forms of presentation of information / messages: writing (discrete), language (continuous) and images (continuous).
At the end of the 1930s, there was a technical breakthrough when, with the help of pulse code modulation , it was possible to discreetly display a message that was present as a continuum in a satisfactory approximation. With this method it became possible to telegraph speech. Shannon, who worked for Bell Telephone Laboratories, was familiar with technical developments. The great importance of his theory for technology lies in the fact that he defined information as a “physical quantity” with a unit of measurement or counting, the bit . This made it possible to quantitatively exactly compare the effort required for the technical transmission of information in various forms (sounds, characters, images), to determine the efficiency of codes and the capacity of information storage and transmission channels.
The definition of the bit is a theoretical expression of the new technical possibilities to transform different forms of representation of messages (information) into a common, for technical purposes advantageous representation of the information: A sequence of electrical impulses that can be expressed by a binary code. This is ultimately the basis for information technology on a digital basis, as well as for multimedia. That was known in principle with information theory. In practice, however, the digital upheaval in information technology only became possible later - combined with the rapid development of microelectronics in the second half of the 20th century.
Shannon himself describes his work as a "mathematical theory of communication". It expressly excludes semantic and pragmatic aspects of the information, ie statements about the "content" of transmitted messages and their meaning for the recipient. This means that a "meaningful" message is just as conscientiously transmitted as a random sequence of letters. Although the Shannon theory is usually referred to as "information theory", it does not make any direct statement about the information content of transmitted messages.
More recently, attempts have been made to determine the complexity of a message no longer just by looking at the data statistically , but rather to look at the algorithms that can generate this data. Such approaches are in particular the Kolmogorow complexity and the algorithmic depth , as well as the algorithmic information theory of Gregory Chaitin . Classical information concepts sometimes fail in quantum mechanical systems. This leads to the concept of quantum information .
Information theory provides mathematical methods for measuring certain properties of data. The concept of information from information theory has no direct reference to semantics , meaning and knowledge , since these properties cannot be measured using information-theoretical methods.
See also
literature
- Claude E. Shannon: A mathematical theory of communication . Bell System Tech. J., 27: 379-423, 623-656, 1948. (Shannon's seminal paper)
- Claude E. Shannon, Warren Weaver: Mathematical Foundations of Information Theory , [Dt. Translated from The mathematical theory of communication by Helmut Dreßler]. - Munich, Vienna: Oldenbourg, 1976, ISBN 3-486-39851-2
- NJA Sloane, AD Wyner: Claude Elwood Shannon: Collected Papers ; IEEE Press, Piscataway, NJ, 1993.
- Christoph Arndt: Information Measures, Information and its Description in Science and Engineering (Springer Series: Signals and Communication Technology), 2004, ISBN 978-3-540-40855-0 , springer.com
- Siegfried Buchhaupt: The importance of communications technology for the development of an information concept in technology in the 20th century. In: Technikgeschichte 70 (2003), pp. 277–298.
Textbooks
- Holger Lyre: Information Theory - A Philosophical-Scientific Introduction , UTB 2289
- Werner Heise, Pasquale Quattrocchi: Information and Coding Theory. Mathematical foundations of data compression and backup in discrete communication systems , 3rd edition, Springer, Berlin-Heidelberg 1995, ISBN 3-540-57477-8
- John R. Pierce : An Introduction to Information Theory: Symbols, Signals and Noise ; Dover Publications, Inc., New York, second edition, 1980.
- W. Sacco, W. Copes, C. Sloyer, and R. Stark: Information Theory: Saving Bits ; Janson Publications, Inc., Dedham, MA, 1988.
- Solomon Kullback: Information Theory and Statistics (Dover Books on Mathematics), 1968;
- Alexander I. Khinchin: Mathematical Foundations of Information Theory ;
- Fazlollah M. Reza: An Introduction to Information Theory , 1961;
- Robert B. Ash: Information Theory , 1965
- Thomas M. Cover, Joy A. Thomas: Elements of Information Theory (Wiley Series in Telecommunication), 1991;
Popular science introductions
- William Poundstone: The Formula of Happiness
