In 1928, a researcher at Bell Labs named Ralph Hartley published a paper that would change the world. At the time, "information" was a vague, psychological concept that we couldn't measure. The transition from that to the current field of information theory has been exponential. Out of curiosity I went back and read "Transmission of Information" by Hartley. What I'm going to do here is talk about the paper.
From the abstract:
"A quantitative measure of information is developed which is based on physical as contrasted with psychological considerations."
That is the only sentence from the abstract that I am going to be writing about. The rest of the paper deals more with engineering applications and I am not interested in it.
In the first page what he does is give an overview of the paper and explain what he is attempting to do. On the second page he begins with the measurement of information, where he talks about the considerations involved in communication and sending messages to people. Then he discusses the psychological factors that need to be considered but that he wants to eliminate. Finally, we get to the main result of the paper, which is the quantitative expression for information. Basically, he says that you have two things: You have symbols and you have the act of picking out something from those symbols to say. What you get from this process are sequences.
He explains that these sequences can be used to define a measure. In defining this measure, he argues that we need the logarithm of the number of sequences. This is his final conclusion for the section and also where we end...the whole discussion builds up to the equation
\[
H = n \log s
\]
which says that the measure of information is the logarithm of the number of possible sequences.
Now let's start with page one.
“What I hope to accomplish in this direction is to set up a quantitative measure whereby the capacities of various systems to transmit information may be compared. In order to lay the groundwork for the more practical applications it will first be necessary to discuss a few somewhat abstract considerations.”
He says that as a commonly used word, information is a very elastic term. In everyday speech information has a very different meaning from the meaning he wants to give it in this paper. If we want a mathematical treatment, we need a more specific definition.
As a starting point he asks us to consider the factors involved in communication: whether communication is conducted by wire, radio, speech, writing, or any other method. The point he is making is that in the physical world we have some set of symbols that we assign meaning to and get information out of them.
He then describes the communication process. In any given communication the sender mentally selects a particular symbol and by some bodily motion, such as the motion of the vocal mechanism, causes the attention of the receiver to be directed to that particular symbol. By successive selections a sequence of symbols is brought to the receiver's attention.
Hartley then gives an example showing that communication in the physical sense does not necessarily imply that meaning has been transmitted. If two people speak different languages there is a very small chance that they will successfully communicate meaning, even though sounds are transmitted between them. He says he does not want to focus on this issue because it involves psychological interpretation, and his goal is to eliminate psychological factors. He suggests that we imagine a submarine telegraph cable. At one end there is a sender transmitting a message and at the other end there is a recorder receiving the signal.
If the signal is clear the symbols appear distinct. As the signal becomes more distorted it becomes harder to distinguish them. Eventually the signal becomes so degraded that it is impossible to tell which symbol was sent. Hartley argues that what matters in estimating the capacity of the system is not the meaning interpreted by the receiver, but whether the receiver can distinguish one symbol from another.
He writes that in estimating the capacity of the physical system to transmit information we should ignore the question of interpretation, make each selection perfectly arbitrary, and base our result on the possibility of the receiver distinguishing the result of selecting one symbol from the result of selecting another. By doing this the psychological factors are eliminated and it becomes possible to define a quantitative measure of information based only on physical considerations.
Next, he moves to the quantitative expression for information. Suppose we have ( s ) symbols. Suppose also that we make ( n ) selections from these symbols.
If we have three symbols and two selections we obtain
\[
3^2 = 9
\]
possible sequences.
In general, if there are ( s ) symbols and ( n ) selections, the number of possible sequences is
\[
s^n
\]
So, the number of possible symbol sequences is simply the number of symbols raised to the number of selections. Hartley then introduces the idea of primary and secondary symbols. Primary symbols are the basic signals that cannot be broken down further. Secondary symbols are combinations of primary symbols. For example, electrical signal states could be primary symbols while combinations of those signals represent characters or letters. Hartley argues that it does not matter whether we describe the system using primary symbols or secondary symbols. The amount of information transmitted is the same. Finally, he derives the logarithmic measure of information.
Let ( H ) be the quantity we want to measure.
He proposes that this quantity should be proportional to the number of selections ( n ). So we write
\[
H = k n
\]
Now consider two communication systems. If the number of possible sequences produced by the two systems is the same, then the amount of information transmitted by the two systems should also be the same.
Since the number of possible sequences is
\[
s^n
\]
this condition leads to the conclusion that the constant ( k ) must depend on the logarithm of ( s ).
Substituting this relationship gives the final result
\[
H = n \log s
\]
So, the measure of information is the logarithm of the number of possible sequences.
No comments:
Post a Comment