Why Information is Logarithmic: Hartley’s 1928 Insight

 In 1928, a researcher at Bell Labs named Ralph Hartley published a paper that would change the world. At the time, "information" was a vague, psychological concept that we couldn't measure. The transition from that to the current field of information theory has been exponential. Out of curiosity I went back and read "Transmission of Information" by Hartley. What I'm going to do here is talk about the paper.

From the abstract:

"A quantitative measure of information is developed which is based on physical as contrasted with psychological considerations." 

That is the only sentence from the abstract that I am going to be writing about. The rest of the paper deals more with engineering applications and I am not interested in it.

In the first page what he does is give an overview of the paper and explain what he is attempting to do. On the second page he begins with the measurement of information, where he talks about the considerations involved in communication and sending messages to people. Then he discusses the psychological factors that need to be considered but that he wants to eliminate. Finally, we get to the main result of the paper, which is the quantitative expression for information. Basically, he says that you have two things: You have symbols and you have the act of picking out something from those symbols to say. What you get from this process are sequences.

He explains that these sequences can be used to define a measure. In defining this measure, he argues that we need the logarithm of the number of sequences. This is his final conclusion for the section and also where we end...the whole discussion builds up to the equation

\[
H = n \log s
\]

which says that the measure of information is the logarithm of the number of possible sequences.

Now let's start with page one.

“What I hope to accomplish in this direction is to set up a quantitative measure whereby the capacities of various systems to transmit information may be compared. In order to lay the groundwork for the more practical applications it will first be necessary to discuss a few somewhat abstract considerations.”

He says that as a commonly used word, information is a very elastic term. In everyday speech information has a very different meaning from the meaning he wants to give it in this paper. If we want a mathematical treatment, we need a more specific definition. 

As a starting point he asks us to consider the factors involved in communication: whether communication is conducted by wire, radio, speech, writing, or any other method. The point he is making is that in the physical world we have some set of symbols that we assign meaning to and get information out of them.

He then describes the communication process. In any given communication the sender mentally selects a particular symbol and by some bodily motion, such as the motion of the vocal mechanism, causes the attention of the receiver to be directed to that particular symbol. By successive selections a sequence of symbols is brought to the receiver's attention.

Hartley then gives an example showing that communication in the physical sense does not necessarily imply that meaning has been transmitted. If two people speak different languages there is a very small chance that they will successfully communicate meaning, even though sounds are transmitted between them. He says he does not want to focus on this issue because it involves psychological interpretation, and his goal is to eliminate psychological factors. He suggests that we imagine a submarine telegraph cable. At one end there is a sender transmitting a message and at the other end there is a recorder receiving the signal.

If the signal is clear the symbols appear distinct. As the signal becomes more distorted it becomes harder to distinguish them. Eventually the signal becomes so degraded that it is impossible to tell which symbol was sent. Hartley argues that what matters in estimating the capacity of the system is not the meaning interpreted by the receiver, but whether the receiver can distinguish one symbol from another.

He writes that in estimating the capacity of the physical system to transmit information we should ignore the question of interpretation, make each selection perfectly arbitrary, and base our result on the possibility of the receiver distinguishing the result of selecting one symbol from the result of selecting another. By doing this the psychological factors are eliminated and it becomes possible to define a quantitative measure of information based only on physical considerations.

Next, he moves to the quantitative expression for information. Suppose we have ( s ) symbols. Suppose also that we make ( n ) selections from these symbols.

If we have three symbols and two selections we obtain

\[
3^2 = 9
\]

possible sequences.

In general, if there are ( s ) symbols and ( n ) selections, the number of possible sequences is

\[
s^n
\]

So, the number of possible symbol sequences is simply the number of symbols raised to the number of selections. Hartley then introduces the idea of primary and secondary symbols. Primary symbols are the basic signals that cannot be broken down further. Secondary symbols are combinations of primary symbols. For example, electrical signal states could be primary symbols while combinations of those signals represent characters or letters. Hartley argues that it does not matter whether we describe the system using primary symbols or secondary symbols. The amount of information transmitted is the same. Finally, he derives the logarithmic measure of information.

Let ( H ) be the quantity we want to measure.

He proposes that this quantity should be proportional to the number of selections ( n ). So we write

\[
H = k n
\]

Now consider two communication systems. If the number of possible sequences produced by the two systems is the same, then the amount of information transmitted by the two systems should also be the same.

Since the number of possible sequences is

\[
s^n
\]

this condition leads to the conclusion that the constant ( k ) must depend on the logarithm of ( s ).

Substituting this relationship gives the final result

\[
H = n \log s
\]

So, the measure of information is the logarithm of the number of possible sequences.

The Library Bar and Sherlock Holmes

 

The Library Cafe and Bar, Madison, WI, USA

Back in grad school, I used to frequent a bar called “The Library.” They had books, food, and drinks, and the music was always great. There was nothing I enjoyed more than showing up there after a long day at work, finding a spot under the bright lights, and reading Sherlock Holmes. The walls were lined with actual bookshelves, but I usually stuck to the hard wooden chairs with my own worn paperbacks.

If you would have told me back then that I would eventually be seriously attempting to learn math that wasn't essential for my work, I would’ve laughed in your face. My way of thinking felt completely incompatible with the rigid world of mathematics. I saw myself as a person of prose and puzzles, not proofs and polynomials.

My perspective has changed over the last five years. I have slowly begun to realize that the thrill of a Holmesian deduction is not so different from the thrill of a mathematical discovery. 

The Long Way to Understanding (Part 2)

This post was polished using ChatGPT

The turning point did not arrive as a sudden revelation. It arrived as a conversation.

Rahul Roy had been a topper in my class, the kind of student whose path seemed clear and inevitable. Around the time he entered IIT Bombay, our lives moved in different directions academically, yet somehow that was when we began to really talk. It was just after Holi when those conversations started to stretch late into the night. Hours would pass without either of us noticing. At first we spoke about subjects, exams, ideas. Slowly, those discussions became something deeper. He asked questions about how I approached problems, how I read, how I thought. And somewhere in those exchanges, he saw a version of me I could not yet see myself.

Until then, I had quietly accepted a story about who I was: someone curious but inconsistent, interested in ideas but unable to execute them well. Rahul did not accept that version. He never said it directly, but his patience implied something else. He treated me as if I were capable of much more than my recent years suggested.

Taking a gap year felt like standing at the edge of a cliff. It meant admitting that the path I had been on was not working, and choosing uncertainty over momentum. But the more we spoke, the more that choice began to feel less like failure and more like a reset. I decided to attempt the entrance exams again, not out of desperation but out of a quiet belief that maybe I had not yet learned how to learn.

Rahul did not just help me with subjects. He taught me how to think.

We began with physics. At first my approach was scattered, jumping between concepts without grounding. He slowed everything down. Focus on definitions. Understand the structure of a problem before touching the algebra. Question each assumption. Those sessions felt less like tutoring and more like reshaping the way my mind moved through complexity. Gradually, something shifted. Problems that once felt overwhelming began to look like sequences of steps. Mastery did not arrive overnight, but I could feel the fog lifting.

Organic chemistry came next. It had always seemed chaotic to me, a collection of reactions without logic. Rahul showed me patterns instead of memorization. Mechanisms became stories. Electrons moved with intention. For the first time, chemistry felt coherent. From there, the rest of the subject began to fall into place, each piece connecting to another until the discipline stopped feeling like a wall and started feeling like a landscape.

What stayed with me most was his consistency. He never gave up on me, even when I doubted myself or drifted into old habits. While managing the intensity of IIT academics, he still found time to guide me, to correct small mistakes, to insist on detail and clarity. He pushed me to focus, to slow down, to respect precision. Bit by bit, my mind felt less scattered and more directed, as if someone had adjusted the lens through which I saw problems.

Looking back, I realize that the real transformation was not just academic. It was relational. Someone believed in a version of me that I had stopped imagining. That belief created a space where change felt possible. The gap year stopped feeling like an admission of defeat and began to feel like a deliberate rebuilding.

There were still difficult days. There were moments when progress felt fragile. But something fundamental had shifted. Instead of reading endlessly without direction, I was learning to engage deeply with material, to wrestle with it patiently until it yielded understanding. Rahul did not simply teach me subjects; he helped shape a way of thinking that felt stronger, more precise, more grounded.


For the first time in years, effort began to feel meaningful. Not frantic or desperate, but focused. And somewhere within those long nightly conversations, I started to believe that the curiosity I had carried since Rishi Valley could coexist with discipline. That the mind I once saw as unfocused could become something powerful when guided with patience and care.

The Long Way to Understanding (Part 1)

This post was polished using chatGPT

I did not begin with equations. I began with books.

Ninth and tenth grade at Rishi Valley felt less like school and more like a long corridor lined with ideas that were waiting to be opened. The campus itself had a strange quietness. There was space to think, space to wander, and I filled most of that space by reading. I read constantly. Not because I was brilliant or disciplined, but because I was hungry for something I could not name yet. Stories, science, philosophy, anything that hinted at a deeper structure beneath the world felt irresistible.

Math, ironically, was not one of the places where I felt at home. I was not terrible, but I was never the student who solved problems quickly or effortlessly. Numbers did not arrange themselves into clean patterns in my mind. While others seemed to move through algebra with confidence, I felt like I was always slightly behind, translating a language everyone else already spoke. At the time I did not think of it as a struggle. I thought of it as a quiet embarrassment that I hid behind curiosity in other subjects.

Physics, though, felt different. It did not ask me to be fast. It asked me to wonder.

I remember picking up QED by Feynman and reading it with a seriousness that probably surprised my teachers. I did not understand everything. Sometimes I barely understood half of it. But the idea that light could be explained through paths, probabilities, and tiny arrows felt like discovering a secret grammar of reality. I kept asking physics questions, sometimes clumsy ones, sometimes repetitive. I was not chasing grades. I was chasing the feeling that somewhere beneath the confusion there was an order I could almost touch.

One evening the school staged a play about a conversation between Niels Bohr and Werner Heisenberg. It centered on the uncertainty principle, but it did not feel like a lecture. It felt like watching two minds circle around a mystery they could never fully resolve. The dialogue moved slowly, almost carefully, as if each word carried the weight of an entire century of thought. I remember sitting there completely still, feeling something shift inside me. It was not just the physics. It was the realization that ideas could be alive, that disagreement could be intimate, that uncertainty itself could be meaningful rather than frightening.

Until that moment, I had thought of science mostly as answers. That play showed me science as a conversation that never ends.

Looking back, it is strange to think that I was so captivated by quantum ideas while still feeling mediocre in math class the next morning. I could be mesmerized by probability amplitudes at night and then struggle with a simple problem set during the day. At the time it felt like a contradiction. Now I think it was the beginning of a pattern that would follow me for years. I was drawn to the deepest questions before I had the tools to approach them. The imagination arrived before the technique.

Rishi Valley gave me permission to live in that gap for a while. I read under trees, in dorm rooms, in quiet corners where time slowed down. I was not building a clear path toward anything. I was assembling fragments. Physics questions that had no immediate answers. Scenes from a play that refused to leave my mind. A book by Feynman that felt like both a doorway and a challenge.

If I try to locate the exact moment when curiosity stopped being casual and became something closer to devotion, it might be that evening with Bohr and Heisenberg. Watching them argue about uncertainty made me realize that confusion was not a failure. It was the terrain itself.


At fifteen, I did not yet know that I would struggle with mathematics for years, or that learning to think clearly would become a much more personal journey than I expected. I only knew that there was a world made of questions, and that I wanted to stand inside it, even if I did not fully belong there yet.

Shaving my head and other things

 I recently shaved my head to fulfill a religious vow in Tirupathi, India. It's been an interesting experience. I love the feeling of wind on my scalp, and the fact that I don't have to worry about combing and styling my hair every day. The only thing I'm worried about is the in-between short hair phase of growing my hair out. So I asked ChatGPT to make a prediction of what I would look like. Here's a picture of me with short hair as imagined by ChatGPT:

And one of me bald!

Quadratic Residues

Sometimes we might be interested in the question of whether certain numbers are congruent to the square of some other number modulo a prime. One option to answer this question is trial and error, for instance the question of whether 3 is congruent to the square of a number mod 7:

We can see that there’s no 3 here in this table. What I love about Silverman is that he has us “experiment” with different numbers to really see the patterns we care about. So in the spirit of this we can try another example.

1, 2, and 4 are Quadratic Residues (QR), while 3, 5, and 6 are Non-residues (NR). 3 is not a square mod 7. In our p=11 experiment, Quadratic Residues are 1, 3, 4, 5, 9. The Quadratic Non-Residues are 2, 6, 7, 8, 10. If you were a number living in the world of Modulo 11, and you were a 7, you could never be the result of a square. You are an impossible value.

So in a nutshell, that’s what quadratic residues are. Easy enough to understand, but why do we care about them? This is one of those cool cases in math where doing something one way is very easy, but reversing the operation is incredibly hard. For a single prime, these patterns are easy to find. But if we multiply two massive primes together and create a composite number, finding out if a number is a square becomes a hard puzzle for anyone who doesn't know the secret prime factors. This trapdoor—easy to do one way, hard to reverse—is the engine behind Zero Knowledge Proofs and the encryption that keeps your credit card safe online

An interview with a lawyer on Public Policy and Law

 For a long time, my close relative's work in law and public policy felt foreign to me. It seemed distant from the kinds of problems I was used to thinking about, governed by a different language and a different set of concerns. Over time, I’ve come to realize that it isn’t foreign at all. At its core, her work is about choosing objectives and then figuring out how to maximize them under real constraints. The constraints happen to be legal, institutional, and political rather than mathematical, but the structure of the problem is the same. I wanted to understand how legal training shapes that way of thinking, and how it helps turn ideals into something that can actually operate in the world.

Abhijit Banarjee and Esther Duflo 

Q: Good evening! Thanks so much for doing this. When you first started law school, what kind of thinking did you have to unlearn?

I don’t think I had to unlearn anything in a strong sense. When you’re that young, you don’t yet have a very fixed or fully formed way of thinking. I did have some basic critical thinking skills, but what law school really taught me was how to structure my thinking. You learn how to break down arguments, identify assumptions, and reason systematically within a framework. That kind of disciplined structure was the biggest shift for me.

---

Q: What is public policy?

In India, the policy space is often confused with the development sector, and the two terms are sometimes used interchangeably. But they are not the same. Public policy spans multiple domain areas, such as technology policy, environmental policy, labor policy, and so on. These are content areas.

Separately, there are different approaches to policy. You can work on policy from within a tech startup, through grassroots organizations, via government, or by doing direct impact work. Some policy work is nonprofit, some is for-profit. Because of this overlap in actors and approaches, public policy often gets conflated with development work, but it is broader and more structural than that.

---

Q: Why does your law degree give you an advantage in policy?

Very directly, public policy is closely tied to governance, and governance inevitably involves law. Any serious policy work requires interacting with legal frameworks at some level. Beyond that, a law degree prepares you with a strong set of transferable skills. It trains you to apply a critical lens to complex problems, to read carefully, to anticipate consequences, and to think about how abstract rules operate in real contexts. Those skills are valuable far beyond purely legal settings.

---

Q: What is the biggest difference between thinking legally and thinking in terms of public policy?

Legal thinking is usually focused on how the law applies in a specific context. It asks what the law says, how it has been interpreted, and how it operates in particular cases. Public policy thinking is more systems-level. It starts with frameworks and structures, and then looks at how individual laws fit within those systems.

How you regulate something matters enormously, even before you get to the law itself. Questions of design, incentives, implementation, and numeracy are foundational, and many of these are not legal questions at all. Law is an important tool, but it is not the only framework for solving public policy problems.

---

Q: How does a lawyer evaluate whether a policy will actually work once implemented?

This is a difficult question because it is an entire domain in itself. In practice, policy evaluation often relies on tools developed in economics and statistics rather than law. One common method is randomized control trials, which compare treatment and control groups to measure impact. This approach was recognized by the Nobel Prize awarded to Abhijit Banerjee and Esther Duflo.

Other tools include quantitative and qualitative surveys, field research, and empirical studies. Lawyers are not usually the ones running these analyses, but legal training helps in interpreting results, understanding limitations, and thinking carefully about how findings translate into regulatory or institutional changes.

---

Q: Where do you see well-intentioned policies fail because of legal structure?

This happens more often than we realize. First of all, when policies work, we rarely stop to notice them. But many failures stem from poor legal framing, particularly when laws do not intervene at the right point or intervene too much. Good legal framing involves knowing how and when to step in.

Another challenge is that laws are dynamic. They are constantly being challenged, amended, or struck down. Even a well-framed law can fail if it is not implemented as intended, or if the institutional capacity to enforce it is weak. So failure is not always about bad intent or bad ideas, but about the complexity of translating policy into durable legal form.

---

Q: How much of public policy is about ideals, and how much is about constraints?

It is always about both, at the same time. Public policy is deeply intersectional. You are constantly navigating trade-offs and trying to maximize certain ideals given very real constraints. In some sense, this is true of many areas of life, but it is especially visible in public policy.

That is actually what I love most about the field. You are never working in a purely abstract space. You are always balancing values, resources, politics, and feasibility, all at once.