Back Grazing in Familiar Pastures; Shannon’s Milestone Book on Communication Theory

Do you use the internet and personal communication devices such as cell phones? Since you are here, you must! Who doesn’t these days? One look at people in public places with eyes riveted on phone screens or tablets speaks to the popularity of personal communication. DSL (Direct Subscriber Line) services like AT&T’s U-Verse reliably bring broadband television and the Internet into our homes over lowly, antiquated, but ubiquitous twisted-pair phone wire connections. That miracle is only possible thanks to the power of modern digital communication theory.

The gospel of the engineering/mathematics that enable that capability is this 1949 book edition by Claude Shannon of AT&T’s famous Bell Telephone Laboratories. Its title: The Mathematical Theory of Communication. “Bell Labs” made immense contributions to our body of technical knowledge over many decades through its familiar, blue-wrappered Technical Journal. The authors of its featured papers include many of the most important scientists, engineers, and mathematicians of the past century.

Claude Shannon was one of them; the contents of his 1949 book, published by the University of Illinois Press, first appeared in the Bell System’s Journal in 1948. The paper’s unique and important approach to reliably sending electrical/optical signals from one point (the source) to another (the destination) through a “channel” was instrumental in realizing today’s communication miracles. Shannon’s methods are not limited to this or that specific channel technology; rather, his work applies to virtually all forms of communication channels – from digital audio/video disks, to AM/FM broadcasting, to the technology of the Internet, itself. The wide applicability of Shannon’s insights to communication systems as diverse as Samuel Morse’s original telegraph system and modern satellite communications is quite remarkable and underlines the importance of his findings.

Interestingly, some of the foundation for Shannon’s ideas emanated from the early design of Morse’s first telegraph system which began service in 1844 between Washington and Baltimore. The first message sent over that line was Morse’s famous utterance in Morse code to his assistant, Alfred Vail: “What hath God wrought?” While Claude Shannon is fairly identified as the “father of communication theory” thanks to his famous 1948/49 publications, there were also many grandfathers! Most of them made valuable contributions to the speed and reliability of early communication vis-à-vis the telegraph and early telephony, as pioneered by Alexander Graham Bell. One of the early, key contributors to communication technology was R.V.L. Hartley who, in the July, 1928 issue of the Bell System Technical Journal, published a very original treatise titled Transmission of Information. This paper of Hartley’s and one in the 1924 Journal by Harry Nyquist were acknowledged by Shannon as prime foundational sources for his later ideas.

 1928 Journal w/ Hartley’s Paper: Transmission of Information

What Were Claude Shannon’s Contributions?

A brief but inclusive answer comes from the well-regarded book of J.R. Pierce, Symbols, Signals and Noise. I quote, here:

“The problem Shannon set for himself is somewhat different. Suppose we have a message source which produces messages of a given type, such as English text. Suppose we have a noisy communication channel of specified characteristics. How can we represent or encode messages from the message source by means of electrical signals so as to attain the fastest possible transmission over the noisy channel? Indeed, how fast can we transmit a given type of message over a given channel without error? In a rough and general way, this is the problem that Shannon set himself and solved.”

Although Shannon impressively refined our concepts regarding the statistical nature of communication, Samuel Morse and his assistant, Alfred Vail, had, long ago, recognized statistical ramifications, and that fact was reflected in their telegraph code. Notably, they made certain that the most commonly used letters of the alphabet had the simplest dot/dash implementations in the Morse code – to minimize the overall transmission time of messages. For example, the most commonly used letter “e” was assigned a short, single “dot” as its telegraphic representation. Reportedly, this “code optimization” task was handled by Vail who merely visited a local printing shop and examined the “type bins,” equating the frequency of use in print for a specific letter to the size of its type bin! The printing industry had a good handle on text statistics of the English language long before electrical technology arrived on the scene. The specific dot/dash coding of each letter for Morse’s code proceeded accordingly. From that practical and humble beginning, statistical communication theory reached full mathematical bloom in Shannon’s capable hands. As in Morse’s time, coding theory remains an important subset of modern digital communication theory.

Revisiting Communication Theory:
Grazing Once Again in Technical Pastures of the Past

The most satisfying portion of my engineering career came later – particularly the last ten years – when I became immersed in the fundamentals of communication theory while working in the computer disk drive industry, here in Silicon Valley. My job as electrical engineer was to reliably record and retrieve digital data using the thin, magnetic film deposited on spinning computer disks. As the data demands of personal computers rapidly increased in the decade of the 1990’s, the challenge of reliably “communicating” with the magnetic film and its increasingly dense magnetically recorded bits of data was akin to the DSL task of cramming today’s broadband data streams down the old, low-tech telephone twisted-pair wires which have been resident in phone cables for many decades. Twisted-pair wires make a very poor high speed communication cable compared to coaxial cable or the latest fiber-optic high-speed cable, but they had one huge advantage/motivation for DSL’s innovators: They already fed most every home and business in the country!

I retired from engineering in 2001 after a thirty-seven year career and now find myself wandering back to “technical pastures of the past.” During the last ten and most exciting years of my career, I came to know and work with two brilliant electrical engineering PhDs from Stanford University. They had been doctoral students there under Professor John Cioffi who is considered the “father of DSL.” The two were employed by our company to implement the latest communication technologies into disk storage by working closely with our product design teams. Accordingly, the fundamental communication theories that Shannon developed which enabled the DSL revolution were applied to our disk drive channels to increase data capacity/reliability. Under the technical leadership of the two Stanford PhDs, our design team produced the industry’s first, successful production disk drive utilizing the radically new technology. IBM had preceded our efforts somewhat with their “concept” disk drive, but it never achieved full-scale production. After the successful introduction of our product, the disk drive industry never looked back, and, soon, everyone else was on-board with the new design approach known as a “Partial Response/Maximum Likelihood” channel.

I always appreciated the strong similarities between the technology we implemented and that which made DSL possible, but I recently decided to learn more. I purchased a book, a tech-bible on DSL, co-authored in 1999 by Professor Cioffi. Thumbing through it, I recognize much of the engineering it contains. I have long felt privileged that I and our design team had the opportunity to work with the two young PhD engineers who studied with Cioffi and who knew communication theory inside-out. Along with their academic, theoretical brilliance, the two also possessed a rare, practical mindset toward hardware implementation which immensely helped us transfer theory into practice – in the form of a commercially successful, mass-produced computer product. Everyone on our company staff liked and deeply respected these two fellows.

When the junior of the two left our company as our drive design was nearing full production, he circulated a company-wide memo thanking the organization for his opportunity to work with us. He cited several of us engineers by name for special thanks, an act which really meant a lot to me…and, surely, to my colleagues – an uncommon courtesy, these days, and a class act in every sense of the word!

Even in this valley of pervasive engineering excellence, that particular experience was one of a select few during my career which allowed me a privileged glimpse into the rarified world of “top-minds” in engineering and mathematics – the best of the best. A still-higher category up the ladder of excellence and achievement is that of “monster minds” (like Einstein, Bohr, and Pauli) which the Nobel physicist, Richard Feynman, so humorously wrote about in his book, Surely You’re Joking Mr. Feynman. A very select club!

The recent event which tuned me in, once again, to this technology and my past recollections was the subject of my May 2, 2015 blog post, Two Books from 1948 : Foundations of the Internet and Today’s Computer Technology (click on the link). In it, I describe the incredible good fortune of stumbling upon one of the two scarce, foundational books on communication theory and computer control: Cybernetics by Norbert Wiener. More recently, I acquired a nice copy of Claude Shannon’s 1949 first edition, The Mathematical Theory of Communication (the other book). That one came at no give-away price like my copy of Cybernetics, but, given its importance, it still represents a bargain in my mind.

Like many engineers who are familiar with Shannon and his influence, I had never read his book, although I had taken a course on statistical communication theory in my master’s degree program over 45 years ago. Unlike many engineers, today, whose gaze is fixed only upon the present and the future, I have a deep interest in the history of the profession and a healthy respect for its early practitioners and their foundational work. Accordingly, I have been brushing off some technical rust and am now immersed in Shannon’s book for the first time and in the subject material, once again.

Old, familiar pastures – a bit like coming home, again, to peacefully graze. While the overall “view” improves with age and experience, the “eyesight” is not so keen, anymore. But my curiosity is up, yet again, and I will soldier-on through the technical difficulties and see where that takes me, all the while relishing the journey and the landscape.