Back Grazing in Familiar Pastures; Shannon’s Milestone Book on Communication Theory

IMG_2499

Do you use the internet and personal communication devices such as cell phones? Since you are here, you must! Who doesn’t these days? One look at people in public places with eyes riveted on phone screens or tablets speaks to the popularity of personal communication. DSL (Direct Subscriber Line) services like AT&T’s U-Verse reliably bring broadband television and the Internet into our homes over lowly, antiquated, but ubiquitous twisted-pair phone wire connections. That miracle is only possible thanks to the power of modern digital communication theory.

The gospel of the engineering/mathematics that enable that capability is this 1949 book edition by Claude Shannon of AT&T’s famous Bell Telephone Laboratories. Its title: The Mathematical Theory of Communication. “Bell Labs” made immense contributions to our body of technical knowledge over many decades through its familiar, blue-wrappered Technical Journal. The authors of its featured papers include many of the most important scientists, engineers, and mathematicians of the past century.

Claude Shannon was one of them; the contents of his 1949 book, published by the University of Illinois Press, first appeared in the Bell System’s Journal in 1948. The paper’s unique and important approach to reliably sending electrical/optical signals from one point (the source) to another (the destination) through a “channel” was instrumental in realizing today’s communication miracles. Shannon’s methods are not limited to this or that specific channel technology; rather, his work applies to virtually all forms of communication channels – from digital audio/video disks, to AM/FM broadcasting, to the technology of the Internet, itself. The wide applicability of Shannon’s insights to communication systems as diverse as Samuel Morse’s original telegraph system and modern satellite communications is quite remarkable and underlines the importance of his findings.

Claude_Elwood_Shannon_(1916-2001)[1]Interestingly, some of the foundation for Shannon’s ideas emanated from the early design of Morse’s first telegraph system which began service in 1844 between Washington and Baltimore. The first message sent over that line was Morse’s famous utterance in Morse code to his assistant, Alfred Vail: “What hath God wrought?” While Claude Shannon is fairly identified as the “father of communication theory” thanks to his famous 1948/49 publications, there were also many grandfathers! Most of them made valuable contributions to the speed and reliability of early communication vis-à-vis the telegraph and early telephony, as pioneered by Alexander Graham Bell. One of the early, key contributors to communication technology was R.V.L. Hartley who, in the July, 1928 issue of the Bell System Technical Journal, published a very original treatise titled Transmission of Information. This paper of Hartley’s and one in the 1924 Journal by Harry Nyquist were acknowledged by Shannon as prime foundational sources for his later ideas.

Hartley Bell Journal_2 1928 Journal w/ Hartley’s Paper: Transmission of Information

What Were Claude Shannon’s Contributions?

A brief but inclusive answer comes from the well-regarded book of J.R. Pierce, Symbols, Signals and Noise. I quote, here:

“The problem Shannon set for himself is somewhat different. Suppose we have a message source which produces messages of a given type, such as English text. Suppose we have a noisy communication channel of specified characteristics. How can we represent or encode messages from the message source by means of electrical signals so as to attain the fastest possible transmission over the noisy channel? Indeed, how fast can we transmit a given type of message over a given channel without error? In a rough and general way, this is the problem that Shannon set himself and solved.”

Although Shannon impressively refined our concepts regarding the statistical nature of communication, Samuel Morse and his assistant, Alfred Vail, had, long ago, recognized statistical ramifications, and that fact was reflected in their telegraph code. Notably, they made certain that the most commonly used letters of the alphabet had the simplest dot/dash implementations in the Morse code – to minimize the overall transmission time of messages. For example, the most commonly used letter “e” was assigned a short, single “dot” as its telegraphic representation. Reportedly, this “code optimization” task was handled by Vail who merely visited a local printing shop and examined the “type bins,” equating the frequency of use in print for a specific letter to the size of its type bin! The printing industry had a good handle on text statistics of the English language long before electrical technology arrived on the scene. The specific dot/dash coding of each letter for Morse’s code proceeded accordingly. From that practical and humble beginning, statistical communication theory reached full mathematical bloom in Shannon’s capable hands. As in Morse’s time, coding theory remains an important subset of modern digital communication theory.

Revisiting Communication Theory:
Grazing Once Again in Technical Pastures of the Past

The most satisfying portion of my engineering career came later – particularly the last ten years – when I became immersed in the fundamentals of communication theory while working in the computer disk drive industry, here in Silicon Valley. My job as electrical engineer was to reliably record and retrieve digital data using the thin, magnetic film deposited on spinning computer disks. As the data demands of personal computers rapidly increased in the decade of the 1990’s, the challenge of reliably “communicating” with the magnetic film and its increasingly dense magnetically recorded bits of data was akin to the DSL task of cramming today’s broadband data streams down the old, low-tech telephone twisted-pair wires which have been resident in phone cables for many decades. Twisted-pair wires make a very poor high speed communication cable compared to coaxial cable or the latest fiber-optic high-speed cable, but they had one huge advantage/motivation for DSL’s innovators: They already fed most every home and business in the country!

I retired from engineering in 2001 after a thirty-seven year career and now find myself wandering back to “technical pastures of the past.” During the last ten and most exciting years of my career, I came to know and work with two brilliant electrical engineering PhDs from Stanford University. They had been doctoral students there under Professor John Cioffi who is considered the “father of DSL.” The two were employed by our company to implement the latest communication technologies into disk storage by working closely with our product design teams. Accordingly, the fundamental communication theories that Shannon developed which enabled the DSL revolution were applied to our disk drive channels to increase data capacity/reliability. Under the technical leadership of the two Stanford PhDs, our design team produced the industry’s first, successful production disk drive utilizing the radically new technology. IBM had preceded our efforts somewhat with their “concept” disk drive, but it never achieved full-scale production. After the successful introduction of our product, the disk drive industry never looked back, and, soon, everyone else was on-board with the new design approach known as a “Partial Response/Maximum Likelihood” channel.

I always appreciated the strong similarities between the technology we implemented and that which made DSL possible, but I recently decided to learn more. I purchased a book, a tech-bible on DSL, co-authored in 1999 by Professor Cioffi. Thumbing through it, I recognize much of the engineering it contains. I have long felt privileged that I and our design team had the opportunity to work with the two young PhD engineers who studied with Cioffi and who knew communication theory inside-out. Along with their academic, theoretical brilliance, the two also possessed a rare, practical mindset toward hardware implementation which immensely helped us transfer theory into practice – in the form of a commercially successful, mass-produced computer product. Everyone on our company staff liked and deeply respected these two fellows.

When the junior of the two left our company as our drive design was nearing full production, he circulated a company-wide memo thanking the organization for his opportunity to work with us. He cited several of us engineers by name for special thanks, an act which really meant a lot to me…and, surely, to my colleagues – an uncommon courtesy, these days, and a class act in every sense of the word!

Even in this valley of pervasive engineering excellence, that particular experience was one of a select few during my career which allowed me a privileged glimpse into the rarified world of “top-minds” in engineering and mathematics – the best of the best. A still-higher category up the ladder of excellence and achievement is that of “monster minds” (like Einstein, Bohr, and Pauli) which the Nobel physicist, Richard Feynman, so humorously wrote about in his book, Surely You’re Joking Mr. Feynman. A very select club!

The recent event which tuned me in, once again, to this technology and my past recollections was the subject of my May 2, 2015 blog post, Two Books from 1948 : Foundations of the Internet and Today’s Computer Technology (click on the link). In it, I describe the incredible good fortune of stumbling upon one of the two scarce, foundational books on communication theory and computer control: Cybernetics by Norbert Wiener. More recently, I acquired a nice copy of Claude Shannon’s 1949 first edition, The Mathematical Theory of Communication (the other book). That one came at no give-away price like my copy of Cybernetics, but, given its importance, it still represents a bargain in my mind.

IMG_2479 PSLike many engineers who are familiar with Shannon and his influence, I had never read his book, although I had taken a course on statistical communication theory in my master’s degree program over 45 years ago. Unlike many engineers, today, whose gaze is fixed only upon the present and the future, I have a deep interest in the history of the profession and a healthy respect for its early practitioners and their foundational work. Accordingly, I have been brushing off some technical rust and am now immersed in Shannon’s book for the first time and in the subject material, once again.

Old, familiar pastures – a bit like coming home, again, to peacefully graze. While the overall “view” improves with age and experience, the “eyesight” is not so keen, anymore. But my curiosity is up, yet again, and I will soldier-on through the technical difficulties and see where that takes me, all the while relishing the journey and the landscape.

Two Books from 1948 : Foundations of the Internet and Today’s Computer Technology

IMG_2350_PSI had a very good day recently. I bought a beautiful $400 book for $20 in Ventura, California! It also happens to be a very important book – literally, a foundation work for today’s Internet and our computer-based technological age . The book is titled: Cybernetics. While traveling south three weeks ago to the annual Santa Clarita Cowboy Music Festival – an annual event for us (see last week’s post) – Linda and I stopped in downtown Ventura, California – also an annual ritual. As always, we had lunch at our favorite hole-in-the-wall Italian restaurant and browsed for a bit at one of our favorite used bookstores in town, The Calico Cat.

Often we find a book or two in this little shop, and sometimes, we do not. After perusing various sections for close to an hour with no luck, I moved to the science/math section. As I ran my eyes along the shelves, I recognized many of the books they held. My scanning gaze froze as I came upon a pristine little book titled Cybernetics: or Control and Communication of the Animal and the Machine, written by Norbert Wiener. Wiener was a mathematics prodigy in his youth who enjoyed a long and distinguished career as a professor of mathematics at M.I.T., the Massachusetts Institute of Technology.

The book’s title reflects its ground-breaking categorization of the messaging and control systems inherent in the two closely-related realms of computer control and the human/animal brain/body connection. Cybernetics appeared at precisely the same time as the first, large-scale electronic computers, and this little book was instrumental in determining the future path of computing and control (robotics).

I immediately recognized the title and author as very important, but could this be the 1948 first edition – a book I knew to be of considerable value? I excitedly pulled it from the shelf and opened to the verso of the title page which stated “Second printing. November, 1948.” I was holding the second printing of the first U.S. edition printed by John Wiley and Sons, Inc. There was a companion edition of the text published in Paris by Hermann et CIE, also in 1948.

I became very excited and called Linda over to show her the book and explain, “I believe this book is worth several hundred dollars in the book trade: It is a very famous, seminal work in communication and control engineering. For a copy in the like-new condition of this one, $400 is easily a realistic value on the market. The apparent penciled price for this pristine copy with an almost perfect dust jacket: $25! The store’s owner called my attention to the fact I misread the price which was actually only $20. With no hint of hesitation, I coolly announced, “I’ll take it!” The book’s original price, still on the dust-jacket: $3! Understandably, the store owners had no clue as to the book’s engineering/mathematical significance to today’s Internet and computer technology.

For a retired electrical engineer, like myself, finding this little book in such perfect condition at such a price is akin to tripping over a diamond protruding from the footpath. My many years in Silicon Valley spent designing computer disk drives all ultimately stemmed from a very few foundational works (books and technical papers) such as this one. Summoning my engineering background, I can read and understand the material in this book –while difficult, it does not require a PhD in Mathematics. That is the beauty of a foundational technical work such as this – profound, yet accessible to most engineers and scientists – given some effort.

The “Other” Book

11120467054[1]There exists another similarly concise book whose pedigree exceeds even that of Cybernetics. That book was authored by Claude Shannon at the Bell Laboratories and titled The Mathematical Theory of Communication. Interestingly, the book was published in 1949 after being first introduced as a technical paper in the Bell Systems Journal of 1948 – the same year that Cybernetics was published.

Dare I press my luck and hope to find a similar bargain for Mr. Shannon’s book? Not likely to happen, but it would be the perfect complement to Wiener’s little volume.

Shannon’s book elegantly achieves the unenviable task of defining “information” in mathematical terms and in a manner which lends itself to quantifying the maximum flow of  information possible over a given communication channel such as the Internet or the radio/television airwaves, to cite two of many possible real-life applications. Reading and decoding the magnetically recorded binary bits of data (1’s and 0’s) stored on computer disk drives occurs in the “read channel” of the drive electronics, as we engineers in the industry referred to it. All such applications concerning “communications” succumb to the mathematics presented by Claude Shannon in this little volume containing a mere 117 pages! Shannon’s methods are equally applicable to yesterday’s analog channels (radio transmissions, for example) and today’s pervasive digital implementations (computers, the Internet, et all).

The next time we are passing through Ventura, I will keep a sharp eye out for this second book and any other bargains like Cybernetics! Good fortune usually takes luck, but when good luck comes knocking, one needs to recognize the sound!