Communication extrème: Writing in the blink of an eye

You found covid-19 lockdown hard to live with? How about this experience of time-filling when you have literally nothing else to lose, sensorially speaking.

In 1996, the middle-aged French journalist Jean-Dominique Bauby had a stroke that left him conscious of the world’s activities yet totally cut off from everyday intercourse. He was “locked in” to his own perceptions of the world, able to see and hear what was going on, but unable to communicate with people around him using either language or hand gestures. A dumb, immobile but mentally active fly trapped on the wall of life.

The only way he could send a message to the outside world was by blinking the lashes of one eye. Using a simple code of single or serial blinks, he was eventually able to transmit meanings to his attendant conversationalists by blinking either “yes” or “no” to the relevant letters of the alphabet shown on a card and spoken aloud to him, so that words and sentences could be laboriously spelt out and messages communicated.

In the end, he blinked out a complete book[i] — an amusing yet poignant account of his life and condition, that became a bestseller. But the actual production process raises fascinating practical issues that go to the heart of language and technology.

In the first place, someone must have had a Eureka moment and decided to use an abecedarium to stimulate that faintly blinking eyelash. Bauby on his own may not have been able to communicate an unambiguous intention to communicate — frantic eye blinking, for example, might have been interpreted as a mere (fran)tic… Second, his support team were smart enough to list the letters of the French alphabet in descending order of frequency, and use exclusively upper case (no accents) items: ESARINTUL … ZYXKW, thereby speeding up input and reducing the added stress of an information transfer process based on the antique and arbitrary ABC system.

In some ways, the entire development of our real-time communication technologies mirrors such lock-in syndromes. You usually can’t just shout louder to communicate to someone miles away; you are, as it were, locked inside your own auditory circumference. Until the end of the 19th century, you needed some kind of substitute — a technology-driven ‘blink’ system such as written messages, smoke rings, flags, Chappe’s telegraph, or eventually Morse code that used either shared signing symbols or written language itself to get a message out to a mute world beyond the natural perimeter of acoustic speech or hand gestures. And finally came telephony that used an outgoing physical line to carry spoken words as impulses far beyond the reach of shouting through a megaphone, or posting/publishing written messages on village board.

All language technology, then, ultimately addresses a foundational human disability: As humans do not (yet) share brain space, we need the awkward, fuzzy mediation of linguistic codes to stream articulated ideas and feelings from one brain to the next via a media — speech, text, or less effectively, images. This suggests we could look upon the evolutionary advance offered by articulate language as a leap forward for humans, but a still poorly designed instrument for sharing cognitive insights. Especially as most of the world’s speakers are also “locked into” practicing only one or two languages of the many thousands spoken. But Bauby’s case is indeed extrème…

Morse and letter statistics

Back in the mid-19th century, when Wheatstone and Cooke set up the first railway telegraph system in England, and Samuel Morse invented the first telegraph system in the US, his Morse code — a machine encoding of certain human languages — was put to work as an alphabetic letter coding system (dots and dashes) designed around obvious facts about letter frequency for the world’s alpha-written languages.

For English, the most frequent letters had the shortest code items: E = one dot and T = one dash. Logically enough, Samuel Morse ‘discovered’ these frequencies by counting the number of stereotypes (solid or 3D — hence the “stereo” — metal letter shapes) kept in a printer’s type box. Compositors who made up pages of printable text were used to using many more “e” letters than “x” letters, for example — a simple statistical fact about English spelling. For Albanian type setters, it might have been the opposite.

Notice how that upper case series of letters works as a communication aid[ii]. Bauby’s French-speaking amanuenses (“hand” workers because in antiquity it was they who copied down spoken linguistic content on the local equivalent of paper) rightly ignored the print particularity of French accented letters such as é, è, à, ô, and ç, even though this decision would have occasionally caused ambiguity. Note that even today a search engine on the web will often ignore the semantic differences between capitalized and uncapitalized words.

Capital punishment doesn’t hurt

Uppercase writing of the alphabet for French and English needs just 26 letters, though French of course has several more lower-case accented letters (é, à, ç, è) which are also printed in government and similar documents with uppercase equivalents such as É or À). But they can be ignored because as humans we can rapidly predict meanings over ‘incomplete’ data such as those found in SMS messages or telegrams. The 26-letter alphabet works perfectly well in French if you want to communicate a message as quickly as possible over reduced bandwidth (i.e., eye blinks, telegraph wires, early computer screens, internet messaging) provided you have a human to decode and disambiguate the result the other end.

This is why most alphabetic languages used only capitals when printing telegrams, and again during the early years of computer inputting even for languages that often use far more complex sets of accented letters. By simplifying the code, as it were, to ASCII-like proportions, Bauby could save energy and gain time, while his interlocutors had to do the slight extra work of parsing what he meant and writing it out again in “full” as we say, for the published text[iii].

Other sufferers have been confronted by this extreme situation, of course. Towards the end of his short life the German philosopher Franz Rosenzweig suffered from the degenerative disease amyotrophic lateral sclerosis (Lou Gehrig’s disease). To keep writing, he obtained a specially equipped General Electric typewriter in 1923 to help him overcome the handicap of his weak muscles. To use the typewriter, he would pass a pointer over a disc of alphabetic letters, choose a letter, and the machine would then imprint his choice. When his muscles failed completely, he would — somewhat like Bauby — blink his work to his wife as she passed her finger over a board with the German alphabet on it.

It is also obvious that as Bauby started to blink a new word, his receivers would in many, perhaps most cases[iii] be able to guess the word intended and even subsequent words. Alphabetic writing can be up to 40 or 50% redundant: If in an English context and in Bauby mode I started blinking a new word with a W, there is a fairly broad range of options for the next letter with plenty of words beginning with WA, WE, WH, WI, WO, or WR, etc. But if I blink W and then R, I considerably reduce the search space for the word in question to WRA, WRE, WRO or WRI, enabling my amanuensis to point immediately to either N, C or T, and if T then guess the ending (writer, written, write, writes, writing) by computing intuitively from the linguistic context already delivered.

So in fact Bauby only had to blink out say 60% of his 70,000 or so word text, not the full word count. And in any case, preceding words would help predict the next one in a sentence. Language forms and patterns have been cognitively evolved to be fairly redundant to avoid the problem of losing meanings in noisy situations, and recent technologies can use data from users of machinery to benefit from this fact. The emergence of predictive keyboards for mobile devices has shown this to be a computable benefit.

Beyond the Latin alphabet

Let’s now run a few thought experiments around a semi-fictional Bauby. Extreme situations such as his have the unique virtue of allowing us to vary the parameters of the communication context and focus the mind on one aspect at a time. For example, had Bauby spoken Russian or Greek, he would have had no trouble in applying his yes/no blink method to a statistically organized alphabet, although different from our “roman” signs.

If he had been an Arabic or Hebrew writer, Bauby would also have sailed along right to left using the built-in shorthand design of the Semitic alphabets. His helper’s alphabet card would have shown lists of Arabic consonants and stressed or long vowels, not Evry vOwl n Evry wrd. The Arabic ajab alphabet may have many more distinct letter forms than the roman alphabet (even though it does not use an upper/lower case distinction), but this formal variation of Arabic letters depends on contexts where specific ligatures are used in both handwritten and print media to join one letter to another. This does not prevent legibility when speed and simplicity are at stake.

There is no actual account in his book that the real Bauby blinked abbreviated forms for words (e.g., leaving out predictable vowels), saving energy and letting his readers compute the full forms. But we can be pretty sure it would have happened. In any case, his readers would have guessed most words due to context after the first few letters, as we have suggested.

Chinese boxes

Widening the language circle even further, could Bauby have blinked out his book if he had been a native speaker of Mandarin Chinese? The Han languages are traditionally written using logographs — more or less complex visual “characters” that represent syllables that themselves represent the morphemes or meaning units of words. Traditional Chinese script does not typically encode the “sounds” of the language into a restricted set of pronounceable alphabetic letters, though pinyin offers a romanized alphabet for Mandarin today.

Many other writing systems use logographs as part of their toolkit: English glyphs such as ‘@’ are pronounced (but don’t spell[iv]) “at” (or historically “alias”), and ‘#’ indicates (but doesn’t spell) the contextual meanings “number”, “pound”, “hashtag” or a “musical sharp” in addition to its digital role for social media tags. All number signs — 1, 2, 3 etc. — derived from Arabic versions of Indian characters, are logographs that we share across many languages using specific word forms.

So in a logographic writing system such as Chinese you need several thousand separate items to cover all the morphemes in the language, be they single or multi-character words. How would Bauby’s carers have been able to display a list of all these characters for him to blink a binary choice as he tried to stitch together his request or tell his story?

My guess is that his short-term memory would have found it hard to keep his intended phrase in store while scanning through several hundred characters (taking minutes) indicated one by one with a stick by the amanuensis on the spelling board to find the character he needed. But Bauby could of course have used pinyin to refer to the spelling conventions of Han logographs, just as Chinese users type in Chinese words to a search engine using an alphabet input system on a mobile phone.

Indeed it is interesting to learn from his book that Bauby would lie there entombed in his bed, composing what he wanted to say and committing it to memory so that he could draw on his memorized version when the time came to blink out more text. I imagine that the receiver would then write out the message so that he had visual feedback about what he had been saying.

Rapid feedback from a memory store is a vital resource when composing utterances, and this is precisely one of the affordances of writing — you can check what you’ve just thought/said. Think how often people who are telling a story spontaneously for several minutes will get side-tracked and exclaim they have lost the thread. Where was I? And someone will offer a prompt.

Composition plus memorization might well be a typical adaptation in lock-in and related cognitive disabilities. David Masson (in The Life of John Milton, Volume 5, 1654–1660) reports how the English poet John Milton who had been blind for four years already “would compose verse upon verse at night in his head and then dictate them from memory to his aides (usually his daughters) in the morning” when composing/uttering/transmitting his major poem Paradise Lost. There is also evidence that Franz Rosenzweig would compose his work at night, memorize it and then type or later dictate the results next morning.

And how about this typographic cum phantasmagorical memory from the 19th century English writer Thomas de Quincey: “I became a distinguished compositor in the darkness, and with my aerial composing stick, sometimes I “set up” half a page of verses, that would be found tolerably correct if collated with the volume I never had in my hand but once….words revive before me in darkness and solitude, and they arrange themselves gradually into sentences, but through an effort sometimes of a distressing kind, to which I am in a manner forced to become a party.”

For a more recent case of Lou Gehrig’s disease, and the attendant need for a “memory palace” approach to night time cogitating, see this touching description from his essay Night by the late Tony Judt, an academic and writer who contracted the disease in 2008: “My memory — already quite good — has improved considerably, with the help of techniques adapted from the “memory palace” (…). But the satisfactions of compensation are notoriously fleeting. There is no saving grace in being confined to an iron suit, cold and unforgiving (cf. Bauby’s title — Scaphandre or ‘diving suit’).”

In a separate (non-lock-in) case, the UK scientist and historian of Chinese science Joseph Needham also had a photographic memory. Before the publication of his three- volume treatise on chemical embryology his wife recalled watching him lying awake in bed, mentally visualizing the book’s page proofs and then correcting in a notebook any errors or infelicities. Once this activity became too humdrum for him, he further occupied himself by translating the same pages into French, also in his head and then correcting any errors he could see in this newly translated text — a photographic “memory” of an imaginary text!! (see Simon Winchester, The Man Who Loved China (London, Harpers 2008).

Non sense

Now let’s tighten the sensorial screw. How about a Bauby who was not only cognitively locked-in but sightless as well? All he had as communication technology was his hearing and a capacity to blink. Could he still have “written” his book? Someone would have had to start the experiment by deciding to speak out loud the letters of the alphabet — eh, bee, see, dee, ee etc. — and watch when Bauby systematically double-blinked at the enunciation of certain letters. They would then have to deduce that Bauby is trying to spell out a meaningful message from these oral cues. The letters accumulate slowly into words that make sense, such as WOULD LIKE TO START CHAPTER TWO etc., suggesting the existence of intelligent life inside that inert diving suit of a body.

Since the speaker is the initiator of the communication of the messages, it is they who would propose the alphabet sounds first, though Bauby and his amanuensis might well somehow agree on a set of other blink signs that allow Bauby to optimize their conversation. For example, they can tell him: “if you want to end a paragraph, blink three times in rapid succession”, or “to start a new chapter blink four times” and so on. A metalanguage to talk about the communication process itself. They could also shift up from reading sounds to saying whole words after the first two or three letters have been spelt, and then read it all back to him for him to check. Theoretically, then, a blind Bauby would have been able to blink a book.

Now let’s give a final twist to the experiment by fast-reversing Bauby to Mesopotamia in the 3rd millennium before the common era (say 2,500 BCE). Human beings had not yet developed any comprehensive system of semantic exo-memory (aka writing) as we now know it, and certainly could not spell out words as an alphabetic or similar code to convey meanings shared by the tribe. Writing was about to emerge as semantic scratches on clay pots. The question then is whether some sort of writing technology is a necessary condition for such communication extrème (as we have tacitly assumed). Or could our Bauby still share some sort of meaningful dialogue with the world from the depths of his isolation?

Remember that in this case, Bauby can hear and see, and can use blinking as a feedback channel for his interlocutor. But there are no sign makers — just moving lips for him to look at. Once again, his Sumerian-speaking sighted helper would be able to say things and observe some sort of reaction to a question — maybe lots of fluttering lashes for “yes”, but one laconic flip for “no”, using a well-known iconic process whereby it is more communicative to use a lot of repeated items in contrast to a very minimal one to get the simple message over. Are you hungry? Do you need X, Y, Z in a long, slow list, to which Bauby would flash a rapid lash for no or flutter it longingly for a yes, for example.

Memories are made of this

People usually tend to work these extreme cases out by mutually targeting the greater interest for both parties. But it looks as if before some sort of shareable signing system existed, Bauby would not have been able to negotiate the creation of a system for expressing his own thoughts by using a shared code similar to what today we call “spelling out” or even signing wildly. In any case, no one presumably used writing for personal narratives at that time anyway.

In an oral culture, a single-channel inarticulate speaker can really only sign assent or dissent (or show surprise and other emotions) to an incoming message — like or don’t like according to the new orthodoxy of social media. The communicators would not even share a separate code for denoting code elements — for example referring to signs, letters, syllables or words, which are distinctly visual concepts meaningful only in a literate culture. This is rather like the binary road-driving code whereby a driver flashes her lights at an oncoming car to signal that the police are down the road ahead of them so slow down. But what if you flash your lights by mistake? There is no back channel, no shared metalanguage that allows a driver to communicate to another car the message of “sorry, didn’t mean to signal a police presence so please ignore that message.”

Yet we know that the Sanskrit Brahmins managed to develop a system for describing their own “perfect” (samskrita) language orally, so as to denote the right way to pronounce a given sequence without having recourse to the visual metalanguage provided by writing. They were able to preserve a pronunciation-only tradition that lasted nearly 3,000 years.

If writing has any global property, then, it is the crucial capacity to provide a physical memory store of pointers (letters, syllabary or logograms) that can itself act as a message generator, independently of a communication channel. All the rest of the computation needed to actually interpret messages is performed, of course, by the human brain. Consider in this context the emergence of chatbots — AI versions of conversations whereby a human asks a device to do something and the device answers back with questions to finetune the requirement. These bots have very limited repertoires of “understanding” but can ask endless questions to reach some worthwhile state for their human user. They could perhaps have helped poor Bauby if they had been invented in time…

Quick update May 2022

Check out this TED demo by Tom Oxley of a Brain Computer Interface that could solve the sort of problem I recounted for Bauby. https://www.ted.com/talks/tom_oxley_a_brain_implant_that_turns_your_thoughts_into_text

[i] Jean-Dominique Bauby, Le scaphandre et le papillon, Paris 1997

[ii] In the 2007 film of The Diving Bell and the Butterfly, finely directed by Julian Schnabel, you see the speech therapist (who would have been the person who had access to knowledge about letter frequency) read out the letters and await Bauby’s blink rather than show him the actual letter board and track the letters with the point of a pencil, for example.

[iii] Bauby notes (p. 27) that some of his interlocutors would not play the game, and would meticulously force him to blink out a complete word such as CHAMPIGNON (mushroom) even though French speakers compute naturally that the next letter after CHAMPI is going to be either CHAMPI(ON) or CHAMPI(GNON) in 99.9% of cases (the place name CHAMPIGNY is also a plausible candidate), and the immediate textual context will usually provide enough information to choose.

[iv] Interestingly, the English term spell comes from a Proto IE root *spel- associated with the idea of speaking/reciting. The writing era meaning of “write or say aloud the letters of a word” dates from the 15th century, based on an earlier notion of “read letter by letter, and even “read with difficulty” (c. 1300).