* During World War II, Allied codebreakers enjoyed an even greater level of success than they had in World War I. The Americans deeply penetrated Japanese codes, while the British performed as or more impressive penetrations of German codes. When war broke out between the United States and Japan in late 1941, the Americans had been decrypting Japanese codes and ciphers since the 1920s, as an outgrowth of US codebreaking activities begun during the First World War. This chapter outlines the evolution of US codebreaking activities up to the beginning of World War II in the Pacific.
* The first formal US codebreaking organization was established by Herbert O. Yardley, who came to Washington DC from the state of Indiana in 1912, when he was 23 years old. He landed a job as a code clerk with the US State Department. He grew interested in encryption and read everything he could find on the subject. In 1916, Yardley received an encrypted message for US President Woodrow Wilson and decided to crack it as an amusement. To his shock, he cracked it in two hours. He continued to study the security, or rather the lack of it, of American codes and ciphers, and in 1917 presented a paper on the subject to his boss, David Salmon. Salmon was startled, but to ensure that Yardley wasn't simply blowing smoke, handed him a set of encrypted messages to crack. Yardley cracked them all in a few weeks.
The United States finally declared war on Germany on 6 April 1917. Yardley decided that the War Department would be a better path for advancement than the State Department, and enlisted in the Army in late June 1917. He was immediately put in charge of Military Intelligence Section 8 (MI-8), which was responsible for codes and ciphers. Yardley promptly built up an empire, establishing sections to improve the pathetic security of American codes and ciphers, and to crack the codes and ciphers of other countries.
When the Armistice was signed on 11 November 1918, ending the fighting in Europe, Yardley worried that he might be out of a job. However, the value of codebreaking organizations had been recognized by the US military and government, and less than a year later, in May 1919, Yardley was put in charge of an "American Black Chamber", as Yardley called it, jointly funded by the State Department and the War Department.
* The Black Chamber was set up in New York City, both to conceal its operations and because some odd law prevented the State Department from spending parts of its budget in Washington DC. Yardley set up a commercial business, the "Code Compilation Company", in the bottom floor of the brownstone building where the Black Chamber had its offices. The Code Compilation Company was partly a front, partly a way to obtain a little funding by selling cryptographic services to businessmen.
In 1919, Yardley managed to crack the Japanese diplomatic code. This information proved extremely valuable in assisting the Americans in negotiations during the 1921 Washington Conference on naval arms limitation. Unfortunately, after the conference, the need for the Black Chamber began to fade. The problem with justifying the organization's existence was complicated by the fact that the US government had passed the Radio Communications Act in 1912, which essentially outlawed "eavesdropping" on radio or cable traffic. The Radio Communications Act had been in abeyance during the war because of mandatory censorship measures, but now that the war was over, the act was back in force. Basically, the Black Chamber was an entirely illegal operation, although clandestine agreements were carefully negotiated with Western Union and other telegraph companies for access to diplomatic cable traffic.
In 1924, the budget of the Black Chamber was cut in half. Even more severe legal restrictions were imposed by the Radio Act of 1927, reducing the flow of intercepts once more. Finally, in 1929, the new Secretary of State, Henry L. Stimson, pulled all State Department funding for the Black Chamber. Stimson commented that: "Gentlemen do not read other gentlemen's mail." This is often cited as evidence of his naivete, but Stimson was a hawk who favored American military assertiveness. He simply didn't think it was the State Department's particular business to play at codebreaking.
The Black Chamber was formally shut down on 31 October 1929. This was a week after the Black Thursday stock-market crash that initiated the Great Depression in America. Yardley was out of a job, and the market for unemployed codebreakers was not good.
* In need of money, Yardley considered writing a book on the Cipher Bureau. He knew this was a breach of security, and certainly realized that if the Japanese found out that their diplomatic codes had been compromised, they would adopt new codes that would be much tougher to crack. Despite this concern, Yardley also felt he had an obligation to act as what a later generation would call a "whistle-blower", alerting the public to the feeble and dangerous state of American codes and codebreaking. He approached publishers on the prospect of selling such a book, and found a buyer.
Unfortunately, one of the other publishers Yardley had approached went straight to the government and told them what Yardley was up to. Yardley was given warnings; he ignored them. In the spring of 1931, three articles written by Yardley on the Cipher Bureau were serialized in THE SATURDAY EVENING POST magazine, leading up to the publication of Yardley's book, THE AMERICAN BLACK CHAMBER, on 1 June 1931. The book was a best-seller, and it was translated into several languages, including Japanese. Of course, the Japanese quickly decided to adopt improved encryption schemes. The US government denied everything in Yardley's book. Government officials considered both arresting Yardley and suppressing his book, but decided the damage had been done and that taking action against him would only make things worse.
Yardley then decided to follow up his success with THE AMERICAN BLACK CHAMBER with a new, more scholarly book titled JAPANESE DIPLOMATIC SECRETS: 1921-1922. The government wasn't being taken by surprise this time; in 1933, the authorities told Yardley that if he persisted in trying to publish the new book, he would be arrested and charged with espionage. JAPANESE DIPLOMATIC SECRETS: 1921-1922 was classified and not released to the public until 1979. The whole fiasco led to the passage in 1933 of a US national-security law that still remains in force as "Section 952 of Title 18 of the United States Code", which threatens anyone who has been a US employee and publicly releases official diplomatic documents with up to ten years in prison. Yardley had been put in his place.
BACK_TO_TOP* Despite Yardley's concerns, the death of American codebreaking had been "greatly exaggerated". By that time, the US Army had the nucleus of a codebreaking organization, centered around a civilian cryptologist, William Frederick Friedman.
Friedman's parents were Hungarian Jews who had been living in Russia when he was born in 1891. Antisemitism drove the family out of Russia to the United States, and the family did well in the new country, sending their son to Cornell University. He graduated in 1914 with a degree in genetics, and got a job with Colonel George Fabyan, a wealthy and unconventional Illinois textile merchant who had a large estate outside of Chicago. His title of "Colonel" was a state militia rank, awarded to him by the governor of Illinois.
Fabyan hired Friedman to perform research into plant and animal breeding. Fabyan had other projects, and one involved a young woman named Elizebeth Smith. Smith was a librarian, and was hired to assist two sisters named Gallup, who were "Baconians", meaning they believed that Sir Francis Bacon had actually written the works of Shakespeare, leaving clues in the form of a subtle code in the plays. Friedman was an amateur photographer, and Fabyan enlisted him to take pictures of Shakespearian texts. Friedman became curious and began to study codes and ciphers on his own, becoming very proficient in the art. Elizebeth Smith also got interested in codes and ciphers, and became an expert as well. Fabyan finally established a cryptanalytic team, putting Friedman in charge of it.
The Colonel had a few government contacts and let them know he had a codebreaking organization that might be useful to the government; his contacts took him up on his offer. In 1916, Fabyan began to receive encrypted messages from the government. Friedman's little codebreaking group cracked them and sent them back. When American involvement in the European war began to appear inevitable in early 1917, Fabyan decided to offer the services of his little cipher group to the US War Department. The War Department send Lieutenant Joseph Mauborgne, one of the few people there who knew anything about the subject, to investigate, and Mauborgne was thoroughly impressed.
This was a busy time for William Friedman. He married Elizebeth Smith in May 1917, and in the fall of that year, taught a six-week cram course in cryptanalysis to a handful of military officers. The course went over so well that he taught the same course to a much larger group of officers in early 1918, with a follow-up session later in the spring. By June 1918, Friedman was in the Army, which sent him to France to break German codes. He returned home and was mustered out in April 1919. He tried to find work as a geneticist for a while, but failing in that, ended up working for Colonel Fabyan again.
* Friedman was writing down his ideas about cryptanalysis, publishing seven pamphlets, collectively known as the "Riverbank Publications" after Colonel Fabyan's Riverbank estate. All were original and insightful, but the seventh pamphlet, titled "The Index Of Coincidence And Its Applications In Cryptography", was the most significant. In this pamphlet, Friedman considered the problem of trying to decrypt a Vigenere cipher with a long random key. His analyses showed that in any long string of English text, any two consecutive characters selected from the plaintext would be the same character 7% of the time on the average, due to the common nature of digraphs such as "ee" or "oo" or "tt". In contrast, in purely random text, the same character would only occur twice 4% of the time on the average.
Suppose Holmes has two ciphertexts encrypted by a Vigenere cipher. If the cipher system is implemented properly, the percentage of digraphs consisting of the same two characters in either ciphertext is 4%.
Now further suppose Holmes compares two ciphertexts that he believes may have common key sequences at some location or another, but not necessarily the same location in each. He arranges the two ciphertexts side by side and then obtains statistics on how often the letters in both ciphertexts are the same. If the two ciphertexts don't have a matching key, the coincidences will be random, and matches will occur only 4% of the time.
Holmes then shifts the relative positions of the two ciphertexts and performs the analysis again. If the result of the analysis shows that the matches are again about 4%, then he knows the two ciphertexts still don't have the same key.
Holmes keeps on doing this, until he finally shifts the two ciphertexts into a relative position where they have a key that matches. The percentage of duplications will then jump to 7%. Once this happens, Holmes can use one ciphertext as a crosscheck against cryptanalysis of the other ciphertext by conventional means, in much the same way that a one-time pad cipher can be broken if two messages are encrypted with the same cipher key. This is certainly another "word puzzle from Hell" exercise, but it does give him a lever to crack open the cipher. This analysis and related ones Friedman performed in the same pamphlet took the use of statistical tools for cryptanalysis to an entirely new level, and would eventually establish Friedman as one of the most significant figures in the history of cryptology.
* Friedman finally tired of working for Fabyan, who wasn't the most scrupulous boss, and decided to find work elsewhere. In November 1920, the Friedmans accepted positions with the US government as civilian cryptographers and moved to Washington DC. William took a job with the Army Signal Corps, where he became the chief cryptologist, while Elizebeth worked briefly for the Navy, and then settled down as a housewife to raise a family. She would perform part-time work for the Treasury Department, and during Prohibition cracked ciphers used by rum-runners.
While the Friedman family lived a comfortable middle-class life, William continued to advance the state of the art in devising and breaking ciphers. In 1923, as a consequence of teaching courses on cryptology to US Army personnel, he published a short but classic text on the subject titled ELEMENTS OF CRYPTANALYSIS. The little book was clear, concise, and proved influential. In particular, Friedman used it to lay down precise definitions of cryptologic terms. Up to this time, the terminology used in the field was inconsistent and confusing, but Friedman's definitions cut through the confusion so well that they became the accepted terms. Even the terms of "cryptanalysis" and "cryptology" as used in this document owe their origins to Friedman's book.
* Through the 1920s, Friedman lived a modest existence working for the Army. At the time, he was the only cryptanalytic resource the War Department owned, and the only help he had was a retired prize-fighter who was his clerk and typist. Months before Secretary Stimson closed the Cipher Bureau, in the spring of 1929 the US Army had decided to switch responsibility for cryptanalysis from the Military Intelligence branch to the Signal Corps, leading to the establishment of the "Signal Intelligence Service (SIS)" on 24 April 1930.
With the establishment of the SIS, Friedman acquired greater responsibilities and authority, but not by leaps and bounds. He was authorized to hire am additional staff of four junior cryptanalysts and an assistant cryptanalytic clerk. The pay for a junior cryptanalyst position was only $2,000 USD a year, which was modest even for the time, but the Depression was on and jobs were scarce. Friedman managed to hire three highly skilled mathematicians who also had useful language skills: Frank B. Rowlett, who spoke German; Abraham Sinkov, who spoke French; and Solomon Kullbach, who spoke Spanish.
Friedman expected to have a difficult time finding someone with fluent Japanese language skills to fill the fourth slot. An acquaintance in the government called to request a job interview for his nephew, John B. Hurt. Friedman accepted out of courtesy and arranged the interview, with little expectation that Hurt would be worth the bother, but the Army officer who conducted the interview was astonished at Hurt's fluency in Japanese. With the hiring of an assistant clerk, the SIS was fully staffed with seven people. Until 1937, the yearly budget of the organization never exceeded $17,400 USD.
In August 1935, Major Haskell Allison replaced Friedman as administrative head of the SIS, freeing the cryptologist to focus on technical issues. Friedman gave his people the full benefit of his two decades' experience as a master of codes and ciphers. As international tensions continued to rise through the 1930s, the SIS grew accordingly, but Friedman and his four disciples remained the prime movers.
At the outset, cracking cryptosystems was a secondary goal for the SIS; the primary job of the organization was initially to develop secure cryptosystems and procedures for the parent service. In fact, the SIS wasn't officially connected to the Army intelligence apparatus; it was part of the Army Signal Corps. However, the Army did set up several intercept stations during the following decade -- mostly stateside, though one was set up in Manila in the Philippines.
BACK_TO_TOP* US Navy efforts in cryptology following World War I paralleled those of the US Army. In 1924, the Naval Code and Signal Section of the Office of Naval Intelligence (ONI) had formally assigned one officer, Lieutenant Laurance F. Safford, and one civilian, Agnes Meyer Driscoll -- who had been a Navy "yeomanette" during the war -- to work on crypto issues. Their little office was designated "OP-20-G". Its main goal was to ensure the security of Navy cryptosystems, with codebreaking secondary to that goal; the operation also conducted training.
OP-20-G's work with operational Navy organizations helped build up an informal network of Navy men interested in cryptology and establish field contacts for OP-20-G. From 1926, OP-20-G conducted instruction courses for Navy and Marine officers, which helped further expand the influence of the organization. The Navy also set up a radio intercept post in Shanghai, China, and in 1928 set up a school for enlisted Navy and Marine Corps radio intercept operators at the Navy Department in Washington, DC. A classroom with training sets was set up on top of the main Navy building, probably as much for the sake of privacy as for the lack of space. Understandably, student graduates became known as the "On the Roof Gang."
* Along with the training activities, OP-20-G also had obtained a major success in breaking the code used by the Imperial Japanese Navy (IJN), though the method used to do so was no feat of intellectual brilliance: ONI operatives quietly "borrowed" a code book from the Japanese consulate in New York City (NYC) and photographed it. The photos were placed in a red binder, and so that particular IJN code became known as the "RED BOOK" code. The binder was transcribed and translated, with a handful of copies made and carefully distributed. Two more "borrowings" from the Japanese consulate were performed into the 1930s.
Of course, the IJN sensibly changed their cryptosystems every now and then. By late 1930, OP-20-G had broken the successor code, this time the hard way, by cryptanalysis. The new code was stashed in a blue binder and was appropriately known as the "BLUE BOOK" code. It was of course completely different from the RED BOOK code, but the knowledge of the RED BOOK code helped crack BLUE BOOK, since BLUE BOOK could be (correctly) assumed to have a similar vocabulary.
OP-20-G used Hollerith punch-card "tabulating" (sorting) machines built by the Tabulating Machine Company -- later known as IBM -- to help crack BLUE BOOK. The punch card machines could perform simple sorts and other operations on data typed into punch cards, assisting in statistical analyses of encrypted traffic. The notion of using card sorters in cryptanalysis had been suggested by Captain Stanley Hooper, the Director of Naval Communications, in 1930.
* OP-20-G established contacts with the Army SIS from early on. It seems there was a degree of professional respect between the two offices, in particular that Safford and Friedman were on collegial terms, trading insights on cryptanalysis, but going much farther than that proved difficult. Cooperation in cryptological matters was obtained in establishing common cryptosystems and communications / security procedures so the Army and Navy could perform secure communications with each other. Since the Navy had a leg up on such matters, the Army was inclined to follow the Navy's lead.
However, each service targeted the corresponding service of adversaries -- that is, the US Navy wanted to know about the IJN while the US Army wanted to know about the Imperial Japanese Army (IJA). That really wasn't much of an issue; the problem was that both services regarded diplomatic communications as a secondary target, and there was an inclination in the overloaded crypto office of one service to want to offload the task on its counterpart in the other service. Neither had any problem working on commercial communications, simply because they weren't regarded as all that significant or troublesome.
Although meetings were set up and committees were assembled to discuss the matter, cooperation proved elusive, though personal communications between the two groups remained good. Some of the brass would try to shut off the channel, but it would always open back up when the management went on to a new posting.
BACK_TO_TOP* By the early 1930s, both the US Navy and the US Army had expanded their network of radio intercept stations, with posts set up on Guam and in Hawaii. It was all done on a shoestring, and security was a problem: it made no sense to secretly listen in on adversary communications from remote islands, and then radio home reports on intercepts that could be picked up by that adversary in turn. The result was that communications were mostly conducted by mail. Special procedures were set up, in particular via a strongbox installed on commercial "Clipper" flying boats flying Pacific routes, with the strongbox reserved for top-secret documents. It was reasonably secure but definitely slow.
By the mid-1930s, Japanese aggression in China was creating antagonism with the United States, and a conflict between Japan and America seemed increasingly possible. In the wake of Yardley's THE AMERICAN BLACK CHAMBER, the Japanese had become interested in cipher machines. In 1932, they had adopted a mechanical cipher machine for diplomatic communications. The system was codenamed "RED" by the Americans, though it had little or nothing to do with the RED BOOK codes. The SIS made cracking RED a top priority in 1935 and cracked it in 1936.
RED was a relatively primitive machine, and SIS cryptanalysts managed to crack it using the statistical analysis techniques pioneered by William Friedman. Frank Rowlett devised a simple code wheel device to allow RED messages to be quickly deciphered.
In 1937, under the direction of Joseph Mauborgne, now a major general and as of October of that year the Army's Chief Signal Officer, the SIS began to grow. The organization was soon faced with a new challenge: the Japanese gradually recognized that RED was vulnerable, and so in the spring of 1939 they moved on to a much more sophisticated cipher machine for diplomatic communications, which was codenamed PURPLE. It was an electromechanical design, based on moving-arm relays that were made for use in automated telephone exchanges, ganged in series. The machine was expensive and only the most important Japanese embassies obtained it: Berlin, Rome, London, Washington DC. The other embassies retained use of less sophisticated cryptosystems.
* The nature of the Japanese written language adds some complications to the design of a code machine that wouldn't apply to Western languages. At the core of the Japanese written language is the "kanji", which is basically the Chinese ideographic symbol set. About 2,000 kanji are needed for minimal Japanese literacy, with the full set amounting to more than 20,000 symbols, though few Japanese know them all.
The kanji are supported by a much smaller phonetic set known as "kana". The kana describe the sounds used in Japanese pronunciation, and they can be written in two styles known as "hiragana" and "katakana". Some Japanese words are written in hiragana, but hiragana characters are more significantly used as modifiers to adapt the Chinese characters to Japanese syntax, the two languages actually having little in common besides the character set. Katakana characters are used to write down foreign words for which no kanji exists. Both are used in a Japanese dictionary, in different contexts, to give the pronunciation of kanji symbols. The two sets of kana characters are equivalent; they just look different. There are 50 basic kana characters, with about 20 auxiliary modifications.
The Japanese also adopted the Western Roman character set, logically calling it "Romanji", using it for stylish titles and particularly for acronyms, which don't really make sense in the traditional Japanese symbol set. Since kanji was too cumbersome for a cipher machine, that left only the alternatives of sending kana symbols or spelling out the same set of sounds using Romanji letters.
The Japanese diplomatic machines sent messages in Romanji. Japanese is very context-sensitive, with many words that sound the same, but transmitting the language in Romanji using phonetic spellings is conceptually little different from speaking it over the telephone: as long as some care is taken to avoid ambiguities, there should be little confusion. Although writing Japanese phonetically in Romanji was the least obvious approach -- it's not something Japanese normally do -- using Romanji had the advantage that Western documents could be sent verbatim, a useful capability for an embassy.
PURPLE's encryption scheme was along the lines of that of the Enigma machine, but instead of rotors, it used banks of telephone "stepping switches" -- a type of electromechanical switch used to establish telephone connections in a telephone exchange. The basic element was a switch with a single input lead and a moveable "wiper" arm connecting it to 25 possible output leads. The wiper arm was moved by an electromagnetic solenoid; the solenoid would move the arm from position to position until it reached the last position, and then a spring would pop the arm back to the first position.
The arrangement of the PURPLE machine was confusing. Six letters, nominally the English vowels, were encoded by a bank of six switches; the other 20 letters, were encoded with three banks of 20 switches, the banks being connected in series, with each bank performing one level of scrambling. All the switches in a bank were shifted together -- but, like the rotors in an Enigma machine, each of the three banks shifted at its own rate.
To explain how the scrambling worked, suppose we consider a simplified or "toy" scheme with a bank of only four switch elements labeled "A" through "D", with each switch having only four positions, "1" through "4", and connected to a second bank of four switch elements labeled "W" through "Z". We can then represent wiring, say, leads A1, B1, C1, and D1 together and connecting them to the input of W as follows:
W = A1 + B1 + C1 + D1
The simplest possible wiring arrangement is as follows:
W = A1 + B1 + C1 + D1 X = A2 + B2 + C2 + D2 Y = A3 + B3 + C3 + D3 Z = A4 + B4 + C4 + D4
This is also the most absolutely useless arrangement: since the wipers of all four switch elements move in step to the same positions, they would all short each other out, all the time. The rule is that a specific lead on one switch element cannot be connected to the same lead on another switch element -- for example A1 can't be wired to C1, B3 can't be wired to D3, and so on. A more workable wiring pattern would be of the form:
W = A1 + B4 + C2 + D3 X = A2 + B3 + C4 + D1 Y = A3 + B2 + C1 + D4 Z = A4 + B1 + C3 + D2
-- or, resorted by wiper position instead of switch element:
W = A1 + C2 + D3 + B4 X = D1 + A2 + B3 + C4 Y = C1 + B2 + A3 + D4 Z = B1 + D2 + C3 + A4
This resorting reveals the connection routing for each wiper position. In the first position, WXYZ = ADCB; in the second position, WXYZ = CABD; in the third, WXYZ = DBAC; and in the fourth, WXYZ = BCDA. Another workable wiring arrangement would be as follows:
W = A4 + B3 + C1 + D2 = C1 + D2 + B3 + A4 X = A2 + B1 + C3 + D4 = B1 + A2 + C3 + D4 Y = A1 + B4 + C2 + D3 = A1 + C2 + D3 + B4 Z = A3 + B2 + C4 + D1 = D1 + B2 + A3 + C4
Of course, the switch banks of a PURPLE machine couldn't be yanked out like rotors and rearranged, but it is possible to produce an equivalent effect by changing which banks step at which rate using a master selector switch. With three banks, there are six possibilities; the table below gives the arrangements of banks "B1" through "B3" for selector switch positions "SW1" through "SW6", with the banks selected to step by 1, 25, or 625:
B1 B2 B3
---------------------
SW1 1 25 625
SW2 1 625 25
SW3 25 1 625
SW4 25 625 1
SW5 625 1 25
SW6 625 25 1
---------------------
* Although PURPLE was much more difficult to crack than RED, the SIS group had some advantages. As mentioned, PURPLE had some similarities to RED, and during the period when Japanese embassies were upgrading to the new code machine, the same messages were foolishly sent in both RED and PURPLE. Decryption was further helped by the fact that the format of the contents of the messages was very uniform. Certainly this is how the Japanese do things, but it is possible that the uniform format was partially imposed by the necessity to reduce the ambiguity of messages sent phonetically rather than in kanji.
The SIS used punch-card sorting machines to perform statistical analyses of Japanese traffic. While working on the RED code, Friedman had obtained an IBM punch-card machine that the US Army Quartermaster Corps had used for accounting purposes and was phasing out. The punch-card machine proved so useful that Friedman had bought another one.
In the spring and summer of 1940, with war in full gear in Europe, PURPLE traffic increased dramatically, particularly for communications between Tokyo and the Japanese embassies in Berlin and Rome. These communications led to the "Tripartite Pact", which established the "Berlin-Rome-Tokyo Axis". The large flow of encrypted data gave SIS an opportunity to obtain a better grip on PURPLE. On 27 September 1940, the Tripartite Pact was signed. Two days previously, the SIS had been able to adequately read a non-trivial message encrypted in PURPLE for the first time.
Being able to crack PURPLE messages was only half the job. US intelligence needed to know what the messages contained as soon as possible, and that meant duplicating the Japanese cipher machine. Remarkably, the SIS managed to reverse-engineer the PURPLE output and build an equivalent cipher machine. With the "PURPLE analog" machine available, the SIS was able to decode Japanese diplomatic messages as quickly as Japanese code clerks themselves.
Since the SIS didn't have the funds to build the machines themselves, Friedman lobbied the US Navy for help, and five of the PURPLE analog machines were produced by the Navy Yard machine shop. The SIS was beginning to produce proper decrypts of PURPLE traffic by late November 1940. The PURPLE analog machines were expensive, and they could only be turned out slowly, with allocation of the precious machines proving troublesome.
No actual Japanese PURPLE code machines survived the war; the Japanese were strict about procedures, and procedure was to completely destroy the code machines when pulling out of an embassy. However, fragments of a code machine found in the Japanese embassy in Berlin after the war suggest that the PURPLE analog machine was remarkably accurate.
The difficulty of managing a greatly expanded organization and the prospect of war put Friedman under intense pressure. In January 1941, he was admitted to the Neuropsychiatric Ward of the Walter Reed Army Hospital in Methesda, Maryland. He was back on the job by 1 April, but never really carried a full workload again. His insights were unsurprisingly still regarded as extremely useful.
* As a footnote to the RED and PURPLE story, the Imperial Japanese Navy (IJN) also used cipher machines based on commercial telephone-switch banks. The first was the "CORAL" machine, which was used by naval diplomatic attaches. It enciphered Japanese written phonetically in Romanji, fed through three levels of switching elements, without any division of the letter set as was done in PURPLE and RED. CORAL apparently had a plugboard, but the selector switch could only allow the operator to choose one of three bank-stepping arrangements, not six as with PURPLE.
The "JADE" machine was actually used tactically by the IJN from late 1942 into 1944. JADE was very similar to CORAL, the main difference being that JADE actually enciphered kana directly, using a shift key to allow entry of the complete set. Since it was not used by diplomatic officials, there was no reason to worry about sending Western documents with it. CORAL and JADE were both apparently cracked, but the details of these exercises are hard to find. It appears that these ciphers were never of more than secondary importance. JADE machines were captured intact during the war.
BACK_TO_TOP* By the late 1930s, the US Navy codebreaking organization was structured around three main bases: headquarters in Washington DC, a section at Pearl Harbor in Hawaii, and a section in the Philippines. The codebreakers were backed up by a far-flung network of listening and radio direction finding stations.
The primary target of Navy codebreakers was the Imperial Japanese Navy. The IJN was still focused on manual cryptosystems, introducing what was codenamed "BLACK" in 1938, being followed by two new systems in 1939: the "Flag Officer's" code, which as the name implies was for communications at top command levels, and a five-digit superenciphered code given the designation of "Japanese Navy 25 (JN-25)".
The Navy would never crack the Flag Officer's code, mostly because there wasn't much traffic in it. The IJN also had several other minor cryptosystems -- some of which were cracked, some of which had too little traffic to crack, and some of which were regarded as not worth cracking. It was JN-25 that became the primary target.
JN-25 was a book system, not a machine cipher, with operators provided with an instruction manual and two books implementing the cryptosystem itself: a book of codes, and a book of superencipherment tables known as an "additive book". Originally, the JN-25 code book consisted of about 30,000 five-digit codegroups that represented Japanese phonetic kana characters, numbers, place names, and a long list of other items. For error checking, when the individual digits of each five-digit number were added up, the result was always divisible by three.
The additive book was filled with 30,000 five-digit numbers selected at random. It consisted of 300 pages, with each page providing a hundred random numbers in ten rows of ten numbers. It was referred to as an "additive book" because these numbers were added to codegroups without using a numeric carry from digit to digit. For example:
: random number: 28506 : codegroup: 58905 : ----- : add without carries: 76401 -- NOT 87411!
Stripping out the superencipherment was then just a simple no-borrow subtraction:
: superenciphered code: 76401 : random number: 28506 : ----- : subtract without borrows: 58905 -- codegroup
The appropriate set of random numbers was selected by a key consisting of the page number in the additive book and a row of numbers on the page, meaning that there were 3,000 possibilities of where to start. Presumably, if a message went beyond the end of the book, the operator just "wrapped around" and started again from the beginning of the book. The key was included at the beginning and end of each message.
Cracking JN-25 meant sorting through the superencipherment before the code itself could be dealt with. Punch-card sorters were used to sift through the superencipherment, with the codebreakers then gaining ground on the codegroups -- but any success they had was only temporary. The IJN sensibly updated JN-25 periodically, often changing the additive book, occasionally changing the codegroups. Every time they did, the Navy codebreakers would have to reset and have to start again.
In the meantime, intercepted messages were piling up, dumping an intolerable load on American codebreakers. By the fall of 1941, the number of IJN intercepts had reached about 10,000 a month, with roughly two-thirds of them encrypted in JN-25. The US military simply didn't have the staffing to keep up with it all; it wasn't just a question of having cryptologists, but also of having translators who could read Japanese. Although US military funding was on a rapid increase, codebreaking wasn't a problem that could be solved by simply throwing money at it. Coordination between the different Navy codebreaking stations was also not the best, partly due to the fact that communications were conducted by secure mail.
* There is a saying that a person with a hammer tends to see things as nails. Since the US codebreaking organization had cracked PURPLE but not JN-25, the focus was on reading Japanese diplomatic traffic -- even though both the Army and Navy codebreaking organizations regarded it as second priority. The concentration on diplomatic traffic inevitably diverted resources from work on JN-25 and other Japanese military cryptosystems.
There was cooperation between the Army and Navy on handling the diplomatic traffic. The Army had, as mentioned, handed over precious PURPLE analog machines to the Navy, and with good reason: the Army didn't have the resources to keep up with the traffic. The Navy had good reason to play along as well. The decrypted diplomatic traffic was routed to President Franklin Delano Roosevelt and a very restricted list of other high officials under the codename MAGIC, which effectively translated to "absolute top level security clearance". MAGIC decrypts had visibility at the very top, and the Navy was not going to let the Army keep the spotlight to itself.
There was the question of how to coordinate MAGIC activities between SIS and OP-20-G. In January 1941, a compromise of sorts was reached, with the Navy and the Army responsible for MAGIC on alternating months. In October 1941, the scheme was changed to alternating on a daily basis, with the Navy handling odd days of the month and the Army on even days. This was the sort of measure that in hindsight seems feeble, but it appears the rationale was that, lacking the ability to work in an interlocked fashion, the two groups at least understood that they didn't want to end up handing Roosevelt or other high US officials two conflicting decrypts of the same message, which would have led to considerable trouble for all concerned. The compromise otherwise did little to relieve the pressure on American cryptologists.
The British were working on JN-25 as well, with the "Far East Combined Bureau (FECB)" in Singapore freely trading information on the problem with US Navy cryptanalysts at the Philippines station -- and even the collaborative effort of both nations wasn't making enough progress. As discussed later, the British did have a major cryptologic organization at Bletchley Park north of London, but it was focused on dealing with German and Italian communications and had few or no resources allocated for cracking JN-25. In late 1941, JN-25 was a closed book to both US and British codebreakers.
BACK_TO_TOP* Although the US Navy couldn't read JN-25, it was possible to obtain some picture of IJN movements and intent through "traffic analysis", or in other words simply logging the location, type, and callsigns of IJN radio traffic even if it couldn't be read. From the summer of 1941, traffic analysis seemed to increasing confirm that the IJN was gearing up for what the Japanese referred to as the "Southern Operation", targeting Burma, Singapore, and the Dutch East Indies, with the traffic analysis suggesting a major buildup of IJN forces in the Formosa region. The Philippines station, now in the caves at the island of Corregidor and being built up, confirmed Hawaii's impressions.
The one major problem was that the traffic analysis did not reveal the location of several IJN aircraft carriers. The general assumption was that if a vessel couldn't be located at sea, it might just be at dock in its home port. On 1 November 1941, the IJN updated their callsigns, throwing the Navy radio intercept network for a loop for the time being. By late November, however, Navy operators had been able to use traffic analysis to observe a shift of some IJN fleet elements north for purposes unknown, but US attention remained focused on the large naval buildup to the south.
* In the meantime, US codebreakers were continuing to exploit Japanese diplomatic communications as yielded by the PURPLE analogs. The diplomatic traffic was of course valuable, but it provided only the most general sort of insight into Japanese military activities: armed services are not usually in the habit of informing the foreign ministry of details of military plans, and foreign ministries aren't necessarily inclined to inquire about them. By late November, war seemed imminent, though the blow to American forces was expected to fall on the Philippines. A warning was sent out to US Army and Navy commanders in the Pacific on 27 November.
In the dark hours of the morning of 7 December 1941, the US Navy communications intercept station on Bainbridge Island, in the state of Washington, picked up a radio message being sent by the Japanese government to the Japanese embassy in Washington DC. It was the last in a series of 14 messages that had been sent over the previous 18 hours. The messages were decrypted by a PURPLE analog machine at OP-20-G and passed to the SIS for translation from Japanese, early on the morning of December 7. Army Colonel Rufus Bratton and Navy Lieutenant Commander Alvin Kramer independently inspected the decrypts.
They both became alarmed. The decrypts instructed the Japanese ambassador to Washington to inform the US Secretary of State, Cordell Hull, at 1:00 PM Washington time that negotiations between the US and Japan were ended. The embassy was then to destroy their cipher machines. That sounded like war, and although the message said nothing about any specific military action, Kramer also realized that the sun would be rising over the expanses of the central and western Pacific by that time. The two men both tried to get in touch with Army Chief of Staff General George C. Marshall.
After some agonizing delays, Marshall got the decrypts and methodically examined them. He realized their importance as well and sent a warning to field commanders, including Major General Walter Short, the Army commander in Hawaii. However, Marshall was reluctant to use the telephone because he knew that telephone scramblers weren't very secure, and sent it by less direct channels. Due to various constraints and bumblings, Short got the message hours after the Japanese bombs had smashed the US Navy's fleet at anchor in Pearl Harbor. The IJN strike fleet had steamed from the Kuriles north of Japan to a location north of Hawaii to launch the strike, maintaining strict radio silence; having smashed the US Navy at anchor, the fleet then left for home.
In the wake of the disaster, the US military cryptological apparatus would undergo reorganization and grow rapidly. The Army organization, which had only had seven staffers by 1936 and 19 by 1939, had about 180 people in Washington DC and 150 in the field; contrast this with over 10,000 by the end of the conflict. The Navy organization underwent a similar growth. Still, US military codebreaking during the war would never be quite the clockwork operation that those involved would have liked it to be.
BACK_TO_TOP