Consulting Agreement This Consulting Agreement (This Agreement), Effective The Day O: Fill & Download for Free

Welcome to Cryptanalysis 101!!All of the answers here do not really show how Enigma ciphers were broken. I shall rectify this situation by delivering an answer that briefly recounts the history about the Enigma and meticulously describes how it was broken.The short answer was: it was primarily a result of human ingenuity demanded by wartime urgency.To understand how the Enigma ciphers were broken, I think it is necessary to briefly explain how the machine worked. The three most important components of an Enigma machine are:the lampboardthe scrambler unit which was used to set the day key used for enciphering/decipheringthe plugboardThe standard Enigma machine used by the German Army consisted of 3 rotating wheels with internal wirings that altogether constituted the scrambling unit which transformed a plaintext letter into a cipher text through a circuit determined by the internal wiring of the three connected rotating wheels. The plugboard between the keyboard and the first rotating wheel transposed pairs of letters before entry into the wheel.By combinatorial calculation, the respectively contribution to the number of ways of turning a plaintext message into gobbledygook by:The scrambling unit: 1,054,560 waysThe plugboard: 150,738,274,937,250 (using 10 wires which was the most widely used number of wires)For a total of [math]1,054,560 \times 150,738,274,937,250 = 158,962,555,217,826,360,000[/math].(If you are interested in a meticulous description of the internal operation of an Enigma machine and how it was used to encipher/decipher messages, consider reading this answer: Werner Hermann's answer to How exactly did the Enigma machine work? How did the plugboard and the rotors change the letters?)As seen, the greatest contribution to the number of ways of scrambling a message came from the plugboard. That huge number was the primary reason why the Enigma machine was extraordinarily difficult to break during the 1st few years of WW2.So how were Enigma-generated ciphers broken?Without going into much detail, it was the achievement of the man below:Marian RejewskiMarian Rejewski was a brilliant Polish mathematician who worked for the Polish Biuro Szyfrow (Cipher Bureau). Being surrounded by two of the most powerful military powers in the world: Germany in the West and the Soviet Union in the East, the Poles faced constant threat of being attacked from both sides. The danger gave rise to the need for strong military intelligence capability to give Poland advance knowledge of what the country’s potential enemies intended to do.Since the Great War, radio was used extensively for communicating military messages. But radio messages could be read by anyone with interception capability, including the enemy. This necessitated the development and employment of secure encryption to turn plaintext messages into ciphers to prevent sensitive military information from being read by the enemy.Starting in the 1920s, the German military adopted the Enigma enciphering machine which proved highly effective, rendering all of their Enigma-enciphered radio messages secure from Britain’s and France’s attempts to break them. However, because Germany back then did not yet pose a threat (as a result of Treaty of Versailles which restricted the size of the German armed forces), both France and Britain were complacent and made no attempt to break Enigma-generated ciphers.But there was one nation that could not afford to be complacent: Poland. Having faced military threats from Germany in the past, the Poles were determined to find a way to break Enigma ciphers to avoid being caught by surprise as a result of not knowing German military intention.Like the French and British, the Poles were initially puzzled by Enigma-generated ciphers due to the lack of knowledge of how the machine worked . Fortunately for the Poles, they finally got the means to break the Enigma thanks to a German traitor by the name of Hans-Thilo Schmidt. What Hans-Thilo Schmidt did was to provide for the British and French stolen documents which contained not a description of the internal wirings of the military Enigma machine but the information from which those wirings could be deduced to build a replica of the military version of the Enigma. In exchange for this treacherous act, Hans would receive huge sums of money from whoever that decided to buy his documents.As noted before, since Germany had not yet emerged as a military threat, the French and British were apathetic about Hans’s offer. As luck would have it, France and Poland had established an alliance for military cooperation 10 years earlier. As part of the agreement, both nations were obliged to share information with each other. Hence, the French decided to inform the Poles of the offer of Hans Thilo Schmidt. Desperate to uncover German military information, the Poles eagerly accepted the offer.The stolen documents revived the Poles’ hope in cracking the Enigma. However, there was one major obstacle to be overcome. Knowing how the Enigma worked was not enough. Knowing the keys was far more important. Indeed, the key for a message is the most important part of any encryption process. The strength of an encryption system consists not in keeping secret its inner working but in keeping secret the key. The Germans knew that if the keys were protected, their enemies would have no way of deciphering Enigma ciphers even if they had the machines.As described in my answer how the Enigma machine works, the HUGE number of ways of enciphering a message was due to the HUGE number of daily keys produced by a combination of the scrambler initial setting, the selection and arrangement of the scramblers, the ring setting, and the plugboard setting. The challenge for Polish codebreakers was to determine the key associated with a particular message generated on a particular day. With the huge number of possible keys, the Polish codebreakers knew full well that a brute force approach was not going to work. They were convinced there was a shortcut to find the keys.With ingenuity and determination, such a shortcut was eventually discovered by Marian Rejewski. To understand Marian Rejewski’s ingenious method for finding the day keys, it is essential to know how the Germans used the Enigma to encipher messages.Periodically (about every 4 weeks), a code book containing instructions for setting daily keys was issued for German military units everywhere. In theory, all messages in any particular day were enciphered using a single key specified. However, it is a well known cardinal rule in cryptography that if a single key was used repeatedly to encipher a large number of messages, then it became easy for codebreakers to deduce the key. A large amount of messages that were enciphered identically (i.e using the same key) gave rise to a correspondingly large likelihood of identifying the key, as was the case for mono-alphabetic substitution cipher.Thus, to reduce that likelihood, in practice, German Enigma operators came up with one simple and clever idea: used different keys for different messages and used the daily key to encipher each of those message keys. It bears repeating that the daily key was a combination of:1/ the selection and arrangement of the 3 scramblers2/ the initial setting of the scramblers3/ the plugboardWhat the German Enigma operators did was, before enciphering a message, to use the daily key to produce a new 3-letter message key by arbitrarily choosing 3 plaintext letters, then typing those 3 letters in the machine to yield the 3 cipher letters that would be placed at the top of each new message.For example, if as part of the daily key, the initial scrambler setting was AGW, the German Enigma operators would not use AGW to encipher all messages. Instead, they would perform the steps below in the specified order:Choose arbitrarily 3 plaintext letters such as UHY for the message keyType UHY into the machine and noted the corresponding cipher letters. Let’s say they were OIQ.Write OIQ at the top of the cipher text. OIQ was the enciphered message key.Reset the scramblers to UHY and type the plaintext message to produce the cipher.At the receiving end, another German Enigma operator would have a machine set up according to the daily key in the code book, with the 3 scrambler setting being AGW. Once the cipher text above was received, the operator would do the following in the specified order:Type in OIQ at the top of the message which would reveal UHYSet the scramblers to UHYType in the cipher text to recover the plaintext message.Cool huh?Now, due to the possibility of interference which might inadvertently alter the enciphered message key, Enigma operators were instructed to encipher the 3-letter key twice. The purpose was to provide a way for the receiver to double check if the message key appeared twice next to each other before deciphering.For example, if the initial scrambler setting was AGW, the message key was UHY, then the operator would type UHY twice to yield something like: OIQKJH. Then OIQKJH would be placed at the top of the new message.Unbeknownst to the Germans, by enciphering the message keys twice successively, they unwittingly gave Marian Rejewski a way to crack the ciphers.Here is howEveryday, Polish military intelligence intercepted a huge number of radio messages, all enciphered by Enigma. As explained above, all of these messages began with the 6 letters placed at the top representing the 3-letter message keys enciphered twice. For example,Figure 1: example of the 6-cipher-letters for a message key (Image source: The Code Book - Simon Singh)Although Rejewski did not know the internal wirings of the scramblers, he did know that the following had to be true of the lines above:1/ The 1st and 4th letters are cipher letters of the 1st plaintext letter of the message key2/ The 2nd and 5th letters are cipher letters of the 2nd plaintext letter of the message key3/ The 3rd and 6th letters are cipher letters of the 3rd plaintext letter of the message keyFrom this fact, Rejewski was able to deduce some constraint on the initial setup of the machine on any day which was that: there existed a relationship between the 1st and 4th cipher letters, 2nd and 5th cipher letters, and 3rd and 6th cipher letters.For example, in figure 1:Message 1: L and R, O and G, K and M were related to each other.Message 2: M and X, V and Z, T and E were related to each other.and so onNext came the crucial step that would lead to a deeper insight. Rejewski decided to tabulate all the related pairs of letters to see if any hint concerning the scrambler setting of the daily key could be discerned.For the messages in Figure 1, the following table relating the 1st and 4th letters could be produced:Figure 2: partially filled relationship table between the 1st and 4th cipher letters (Image source: The Code Book - Simon Singh)As more messages were intercepted, Rejewski would have more information to fill in the table of relationship between the 1st and 4th letters which could be the following:Figure 3: fully filled relationship table between the 1st and 4th cipher letters (Image source: The Code Book - Simon Singh)Of course, this table still did not provide for Rejewski any hint regarding the day key. And the table above would be different in the next day when a different daily key and different initial scrambler setting was used to encipher message keys.Rejewski asked himself this crucial question: Was there a way of determining the day key just by looking at the tables of relationship?This question prompted him to look for any pattern in the tables. Eventually, he stumbled upon one type of pattern called the circular links or circular chains of letters. For example, in the table in figure 3:Letter A on the top row pointed to letters F in the bottom row. Rejewski proceeded to look at F in the top row which pointed to W in the bottom row. He looked at W in the top row which pointed to A in the bottom row which was the start of the chain. In effect, Rejewski had found a circular link.Further inspection of the table revealed more circular links as shown below:Figure 4: Circular chains of letter deriving from the table of relationship in Figure 3 relationship table between the 1st and 4th cipher letters (Image source: The Code Book - Simon Singh)In addition, Rejewski also noted the number of links in each circular chains.Rejewski would repeat this process for the tables of relationships between the 2nd and 5th cipher letters, the 3rd and 6th cipher letters, recording the circular chains and the number of links in each of those chains.It was at this point that Rejewski had a profound insight: Although the plugboard and the scrambler settings both affected the specifics of the chains, their effects could be, to some extent, separated. What this meant exactly was that there was one aspect of the chains that was affected totally by the scrambler settings but was totally unaffected by the plugboard settings.Here, an illustration will be helpful. From the example above, assume that the day key dictated that the letters S and G were swapped. What would happen if instead of swapping S and G, we swapped T and K instead? The chains would change the following:Figure 5: Circular chains of letter deriving from swapping T and K instead of S and G (Image source: The Code Book - Simon Singh)What do you see? Some of the letters have changed in some of the links (Letter K in the 2nd chain was changed to T and letter T in the 4th chain was changed to K), but the number of links of each chains did not change.Rejewski had made one tremendous breakthrough: the number of links in each chain was a consequence of the scrambler setting and not of the plugboard.Why was this significant? Well, as explained at the start, the plugboard accounts for the greatest number of ways of enciphering plaintext messages. Rejewski knew that the circular chains of letters were related to the daily keys in some way. Now he knew for certain that the one crucial aspect of the chains of letters which was the number of links wasn’t affected by the plugboard, he could safely ignore the plugboard setting for the time being and concentrate solely on a drastically simpler problem: which of the 105,456 scrambler settings yielded a particular number of links for a circular chain?(As a side note[math]105,456 = 26 \times 26 \times 26 \times 6)[/math] (where 6 = 3! = number of ways of arranging the 3 scramblers)Now, although 105,456 scrambler settings was still a very large number, it was at least 15,896,255,521,782,636,000/105,456 = 150,738,274,937,250 times smaller than the total number of possible daily keys!!! More importantly, by putting more people to work, it was certainly within the realm of human endeavor to check which of the 105,456 scrambler settings resulted in a particular number of links.Rejewski proceeded as follows. Thanks to Hans Thilo Schmidt’s stolen document, he could construct a replica of the Enigma. He and his team painstakingly examined each of the 105,456 scrambler settings and catalogued the chain lengths generated by each one.It was an endeavor that took about one full year to accomplish. Once the Cipher Bureau had compiled the chain lengths and the corresponding scrambler setting, Rejewski arrived at the final step in the codebreaking process: determining the day key for messages intercepted within a day.Each day, he would examine the 6-encrypted message keys and use the information to build the tables of relationship between the 1st and 4th, 2nd and 5th, 3rd and 6th letters as shown above. From these tables he would identify the chains and their corresponding lengths.Once all the chains had been identified, he would consult the meticulously compiled catalogue which contained all scrambler setting and the resulting chains. He would compare the lengths of the chains for a day’s intercepted messages to those in the catalogue. Once he found the entry that matched, he had successfully found the scrambler setting of the daily key!! As you can see, the chains could be likened to the fingerprints of the scrambler arrangement and internal wirings and could therefore be used to find the daily key.Now that Rejewski succeeded in determining the initial scrambler setting, he had one final problem to tackle: the plugboard settings. Using 10 wires yielded a total of 150,738,274,937,250 ways of swapping pairs of letters. But, figuring out which pairs of letters were swapped was surprisingly easy. Rejewski would first set the scramblers on his Enigma replica to the already identified scrambler setting. Then he would remove all cables from the plugboard so as to prevent the plugboard from having any effect on the production of the ciphers. Then he would type in the machine an intercepted cipher text and examined the resulting plaintext message. There would be a lot of gibberish in the plaintext message because there was no swapping of the letters by the plugboard cables. But, he could compare the individual words that were incorrectly spelled to the correct spelling of those words and from that deduce the swapping of letters. For example, if there appeared in the plaintext message the phrase alliveinbelrin, he would know immediately that the correct phrase is arriveinberlin indicating that the letters l and r must be swapped, meaning that a cable for swapping L and R had to be plugged in. By this method, Rejewski could deduce the plugboard setting and ultimately the full daily key.And that, ladies and gentlemen, was how the Polish codebreakers cracked the supposedly unbreakable Enigma ciphers.The Polish army continued to use Rejewski’s technique for several years. When the Germans made a minor modification to the machine which rendered the Poles’ meticulously compiled catalogue of scrambler settings - chain lengths useless, Rejewski responded with an ingenious invention. Instead of manually compiling another catalogue, he built a machine that was essentially a mechanized version of his cataloguing system, which could automatically and rapidly search for the correct scrambler settings. It was called the Bomba. Because of the 6 possible scrambler arrangements, the Polish Cipher Bureau built 6 bombas working in parallel, each of which checked through 17,576 scrambler settings until one was found.All in all, the Polish cryptanalytic success was due to 3 factors:Determination engendered by threat of invasion.Hans-Thilo Schmidt’s treachery.Ingenuity, particularly that of Marian Rejewski.While it was true that without Hans-Thilo Schmidt’s treachery, the Poles would not have had the information needed to produce an Enigma replica to check the scrambler settings for the daily key, this did not in any way diminish the accomplishment of Marian Rejewski and his fellow Polish codebreakers. As explained before, the security of the Enigma machine consisted in keeping the code books containing the daily key settings rather than the machine itself from falling into enemy hands. The Polish mathematicians had broken Enigma ciphers without any code book. This was a tremendous accomplishment in cryptanalysis.And here was one interesting obscure fact, Hans Thilo Schmidt provided not only blueprints for the Enigma machine but also the daily keys. In secret meeting with the Polish Cipher Bureau’s chief, Major Gwido Langer, Schmidt provided him with the codebooks, and this secret exchange lasted for 7 years! But instead of giving them to Marian Rejewski and his fellow codebreakers, Langer kept the keys hidden. This begs the question: why didn’t he give the day keys to the Polish codebreakers? It certainly would have saved a lot of effort and simplified the codebreaking task.The answer was: the astute Langer had foreseen that one day Schmidt might no longer be able to help for whatever reasons. Hence, he wanted Marian Rejewski to be challenged so that his ingenuity could be exercised that would enable him to deal with unexpected cryptographic challenges in the future. In other words, Langer wanted his codebreakers to be self-sufficient. And Marian Rejewski had overcome that challenge brilliantly: he cracked Enigma ciphers without the daily keys provided by Hans Thilo Schmidt.Much respect to the achievements of Polish codebreakers.British cryptanalytic achievementNow that I have made my tribute to the Polish codebreakers, it is time to make a tribute to the British codebreakers at Bletchley Park.This would not be complete without a brief discussion of the course of events leading to the British codebreaking endeavor.In 1938 saw the end of success for the Polish codebreakers. After years of successfully breaking Enigma ciphers, German cryptographers made significant changes that defeated further Polish attempt to break Enigma ciphers. One change was the introduction of 2 new scramblers, raising the number of scramblers from 3 to 5. The number of scrambler arrangements had been increased from [math]3! = 6[/math] to [math]5\times 4\times 3 = 60[/math]. To break the new Enigma ciphers, the Poles now had to determine not only the internal writings of the 2 new scramblers but also which of the 60 scrambler arrangements was used for the daily key. The second major change was the increase of the number of cables from 6 to 10, meaning that 10 instead of 6 pairs of letters could be swapped.These two major changes effectively prevented the Poles from breaking Enigma ciphers; and they were made at the worst possible time: just one year before Germany’s invasion of Poland.In desperation, the Poles decided to share their codebreaking knowledge with the British and French. They were invited to a forrest where Marian Rejewski revealed to them not only how he had cracked the Enigma but also his bomba machine. The British were astonished. They had previously thought the Enigma was unbreakable. The Poles had proved them wrong.So now, with the transferred knowledge of the Polish codebreakers and a fierce sense of urgency inspired by the specter of war with Nazi Germany, the British codebreakers committed themselves to one of the biggest cryptanalytic challenges in history: breaking the Enigma machine. Recognizing from Polish achievement the importance of mathematical thinking in breaking the Enigma, the British recruited not just linguists but also mathematicians, among whom was the eccentric genius Alan Turing. He would play a critical role in breaking the strengthened Enigma ciphers.Alan TuringIn essence, Alan Turing thought long and hard about how to crack the Enigma if the Germans changed their system of transmitting message keys. Initial British codebreaking success was based on the work of Rejewski, and as explained above, it was based on the fact that the Enigma operators enciphered the each message key twice successively. Such repetition was intended as a mean for double checking to ensure that the message key was transmitted without error.But Alan Turing correctly predicted that eventually the Germans would realize that the repetition was a chink in the armor of their enciphering process and it would not be long before they stopped doing that. Their prediction was vindicated: On May 1st 1940, the Germans stopped enciphering the message key twice. Alan Turing now concentrated on the problem of cracking the Enigma without relying on repeated message key.Alan Turing examined all the decrypted messages. He noticed that many of them conformed to a rigid structure. The regularity with which these messages conformed to a structure implied that he could correctly deduce with relatively high confidence the plaintext content of a message based on when it was sent and its source.For example, decrypted messages indicated that the Germans regularly sent enciphered weather report after 6:00 AM each day. So any message intercepted at, say 6:15 AM, would very likely contain weather-related terms. Furthermore, because of the strict protocol and commitment to discipline of the German military, it followed that the messages were structured rigidly in a particular style. This implied that Alan Turing could guess with high certainty where the weather-related terms would appear in a ciphertext. For example, a 6-letter cipher word would likely correspond to the plain word wetter (German for weather). When a piece of plaintext could be matched with a piece of ciphertext, this matching was referred to as a crib.You may now ask: How were the cribs found?Well, it was really simple. As explained above, the messages conformed strictly to a rigid structure. The codebreakers would make a well-educated guess of the underlying plaintext, then align the guessed plaintext with the ciphertext. Of course, this was guess work based on past knowledge and hence there was some amount of uncertainty inherent in it. Despite this, a combination of past knowledge and intuition meant that the British codebreakers were fairly or absolutely certain about the existence of a particular plaintext in the cipher. The figure below illustrates the alignment process to find a crib:Figure 6: finding a crib illustration (Image source: The Code Book - Simon Singh)Now here was a neat trick to check if an alignment was correct or not. It exploited a characteristic of the machine: its inability to encipher a letter as itself, meaning that a,b,c etc… could never be enciphered as A, B,C, etc… respectively. This was a direct consequence of the reflector inside the Enigma.Alan Turing realized that he could use the crib to crack Enigma ciphers. If he had a ciphertext and he was certain that a specific section of it, say UYQTWK corresponded to the German word wetter, then the challenge was to establish the scrambler settings of the Enigma machine that transformed wetter into UYQTWK.Alan Turing leveraged the insight of Rejewski, which was that the effect of the plugboard and the scramblers could be disentangled. If he could find something in the crib that he was certain wasn’t affected by the plugboard, then he could check all of the [math]60 \times 17,576 = 1,054,560[/math] scrambler settings. Once the scrambler setting was uncovered, he could deduce the plugboard setting, just like Rejewski had done.The crib that Alan Turing sought after contained internal loops, very much similar to the circular chain links identified by Rejewski. One such looping crib is shown below:Figure 7: a looping crib (Image source: The Code Book - Simon Singh)However, as explained above (about how the Germans would stop enciphering message key twice), Turing’s crib had nothing to do with the message key. Instead, his cribs related cipher letters with plain letters. Also, cribs were largely educated guesses meaning that they could be wrong. This was another key difference between Turing’s crib and Rejewski’s circular chains.Now, here came the crucial step in Turing’s thought process that was truly brilliant. He saw that the loop in the crib pointed to a drastic shortcut to break Enigma ciphers. Instead of testing every scrambler setting using just one machine, he would use n separate machines, each testing the encipherment of one element of the loop. Here, n = number of elements or relations in the loop. In this example, the loop contains 3 elements so n = 3. So there would be 3 machines used, each of which would do the following:the 1st machine would try enciphering w into Ethe 2nd machine would try enciphering e into Tthe 3rd machine would try enciphering t into WThe 3 machines would all have identical scrambler settings. But the 2nd would have its scrambler orientations moved forward one place relative to the 1st machine (S+1 in figure 7), the 3rd would have its scrambler orientations moved forward by three places relative to the 1st machine (S+3 in figure 7).Then, Turing’s imaginative mind alighted on a brilliant idea that would enable him to isolate the plugboard to find the correct scrambler setting. The idea was to connect the 3 machines by running electrical wires between the inputs and outputs of each machine as shown below:Figure 8: Electrical loop representing the looping crib (Image source: The Code Book - Simon Singh)The loop in the crib was simulated by the the electrical circuit loop. With this setup, the codebreakers could determine which scrambler setting yielded the desired crib because the correct scrambler setting caused the circuit to be completed. How could one know if the circuit was completed? By incorporating a light bulb as shown in figure 8. A shining light bulb indicated that the circuit was completed, implying the correct setting had been found. Otherwise, the setting was incorrect and other scrambler settings would be tried. With this circuit setup, Alan Turing was able to isolate the effect of the plugboard to solve a drastically simpler problem of finding the scrambler setting. Once that was accomplished, finding the plugboard setting was easy: Employing the same method Rejewski had used as described above.The machines built to find scrambler settings were the famous Bombe. The codebreaking process began with identifying the cribs to be done by codebreakers. The cribs would be passed on to the bombe operators. Then wait for the result which could take several hours or days.Figure 8: a bombe in operation (Image source: the Code Book - Simon Singh)So in short, it was a combination of solid common sense for finding cribs and simulating the cribs by electrically connected machines that enabled British codebreakers to crack Enigma ciphers. It was truly a remarkable cryptanalytic achievement, one that was made possible by the brilliant mind of Alan Turing.Unfortunately, notwithstanding his tremendous contribution to the Allies’ effort of military intelligence, Alan Turing was mistreated by his own government for homosexuality. It makes me wonder what would have happened if he was persecuted for that during the war. Probably it would have taken far longer for his less brilliant fellow codebreakers to find a way to crack the Enigma. This would have had disastrous consequences for Britain because German U-boats were sending tons of merchant shipping to the ocean floor. Plus, not knowing the German army’s intention in Africa and Europe, the Allies would not have been able to mount successful offensive/defensive operations.I am echoing Churchill’s famous words:Never was so much owed by so many to so fewMuch respect to Alan Turing and other British codebreakers at Bletchley Park. They were the unsung heroes of WW2 for without them, the war would have dragged on and consequently more lives on both sides would have been lost, and very likely Germany would have held the unfortunate distinction of being the 1st nation to be destroyed by atomic bombs rather than Japan.Reference(s)1/ The Code Book - Simon Singh

I was asked to do a piece of consultancy for a team at work - they had developed a fully compliant 7 layer ISO stack (it was a fad). it was written in C so in theory was source code portable all the way down.They had load tested it at around 500 messages per minute on Unix, but when they ported it to DEC/VMS performance fell through the floor - to around 20 messages per minute. I was known as a bit of an expert on DEC/VMS so I was asked to assist.I looked at all the key things - CPU load, memory load, swap thrashing and they all seemed normal. I then noticed a very large amount of disk I/O, and apart from the executable itself, the load tested was opening only one other file - which held the addresses, ports, Application Points etc.This last file wasn’t opened once - it was being opened hundreds of times per message - every single time and address, port or anything else needed to be converted, the file would be opened, read and closed.From my DEC/VMS experience opening/closing files was actually pretty slow - and the solution was a 10 line code change to open the file once and cache the contents; the performance improvement was about 22 fold - and the DEC/VMS performance was more or less comparable with Unix.Addendum: One small detail I neglected to mention - my consultancy was a day in Belfast - in 1993 or so (before the Good Friday Agreement). It was a fun taxi ride from the airport to the hotel - with driver admitting that a week before on the same road a sniper had shot one of his colleagues. I then managed to get a bout of food-poisoning in the hotel, so my meeting the next day I was already starting to feel ill and at least a little bit scared.My day ended at Belfast airport with two armed policemen asking if would be ok to travel after I finally succumbed to the effects of the dodgy food.

There’s a need of some kind. After that, it varies.The Special Case of IntelIntel has been a big creator of universal standards. A big part of this is Intel’s unique position as the driver of the PC industry. Basically, Intel decided a long time ago that other companies aren’t big competition, but other standards might be. So when they see a need, such as the standardizing low-cost PC audio, expansion ports, high speed interfaces, memory interfaces, etc. they develop a technology. Usually, the point of that is to advance PC technology, and ultimately, to drive the demand for new PCs.New standards drive new PC purchases, and it’s much more difficult to get a proprietary technology established as a standard. So Intel creates a technology, patents it, forms an independent groupt to manage that standard, and licenses the patents for that specific use for free. That makes the standard painless for everyone to use, but also discourages others from making something slightly different. It helps prevent the old ‘embrace and extend” practice, wherein a company takes a standard and adds some proprietary “special sauce”, such that the older stuff works with their system, but their new stuff is effectively proprietary. That can break standards, and it’s been tried at various times.PCI and PCI ExpressA case in point is the PCI Bus, and later, PCI Express. Intel had basically been a bystander in the world of PC expansion in the 1980s, adding others work to their chipsets, like the PC-AT/ISA bus, etc. In the early days, the PC-AT expansion bus was really just a buffered extension of the Intel 8086 bus with some additional signals to the specific motherboard chips Intel had used in the PC-XT and had added for the PC-AT. It was a design of convenience, not something terribly well thought out.For compatibility, that bus stuck around as PC computers got faster, even as they went to the 80306 and 80486 buses, which had been optimized for other things, no longer a great match to the PC-AT bus. So eventually, you looked at PCs delivering some things on the “local’ bus — the direct connection to the PC’s CPU, and some things on the ISA bus, because that was an ad-hoc standard.Intel certainly knew they had no interest in being locked into a particular local bus design. So in order to prevent the 80386 or 80486 becoming any kind of standard (well, almost.. see VL-Bus), they introduced the PCI bus. Rather than being designed as the best interface for everything a CPU needed at the time, it was designed as a very good interface for I/O chips. And being CPU independent, it didn’t vanish when the ‘486 was replaced by the Pentium, the Pentium by the Core, etc.PCI Express was in effect a modernization of the PCI interface. It’s nothing like it in hardware, but seems to be the same kind of thing to hardware. Again, the point was selling chips, but also delivering a version of a technology everyone would use, free, before someone else came along with a technology that solved the same problem but required payment of a license.Everyone Loves a Standard… Especially Just One StandardMost new things are driven by some need. For years, PC companies pushed for different media standards, but they ultimately lost most of those battles to the consumer electronics industry. The PC folks are great at establishing interfaces — again, particularly Intel — but not so much on media.But even the media standards were often a battle. Sony and Philips got together and created the Compact Disc, specifically as a way to deliver digital audio. The PC industry saw the success of that and figured out how to use it for PC data as well. That one went pretty smoothly.Of course, then there the inevitable sequel to the CD. Philips and Sony were hard at work on the MMCD (Multimedia Compact Disc), a higher capacity followup disc. Then Toshiba and Panasonic showed up with their own disc, the Super Density Disc (SD), supported by Panasonic, Time Warner, and a bunch of other industry leaders.In this particular case, the industry actually solved the problem. The SD people has asked for IBM’s advice on the file system, and IBM already knew about the development of the MMCD, so they called together the big guns of the PC industry to help avoid another format war — everyone remembered the days of VHS vs. Betamax, and the PC industry, committed by then to CD, was going all in on a single upgrade. They formed the Technical Working Group (TWG) and said they’d back only one standard. The resulting final Digital Versatile Disc used technologies from both MMCD and SD.A Few People Love a Format WarThings were not quite so smooth when it came to the successor to the DVD. Sony had actually jumped way ahead and developed much of what would become the Blu-ray technology. The limit was the lasers used, and it wasn’t until 2000 that Sony and Philips collaborated, again, on some blue laser technology that led fairly quickly to prototype higher density discs in 2001. By 2002, the technology had been named “Blu-ray” and an industry organization had formed around it, as a technology, not really a product yet.Sony released a disc format called Professional Disc, and a recording drive for use with Sony’s XDCAM tapeless camcorders, in 2003. That format required a cartridge to keep it safe, as the data layer is very shallow, and no other consumer components were in place. The industry group worked together and came up with the hard coating used on modern Blu-rays, the DRM encryption system, the basic data structures, the use of Java, etc… many things that together delivered the final Blu-ray format specifications in 2006.At the same time, Toshiba has been working with Microsoft and the DVD organization on their upgrade of the DVD, which they dubbed HD-DVD. The two groups actually met in 2005 in an attempt to avoid a format war, but failed to come to any kind of agreement. Like Sony, Toshiba had lined up a group of companies and studios backing their format, and they had something of a lead on the Blu-ray efforts, given their simpler overall software system.So Toshiba launched HD-DVD, Sony and others launched Blu-ray. It was a battle for a while. Sony needed all the blue lasers they could get for their PS/3 game console, so they didn’t hit the market with strong Blu-ray player presence. That actually made it easier for other companies, like Samsung, to deliver their players.Meanwhile, Toshiba was getting money on the media, thanks to royalties on the HD-DVD disc design, so they could pull the video game console trick of selling their hardware at or possibly below cost. That gave HD-DVD an initial boost, but it also created a problem: no other company delivered a real HD-DVD player. The closest they got was one Samsung model that was essentially a Blu-ray player that could also play HD-DVD discs. The only non-Toshiba-branded HD-DVD players were re-badged Toshiba units sold by product marketing companies like RCA. Ultimately, HD-DVD started losing studio support, and Blu-ray won the day.When UltraHD discs started to be discussed, Blu-ray was already supporting 100GB disc technology via the BD-XL format, so there was no real attempt at disc-based competition. And of course, the disc as a delivery medium is less important than it once was, with video streaming over the internet a popular thing for many consumers. And far as PC technology goes, the PC folks pretty much waited around until the format war was over before embracing the higher data capacity. I know I did… I was waiting out the format war, I didn’t get any Blu-ray gear until Toshiba cried “uncle”.Remember, Sometimes It’s a Battle RoyaleSometimes it’s a little more wild west. Take memory cards…. as they say, the great thing about standards is that there are so many of them. Various different entities in the consumer electronics market came up with different memory card standards, and maybe more of them than you’d expect.One of the first memory cards in common use came from the PC industry, in the PCMCIA Card. This standard was published by the Personal Computer Memory Card International Association in 1990 and got wide support by the PC industry, both as a memory card but even more as a general expansion card. Soon, this was the standard for adding devices of all kinds to laptop computers. But it was too large for small electronic devices like digital cameras.The Compact Flash people basically took a PC standard, the PCMCIA expansion card for laptops, and shrunk it down to fit inside of devices. An industry group called the Compact Flash Association has driven the evolution of this card, one reason it’s lasted a very long time in the industry.The Miniature Card was defined by Intel in 1995, and backed by Advanced Micro Devices, Fujitsu and Sharp Electronics. It supported flash or SRAM memory, but never really caught on among device makers.The SmartMedia Card was a proprietary card from memory maker Toshiba, and supported by various camera companies. Ironically, it was one of few memory cards that wasn’t actually “smart”… it was basically just a NAND Flash chip bound to a card (most cards have a controller chip that abstracts the card interface from the flash memory interface).The Multimedia Card was launched by SanDisk and Siemens AG, with a card small enough to be used for portable media players and in one form (TransFlash), actually pushed for awhile as a possible delivery media (like CD, Cassette, or LP, they had planned to sell music on these cards).The Secure Digital Card is a spin-off from Multimedia Card, pushed by memory manufacturers SanDisk, Panasonic, and Toshiba. It was sold as an improved card: more room for memory, better keying to prevent wrong insertion, faster transfers, and a security mechanism for DRM. The SD card has been a huge hit, and has supported continual evolution in the market as a result of its success. Leading, of course, to more success.The Memory Stick was launched by Sony as a proprietary memory card. They had hoped to use the high cost of flash storage as a way to keep you in the family: everything Sony made that needed a memory card used the Memory Stick: still cameras, PDAs, video cameras, etc. As other cards caught on across product lines, this turned into a liability, a reason NOT to buy Sony devices.The xD Picture Card is essentially a mini-SmartMedia card, created by camera makers Olympus and Fujifilm. It had some of the same flaws as SmartMedia card, and was effectively proprietary to Olympus and Fujifilm.The SxS Card is a memory card for high-end camcorders, essentially a Sony proprietary card, based on a shrunk down version of the ExpressCard expansion used for awhile by more modern laptops. It’s based on PCI Express.The P2 Card is Panasonic’s memory card for high-end camcorders. These are usually implemented as a RAID of SD cards. They’re based on the PCMCIA card, but not 100% the same.The CFast Card is a Compact Flash Association supported standard that uses the Serial ATA interface for higher speed transfers. This is intended to replace regular Compact Flash cards in cameras and other devices; it’s not compatible with the older CF cards.The XQD Card is a standard using PCI Express, started by a collaboration between SanDisk, Sony and Nikon in 2011. It was later picked up by the Compact Flash Association, despite being a direct competitior to the CFast card.The CFexpress Card is yet another Compact Flash Association card. It hasn’t been finalized yet, but it’s essentially a Compact Flash sized card using PCI Express rather than the SATA link in the CFast card. The CFexpress Card supports up to eight PCI Express lanes, so it ought to be crazy fast.So with all these cards, it’s pretty clear there’s chaos. And how they come together… sometimes they don’t. Sony didn’t consult anyone else about the Memory Stick, though they did eventually license SanDisk to make some. Olympus and Fujifilm made the xD card together, probably against some sage advice about going to standards, which they ultimately did — they both use SD cards today.The two memory cards that have lasted are the Compact Flash card and the Secure Digital Card, both cards managed by industry groups, not just a single company or two. They have another thing in common — they have continued to evolve over the years. New interface modes and new interfaces have boosted performance, etc. And the mechanical designs have proven robust enough for consumer and professional use. But it all worked out in the marketplace… more that than any specific coming together.And the race continues now. The CFast card is relatively new (2009) and fast (300MB/s early on, 600MB/s as of CFast 2.0 in 2014)… but the UHS-II SD card spec runs 312MB/s, and the new-this-year UHS-III spec supports 624MB/s. So much for the new guys. Though XQD already supports a faster interface, and CFexpress promises to deliver twice the link speed of a M.2 x4 PC SSD card. So the market is still churning around on these things, and most companies are actually sticking to SD and watching what happens with XQD, CFast, and all on the high-end.