The spy novel: past, present and future

In 1900, after three years bringing the rebellious farmers of the Boer War in South Africa to heel, the bulk of Britain's regular army was outside the country. This helped fuel an invasion scare, which was made worse over the next few years by naval developments. The sheer scale, reputation and experience of the Royal Navy had given Britain undisputed supremacy of the seas, but the launching in 1906 of a new British battleship, HMS Dreadnought, triggered intense naval rivalry with Germany, which soon after announced an accelerated building program of its own. The British Government looked closely at this and other threats, and while many of the arguments put forward by invasion theorists were discounted, fear remained at a high level.

In the public domain, this threat was titillated by a most extraordinary force, one that exerted a powerful influence on the government and caused it to act in a way that defined the future of secret intelligence.

This was the spy novel.

One of the earliest such novels was Rudyard Kipling's Kim, the story of a child messenger used by British Intelligence, and set in India against the backdrop of the Great Game. By the beginning of the 1900s, a spate of similar novels was appearing, with titles like The Sack of London in the Great French War of 1901 and Starved Into Surrender. An early leader in this fiction world was Erskine Childers. His bestselling 1903 novel, The Riddle of the Sands, in the words of Winston Churchill, significantly influenced the Admiralty's decision to establish three major new naval bases around the country. This was followed in 1907 by Joseph Conrad's The Secret Agent, which was inspired by the attempt of a French anarchist to blow up the Royal Observatory in Greenwich in 1894; the man had killed himself in the process.

Writers from the Continent also fed the genre, but it was spy novels in Britain that had a curious effect. Of these, William Le Queux's work was to play, according to the late Phillip Knightley, "such an important role in the founding and development of Britain's first formal civilian intelligence service agency that an examination of his amazing background is essential."

Born in London in 1864, Le Queux had a French father and English mother. Educated partly in Britain and on the Continent, he spoke a number of languages well. Eventually he went into journalism, becoming a war correspondent for the London Daily Mail. He travelled widely and in the process became fascinated by espionage, even claiming to dabble in it himself. Novels on the spying theme soon followed and were immediately successful. He became obsessed with the "German menace" and insisted that a friend of his in Berlin had not only revealed to him the existence of a huge German spy network in Britain, but had also handed him a list of key traitors in Parliament, the Foreign Office, the Admiralty and the War Office. Despite his hounding of the authorities, he was not taken seriously until 1906 when he teamed up with a disaffected soldier, Field Marshall Lord Roberts, who shared the same obsession with the Germans.

Together, they concocted a fictionalised story about a German invasion of Britain, set four years later, which they managed to have serialised in the Daily Mail. It was an instant success, though it led to Le Queux being labelled a "scaremonger" in the House of Commons. The owner of the newspaper was unfazed; after all, the Daily Mail's circulation soared. The book version of the articles, The Invasion of 1910, sold more than a million copies in 27 languages, including in Icelandic and Urdu.

Le Queux continued with his spying activities abroad, and his counter-intelligence work inside Britain, writing more articles and flooding the War Office with more reports on the perfidious Germans. Another book, Spies of the Kaiser: Plotting the Downfall of England, appeared in 1909 and was an instant bestseller. Le Queux made himself the hero of his own story, claiming that it was based on his own personal inquiry into the presence on British soil of some 5,000 German spies. Because of its presentation as fact in fictional form, many of his readers took it as truth. Spy mania swept the country. Le Queux encouraged his readers to be on the alert and to report on suspicious Germans in Britain. Soon he was inundated with letters, which he passed on to the authorities. This coincided with the deliberations of a high-powered government committee appointed on the instructions of the Prime Minister, which was looking into the danger posed by German espionage.

The chairman of the committee – R.B. Haldane, the Secretary of State for War – was at first sanguine about the threat. But he too succumbed when "secret German documents", possibly planted by the French government, fortuitously came into the committee's possession. They purported to show Britain's most vulnerable points in the event that war was to break out. The committee therefore decided to act.

In 1909, Haldane recommended that a Secret Service Bureau be established, divided into two sections, Home and Foreign. The domestic branch should concern itself with catching foreign spies in Britain, in effect, counter-intelligence. This was the forerunner of today's MI5 (Military Intelligence, Room 5). The foreign branch would collect intelligence overseas and would become MI6 (Military Intelligence, Room 6), or the Secret Intelligence Service. The committee saw its detailed recommendations for the establishment of the service "so secret that it is thought desirable that they should not be printed or circulated to members."

Despite Le Queux's claims, on the day that war was declared on Germany in August, 1914, the Home Office reported that the authorities had arrested only 21 German spies. Following WWI, interest in spy novels waned, mainly because people had other things to worry about, like the Great Depression of 1929 that lead to high unemployment.

WWII saw the establishment by Britain of the Special Operations Executive (SOE) as well as the American Office of Strategic Services (OSS). The exploits of these organisations became better known after the War and gave rise to a new generation of spy novels and movies. The Cold War provided added momentum for spy thrillers, as did the CIA, which was established in 1947. Writers like Graham Greene emerged, carrying on the novelist tradition of earlier writers like Somerset Maugham, both of whom had worked for MI6. Greene's novel, Our Man in Havana, is by far the best send-up of spying ever published.