The spy novel: past, present and future

In the decades that followed, the spy came back into literature and cinema with a vengeance. The exposure of British intelligence traitors like Kim Philby added fuel. Two streams of spy literature emerged, which produced not only novels but movies. One was a creation of Ian Fleming, who had been in British naval intelligence during WWII: James Bond. These movies have thrilled generations of cinema-goers but before long they became more and more farfetched and provided little competition for a new genre of movies based on spy thrillers written by John le Carré (real name David Cornwall) who had also worked with MI6. His books, and the movies made on them, like The Spy Who Came in from the Cold, are classics, though not appreciated by all.

Here's an observation by James Parker, a staff writer at The Atlantic in its December 2015 issue: "Writing involves betrayal, and le Carré – after his fashion and to our lasting benefit – double-crossed his own people. His Cold War novels were psychic microfilms of an Establishment hollowed out by deceit, denial, and inadequacy. They outraged his fellow spies." Parker was reviewing Adam Sisman's book, John le Carré: The Biography.

The United States has a rich field of spy novelists who have themselves worked in intelligence, particularly in the CIA. The late Charles McCarry, a former officer who operated under deep cover in Europe, Africa and Asia, created a series of "Paul Christopher" books that many readers regard as equal to John le Carré. Paul Theroux, the widely acclaimed American travel writer and novelist, has noted that, "No one understands Washington – its secrets, its politics, its plumbing – like Charles McCarry." Another writer, George V. Higgins, in a blurb for one of McCarry's novels, wrote in 1998 that he was the "best combination of spellbinding storyteller and silken prose writer. Nobody now at it writes better about intrigue, danger, loyalty, and betrayal than he does."

That choice of words points to why former spies write novels: they have seen a lot in their careers, with much of it unbelievable to the average person. Relayed in a non-fiction work – and that is not always possible, for reasons of secrecy, privacy or defamation – it becomes even more implausible. Truth can indeed be stranger than fiction. Fiction and characterisation are an ideal vehicle for conveying realities that most people would otherwise find difficult to accept.

Alongside spy novels produced by erstwhile practitioners is a proliferation of such books by authors with no direct experience of the spy world. The best of these writers have outstanding imaginative abilities, with some also having contacts in the intelligence world that help with precise details and atmospherics. Many of these writers' most ardent readers are men and women either currently serving in spy agencies around the world or who have done so. By any measure, that is an accolade.

But the art of spying has changed significantly from Cold War days. An astute British writer and security-policy expert, Edward Lucas, explored this in an article, "The Spycraft Revolution", in the Spring 2019 issue of Foreign Policy. Changes in technology, politics and business are all transforming espionage, he notes, and intelligence agencies must adapt or risk irrelevance.

This points to where the next generation of spy novels will come from.

Lucas observes that, "a cover identity that would have been almost bulletproof only 20 years ago can now be unravelled in a few minutes. For a start, facial recognition software – mostly developed by Israeli companies and widely deployed in China and elsewhere – allows governments and law enforcement agencies to store and search vast numbers of faces. They can then cross-check such data with the slew of personal information that most people voluntarily and habitually upload online."

Meanwhile, old staples of spycraft, he suggests, no longer work due to technological advances. Until recently, the dead-letter box was regarded as all but foolproof, an ideal location that both a source and a collection officer could reasonably visit. One party would leave behind intelligence material, perhaps stored on a tiny memory card enclosed in chewing gum. The other party would then collect it. Even a team of experienced observers would struggle to see what was really going on. But these days, such tactics aren't always effective. Counterintelligence can easily track, via mobile phone signals, the movements of suspected spies and the traitors they're paying who have access to the secrets they want.

Tradecraft, a spy's basic operating toolbox, is now inordinately more challenging, with other forms of biometric identification – like iris recognition and voice recognition – commonly in use. In an increasingly sophisticated technological environment the spy has less places, and means, to hide. Artificial Intelligence (AI) is but one of a modern spy's prime stalkers.

As Lucas puts it, "the same algorithmic techniques that digital security experts use to spot malware on networks and computers can easily be tweaked to highlight other unusual behaviour – sometimes much more effectively than human analysts could. Together, these techniques have severely constrained the ability of intelligence officers and their sources to operate safely and secretly. The cloak of anonymity is steadily shrinking."

But since the Industrial Revolution, the craft of spying has always managed to adapt to technological innovation, both in applying it as well as in defending against it.

One expanding area of traditional spying activity is the business world. The booming field of private intelligence companies is, in Lucas's words, "watching these techniques and their practitioners with a greedy eye." Indeed, he observes that the intelligence profession is increasingly overlapping with the corporate world; the realm of spies used to be cloistered. People who joined it never spoke about it and often served until retirement. Penalties for disclosure could include the loss of a pension or even prosecution. That's changed. For example, "anyone responsible for a company's cyber security now has to think like a counterintelligence officer." A stint at the CIA or MI6, Lucas suggests, has become a paragraph on a resume, not a career. Britain and the United States have caught up with Israel, where the private sector has long prized a spell in a senior position in intelligence or defence; in London and Washington, such work is increasingly a launch pad for an interesting career in corporate intelligence or other advisory work.

That's true. But the need for human intelligence gathering will be with us for many years. Nothing can replace face-to-face meetings where the subtleties and nuances in say, a top secret meeting in a president's office, are conveyed. In those areas, AI can't compete.

So, for aficionados of the spy novel, keep your seatbelt fastened. A new wave of storytelling is heading to a bookshop near you. Even the old Great Game in Central Asia is back, with China's Belt and Road project running right through the middle of it.