Metadata's gift to historians

The bulk collection of metadata is a much-appreciated gift to future historians. There remain questions about whether it is effective as public policy but it is certainly outstanding for the historical record. As historians, the private information of ordinary citizens is our best window to the past. Once upon a time this meant hunching over the census, or private correspondence, or love letters and business records. All of which are vital in creating accurate networks of property ownership, political organisation and community.

As law enforcement instruments, both the United States and German governments have questioned the evidence in support of data retention laws. Germany repealed their retention laws after concluding the "marginal increase in the clearance rate [of serious crimes] by 0.006% could raise doubts about whether the provisions in their current form would stand their ground under a proportionality review. In any case, the relationship between ends and means is disproportionate." It's a further bad omen when even the law's champions in Australia, like Malcolm Turnbull and Tim "Freedom" Wilson, are advising the public on ways to circumvent domestic data retention.

The recent USA Freedom Act forces data retention in the United States from federal government servers to private corporations, which is where Australian data will also be held. This is no comfort if you remember the ease and frequency of recent hacks. Government whistleblower leaks from Edward Snowden or hacks of the U.S. Office of Personnel Management (exposing the confidential details of 21 million government job applicants and their spouses) get a lot of press, but that's compared to over 100 major private sector hacks since 2005. And that does not include the recent high profile hacks of Ashley Madison and Hacking Team, who are themselves hackers-for-hire, employed by governments and intelligence agencies.

As a well-meaning historian in the data-capturing world that is coming to a gadget near you, I look forward to perusing your metadata for the benefit of future generations. Granted, American and Australian telecoms are only meant to keep the data for a few years and never share it with the public. But with strong precedent of private sector hacks and intelligence agencies (ASIO, Australia) overstepping the line and permanently (National Security Archive, George Washington University) keeping these marvellous (FBI COINTELPRO) records (NSA PRISM), I am hopeful that this will again be the case.

Beyond the mere metadata, the transcripts of phone conversations, text messages and emails would be of most use to historians, and also to intelligence agencies, who insist they do not collect them already. These sources are especially useful since they're now full-text searchable. For security personnel this means less time wasted on scrolling through low-threat individuals. Simply type in a keyword and click "find." For historians, this means liberation from months hunched over a microfilm reader on the other side of the world, only to glean one footnote, destroy our eyes, and deplete our bank accounts.

The audio of telephone conversations would be doubly valuable. Australia has nothing like Story Corps, aimed to interview so many Americans that it captures the everyday triumphs and tribulations of a nation of 350 million souls. So many ordinary Americans have contributed to Story Corps that within the past year, purely by accident, one of them has been arrested for running the largest drug ring in the history of the internet (Ross Ulbricht has since removed his oral history from Story Corps, but it is still available on YouTube) and another made international news when she was murdered in a Chapel Hill parking garage in a racially-motivated attack (her oral history is available here).

There is something about audio that keeps us honest, unfiltered. In the words of Ira Glass, sage of the popular podcast This American Life, "On the telephone we are who we truly are. Some of the time, anyway." He was talking about generations of audio recorders, like Lyndon B. Johnson, who, as President, installed the first comprehensive recording system in the Oval Office, and used it to record meetings, phone conversations, and most famously, his trouser measurements. His predecessors and successors throughout the twentieth century also recorded their meetings and telephone conversations for posterity.

Historian Michael Beschloss insists that without these recordings of LBJ the man would be entirely lost to history. "He's more honest on the phone than he was in public. In public, Johnson said that he believed the Warren Commission report. On the telephone, he admitted he didn't believe in it. He didn't believe that a lone assassin, Lee Harvey Oswald, acted alone killing JFK. In public, Johnson supported the Vietnam War. On the telephone, he admitted his doubts." In the private words of LBJ: "I don't see what we can ever hope to get out of there once we're committed."

Since our nations' leaders have decided to retain the metadata of ordinary citizens, those same leaders should level the playing field and install recording equipment in their offices and on their telephones, and lock those recordings in the National Archives for the standard 30 years. Fair is fair. More than trapping politicians in petty hypocrisy, these recording would allow future generations to hear our leaders at their most honest, unafraid of consequences, unjudged by anyone but history. After all, millions of words have been written about Presidents Lincoln, Washington, and Obama, but we will never truly know another president like we knew Johnson and Nixon, because against their interest, they were the last to speak for themselves.