Avoiding a digital Dark Age

Advances in information and telecommunications technology present opportunities and risks for research and research data. These advances are propelling us into a new age of research. The question is, “Is this a golden age or a dark age for research information?”

Researchers now use ever-increasing volumes of data about our world. Ever more common and powerful digital instruments and sensors churn out more data in a single session than a human being could deal with in a whole lifetime. By some projections, in 2010, “there will be more data being generated [annually] than has ever been generated in human history up to 2006”.

Fortunately, the same information and communication technology (ICT) revolution has helped to deal with this tsunami of research data by empowering researchers to analyse, assemble, and share research data. This is largely due to the emergence of data management systems, high performance computing to manipulate large volumes of information, and infrastructure and protocols to network or federate information.

If it all works nicely, this represents a golden age for researchers: unlimited new online collections of data and research information with powerful tools for aggregating, analysing, and accessing that information.

But what are the risks?

Being able to preserve digital data is a must for research information, and a major risk is therefore the rapid obsolescence of digital objects. File formats, software, and hardware are constantly being superseded, so the curation of digital objects involves regularly migrating files into currently supported formats.

Who will do this for important research information long after the original research group has been disbanded? Memory institutions such as libraries and archives will have a role, but research disciplines must also assist in identifying the intrinsic qualities that need to be preserved during migration.

Important research collections need to be under the stewardship of a sustainable body committed to (and able to ensure) the continuity of access to these digital research assets. Otherwise these online research collections and datasets will never last long enough to revolutionise the way we do research. At worst a new digital dark age will follow where access to the previous generations’ information is severely compromised.

New research builds on previous research. In the new golden age, references to previous research or supporting data can include the actual digital objects or a link to the referenced digital object. Other scholars in turn refer to this research or include it in their digital works, and the new golden age builds on itself.

However, the mesh of information needs to be reliably persistent for future scholars to re-trace these cross-collection workflows. The risk of a dark age occurs if the whole information infrastructure for scholarly communications is not permanent enough. The simple URL “address” of the World Wide Web is insufficient. We need to use better systems of persistent identification to cope with changes of address; otherwise broken links will usher in another dark age for information.

The new model of scholarly information is decentralised with an unlimited number of online research collections hosted in various types of repositories, data centres, and custom web applications. The beauty of this Internet model is the organic growth of content by authors distributed all over the world. Combining this information into international grids is potentially part the new golden age of research.

The risk involved with this decentralised, distributed production model is the necessary diversity of systems and formats for storing the research data. Attempts to combine atmospheric data from around the globe can be stymied if the underlying data models are not compatible, so the tantalising possibility of aggregating our data remains elusive.