Features

net.wars: The shape of the mushroom

The digital universe is big. Really big. You just can't believe how mind-bogglingly big… Oh, never mind.

There's nothing like a good the-sky-is-falling scenario to get a one-day conference awake, and today at the LSE was no exception.

"It's a catastrophe waiting to happen," said Leslie Willcocks, the head of the Information Systems and Innovation Group at the LSE, putting up a chart. What it showed: the typical data centre's use of energy and processing power. Only 1.5 percent of the total energy usage powers processing; 80 percent of CPU is idle. Well. They weren't built to be efficient. They were built to be reliable.

But Willcocks wasn’t gearing up to save the planet. Instead, his point was that all this wastage reflects a fetish for connectedness: "The assumption is you have to have reliable information on tap at all times." (Cue Humphrey Appleby: "I need to know everything. How else can I judge whether I need to know it?")

Technology design, he argued, is being driven by the explosion in data. The US's 28 million servers today represent 2.5 percent of the US's electricity needs; in 2010 that will be 43 million. This massively inefficient use of energy is trying to fix what he called a far bigger problem: the "data explosion". And, concurrently, the inability to manage same.

In 2007, John Gantz, chief research officer at IDC, said, for the first time in human history the amount of information being created was larger than the amount of storage available. That sounds alarming at first, like the moment you contemplate the mortgage you're thinking of taking out to buy a house and realise that it is larger than the sum of all your financial assets. At second glance, the situation isn't quite so bad.

For one thing, a lot of information is transient. We aren't required to keep a copy of every TV signal – otherwise, imagine the number of copies we'd add every Christmas just for rebroadcasts of It's a Wonderful Life

But once you've added in the impact of regulatory compliance and legal requirements, along with good IT practice, consider the digital footprint of a single email message with a 1Mb attachment. By the time it's done being backed up, sent to four recipients, backed up, and sent to tape at both sending and receiving organizations it's consuming over 51.5Mb of storage.

And things are only going to get exponentially worse between now and 2011. The digital universe will grow by an order of magnitude in five years, from about 177EB in 2006 to 1,773EB in 2011. More than 90 percent of it is unstructured information. Even more alarming for businesses is that while individual consumers account for about 70 percent of the information created, enterprises have responsibility or liability for about 85 percent of it. Think Google buying YouTube and taking on its copyright liability, or NASA's problem with its astronauts' email

"The information bomb has already happened," said Gantz. "I'm just describing the shape of the mushroom."

To be sure, video amps up the data flows. But it's not the most important issue. Take, for example, the electronification of the NHS. Discarding paper in favor of electronics saves one kind of space – there's a hospital in Bangkok that claims to have been able to open a whole newpaediatric wing in the space saved by digitising its radiography department – but consumes another. All those electronic patient records will have to be stored, backed up and stored and backed up again in each new location they're sent to. Say it all over again with MP3s, electronic patient records, digital radio, VOIP, games, telematics, toys…

No wonder we're all so tired.

And the problem the NHS is solving with barcoding – that people cannot find what they already have – is not so easily solved with information.

Azeem Azhar, seven months away from a job as head of innovation at Reuters, said that one thing he'd learned was that every good idea he had – had already been had by someone else in the organisation at some point. As social networks enable people to focus less on documents than on expertise, he suggested, we may finally find a way around that problem.

The great thing about a conference like this is that for every solution someone can find a problem. The British Library, for example, is full of people who ought to know what to keep; that's what librarians do. But the British Library has its roots in an era when it could arrogantly assume it had the resources to keep everything. Ha. Though you sympathised with the trouble they have explaining stuff when an audience member asked why, given that the British Library has made digital copies, it should bother to keep the original, physical Magna Carta

That question indicates a kind of data madness; the information we derive from studying the physical Magna Carta can't all be digitised. If looking at the digital simulacrum evokes wonder, it's precisely because we know that it is an image – a digital shadow – of the real thing. If the real thing ceases to exist, the shadow grows less meaningful.