Can computers replace historians?

Tablet propped up on top of old books

All kinds of big claims have been made about the potential of Big Data. It seems it can predict the course of an election, map the spread of flu, even help police to solve crimes.

But here is the biggest claim so far - crunching through the big data of history can help us spot patterns and work out where the world is heading next.

That is what Kalev Leetaru, a data scientist at Washington's Georgetown University, believes may be possible. Using a tool called Google Big Query, designed for interrogating vast collections of data, he has been sifting through a database of events stretching back to 1979.

This is GDELT, which has collected media reports of events from innumerable sources in more than 100 languages for 35 years. "What we did here," Leetaru explains, "was use this tool to shove in a quarter of a billion records and use this massive piece of software to just in a few minutes sift out the patterns in this data."

What he says he found was complex patterns of events repeating themselves over the years. He has looked at recent events in Egypt , in Ukraine, in Lebanon and tried to draw common patterns.

"If you take the last two months of what's happened in Egypt for example, and you search every country's history for the periods in the past that are most similar to right now, and then you look at what happened after all those periods, that gives you a pretty good estimate of what happens next."

The graphs he draws in a blogpost about this project chart the volume of media interest in various events. Given the modern media's short attention span, this seems like a crude and unreliable way of examining history but Leetaru insists the underlying patterns are more complex than we might assume.

Scene from Tahrir Square, 2012 Tahrir Square, Cairo 2012: What does big data tell us about the pattern of events in Egypt?

He points to the continued ups and downs in coverage of Ukraine as one example: "You don't see this traditional burst of interest and then tailing off, you see this complex up-and-down movement over the two months after the protests started and it turns out this predicts that entire complex up-and-down cycle."

In other words, if you look deep beneath the surface of the output of thousands of journalists, they may indeed be providing us with the first sketch of history.

This project does sound like science fiction and indeed Leetaru talks of Isaac Asimov's writings about psychohistory as an inspiration. But the idea that computing power and artificial intelligence can now start replacing some intellectual disciplines as well as routine physical work will no doubt be greeted with anxiety as well as scepticism.

Leetaru says historians should see this kind of computational tool as just another technique amongst many rather than a threat to their professional expertise. In any case, they may look at the patchy record of big data in areas like election forecasting and flu trends and decide their days sifting through dusty archives are not numbered after all.

You can hear more about this story and plenty more on today's edition of Tech Tent on the World Service at 14:05 BST - or catch up on BBC iPlayer

Rory Cellan-Jones Article written by Rory Cellan-Jones Rory Cellan-Jones Technology correspondent

Kickstarter - now just a store?

How Pebble's new smartwatch is taking Kickstarter by storm

Read full article

More on This Story

More from Rory


This entry is now closed for comments

Jump to comments pagination
  • rate this

    Comment number 167.

    Could "history" (perceived or documented) have "predicted" the creation of the internet or the atom bomb, and their massive impacts on the world? Not since Guttenberg or gunpowder have such massive changes occurred, and historians would be hard put to deal with either.. Otherwise, odd how people poo-poo computers assisting in trends, while still believing in Nostradamus!

  • rate this

    Comment number 166.

    157. stevek.
    "The upside is that most financial data records would bite the dust too"

    What you mean like the records of the $2.3 trillion in transactions that were so destroyed in the so called attack on the Pentagon on 911? I am sure 'historians' will document a positive spin on this and claim there is no link. After all, the Bush government were totally honest never told us a lie... cough!

  • rate this

    Comment number 165.

    I know how to stop ISIS, stop selling them bullets. How do you think these backwards thinking people afford bullets? through the sale of heroin on our streets. So how about our governments stop playing this game and save us billions, you see its easy to predict the future when our governments play the same game over and over again!!.

  • rate this

    Comment number 164.

    "You don't need a computer to work out "where the world is going next"
    It's going down the drain as is the UK."

    There's a big difference between 'making history' and 'being history' - easy to which camp you're in.

    If you got out more you'd see just how good and how significant the UK really is.

  • rate this

    Comment number 163.

    162 GetMeOutOfHere

    "You don't need a computer to work out "where the world is going next."
    Its going down the drain as is the UK."

    I'm more optimistic about the UK. It's still one of the best countries in the world to live. We just need to be aware of the threat of the enemy within and challenge those religious extremists with the full weight of the law.


Comments 5 of 167



Copyright © 2015 BBC. The BBC is not responsible for the content of external sites. Read more.

This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.