BBC BLOGS - dot.Maggie
« Previous | Main | Next »

America: Are you happy?

Maggie Shiels | 10:45 UK time, Friday, 23 July 2010

If you want to know what kind of mood America is in, check the Twitter feed.

That's what a group of scientists at Northwestern University and Harvard did and the conclusions make for interesting reading.

Pulse of a Nation

In a nutshell, the project Pulse of a Nation infers that the west coast of America is an altogether happier place than the east coast. The most consistently happy location was Hawaii, which one can perhaps understand; the most miserable were Mississippi and Alaska.

And the day most people are feeling grumpy is not Monday - sorry, Bob Geldof - but Thursday. That's probably because folk have had enough of the workplace, but there is still one day to go before the weekend.

Pulse of the Nation

"This isn't in any way a mature research project," Sune Lehmann of the Centre for Complex Network Research at Northeastern University warned me.

"But it's a cool project and promises all sorts of possibilities. And while it underlines prejudices people hold about the west coast and the east coast, it is impossible when you see the map not to make that interpretation. But again, we haven't anchored this data down."

On that issue of data, the scientists trawled through 300 million public tweets posted between September 2006 and August 2009. Those from outside the US or which didn't include their exact location were consigned to the dustbin.

The ones that remained were filtered into tweets with keywords that conveyed the mood of the twitterer. Sune and his colleagues used what he described to me as a psychological word-rating system: Affective Norms for English Words [859Kb PDF] (Anew).

A low-scoring word on Anew was rated as a negative while a high-scoring word got a positive rating. The 1,000-odd buzzwords included some that might be obvious including "happy", "sad", "glad", "love" and "joy" and some that are perhaps more obscure: "mildew", "elevator", "umbrella" and "pie".

Mr Lehmann says the inspiration for the project came from a map made by the New York Times during the Super Bowl to evaluate how people were feeling throughout the game and what the national conversation was about.

"It was a really neat visualisation that followed the whole game and let you learn a lot about America at the time. You could see who was rooting for which team and where they were, what commercials resonated, what the scores were and how people reacted," says Mr Lehmann.

"Then we had this idea that if we could connect that to a mood, it would be really interesting. We saw someone else do something simple and we thought: what would happen if we could scale it up?"

Again, Mr Lehmann stresses that "the results are not scientific" and "there are too many biases."

Since they only looked at tweets, Mr Lehmann says you could easily conclude that this is not really a true representation of America and its mood - because not everyone tweets.

He would guess that younger people rather than older use the service and that they are more likely to be pretty well educated with a higher income because tweeting means you need access to a computer or mobile phone.

Mr Lehmann says any further work would also need to include external factors: for example, looking at the weather because the belief is that people on the west coast are happier because of the climate. But then that mood might shift depending on the weather that day. Or if you live in San Francisco, the intensity of the fog might hold sway on your sense of well-being.

On a practical level, Mr Lehmann and his cohorts foresee applications including real-time reactions to a company's event or a product in different markets, responses to a President during a State of the Union address and the kind of thing that interests pollsters during election campaigns.

Pulse of the Nation

"The visualisations are amazing and I think it is absolutely fascinating to see the nation's mood vary in near-real time," said Johan Bollen of Indiana University in Bloomington, who was not involved in the work but who is one of several other researchers using Twitter as a tool to try to track the public mood.

He told the New Scientist that Twitter and similar services will spawn "sophisticated systems" for mood tracking.

There's an indication of the potential in Twitter's apparent success in predicting a film's box office success. Earlier this year, scientists Sitaram Asur and Bernardo A Huberman at HP Labs said that Twitter had a greater hit rate than the prediction markets in telling how well a movie will do in its opening weeks.

"The potential is there for this kind of work to be an incredibly powerful tool," says Mr Lehmann.

One of the stumbling blocks, he explains, is getting access to Twitter's fire-hose: those hundreds of millions of tweets circulating around the digital hemisphere.

Mr Lehmann cites Google's project Flu Trends as a parallel. In that case, the search giant was able to predict an outbreak by honing in on the search queries associated with flu and see where they were coming from. The data they collected consistently replicated that found by the Centres for Disease Control, but Google was two weeks ahead of the game.

Imagine if you could apply this kind of mood map to your boss's twitter feed before you head in to ask for a raise - or what about before you do a major pitch for a job or for venture capital funding?

The possibilities, Mr Lehmann says, are endless - if of course the data can be totally relied upon.

Comments

  • 1. At 11:53am on 23 Jul 2010, Fwd079 wrote:

    A big hmmm...

    Complain about this comment

  • 2. At 12:21pm on 23 Jul 2010, marcdraco wrote:

    For a fleeting moment there, I thought, wow.. a change from the usual.

    Then in less than 15 words: TWITTER. It's another article about the irrelevant social virus that the BBC is in love with.

    Enough already.

    Complain about this comment

  • 3. At 5:00pm on 23 Jul 2010, cping500 wrote:

    Will it would be a big hmmm if you hadn't looked at Theo Jones entry of the BBC Research and Development Blog about Zeitgeist where he looks at 'Twitter Sharing' patters and numbers (actuallly about music but could be about anything). So the BBC already has a sophiscated means of tracking mood , and the Trust has approved it! And its got a good name too! I hope it is patented and the BBC will make money for the the license fund.

    Can Mardraco is prove it is 'irrelevent'. 'Dangerous' would be more likely adjective.

    Complain about this comment

  • 4. At 9:47pm on 23 Jul 2010, Aidy wrote:

    Here's an interesting visual of the last 10 things covered in this blog

    Social media (Facebook) -> Social media (general) -> phones (general) -> phones (general) -> Apple -> Apple -> Apple -> Apple -> Social media (Facebook) -> Social media (Twitter)

    Complain about this comment

  • 5. At 09:18am on 24 Jul 2010, JunkkMale wrote:

    It's a story I am sure play well as editorial next to one of the ads accommodated by the new website design triumph.

    Not written on my iPhone as I don't have one, as presumed all do by a certain collection of media folk inside the M25

    Complain about this comment

  • 6. At 12:14pm on 24 Jul 2010, Visualisingdata wrote:

    Great to see experimentation like this but I don't think the finished visualisation actually provides a particularly useful means for people to explore the data or draw insight. Interesting that they were inspired by the NYT superbowl interactive Twitter Chatter - I've compiled some potential alternative design approaches here:
    http://www.visualisingdata.com/index.php/2010/07/twitter-visualisation-of-happiness/

    Complain about this comment

  • 7. At 3:06pm on 24 Jul 2010, Auqakuh wrote:

    Yay. An article on complete pseudoscience babble, AND it's about Twitter.

    Who cares? Really? This isn't even really scientific research. It's a meaningless load of twaddle. Twittertwaddle.

    Complain about this comment

  • 8. At 00:17am on 25 Jul 2010, marcdraco wrote:

    A correlation does not a causation make. Now write that out 100 times Ms. Shiels and then perhaps while you're doing that, note that most of us out here in BBC land know a lot more about the world of technology than you do - and that's not a good thing.

    Complain about this comment

  • 9. At 00:24am on 25 Jul 2010, marcdraco wrote:

    Well, CPing, in response to that question, I'm happy to oblige.

    Twitter is a sort of self-fulfilling prophecy. I'm sure it can accurately predict the mood of it's regular users (say 13-30 year olds); but that's a self-selected sample: not a random one.

    The signal/noise ratio is around 5% or less and it will be replaced, in time, by the next big thing.

    The BBC is in love with Twitter, of that I'm sure, but only because the people in BBC tech are typical Twitter users; blinded to the fact that most of us are not.

    They should all go work for a commercial station and then I won't complain.

    Complain about this comment

  • 10. At 04:17am on 25 Jul 2010, kary Fernandez wrote:

    Does the fact that you are in a good mood implies
    you are willing to give your employee a raise? I'm sure there is more that one factor involved (good mood may not be the only one) Before reading the reactions to this I thought this is indeed a neat idea or thought, before voicing those ideas one should get the work done... a lot things are easier said than done.

    Complain about this comment

  • 11. At 1:29pm on 25 Jul 2010, BluesBerry wrote:

    If you want to know what kind of mood America is in, check the Twitter feeds, which of course, will tell you one rather insignificant thing: how happy twitterers are feeling....maybe.
    If the scientists at Northwestern University and Harvard did not perceive the limitations of their own study, I have to wonder where they received training in validation.
    So, twitters are happier on the west coast than the east (maybe), and happiest of all in Hawaii (maybe). Sadder on Thursday vs, Monday (maybe).
    Sune Lehmann, Centre for Complex Network Research at Northeastern University: "But again, we haven't anchored this data down." Well, Sune, in my opinion, don't bother. It's not worth the effort; it's not "Complex", and I can't imagine anyone who would want this data, except a Government interested in
    - what words to use and where to use them in order to make happy the little tweets and
    - assuaging mood preliminary to revolution.
    Studies like this do not make me happy. They are a waste of money in times of austerity, and if the Government does not perceive it this way, then it's time to get nervous about what the Government is really up to.

    Complain about this comment

  • 12. At 10:44pm on 25 Jul 2010, AndrewMaus wrote:

    Among all the other variables not considered, the one that stand out to me is that even within one country the same words do not have a standard use. I've noticed that in the Mid-Atlantic [NJ, PA, NY] we curse much more often than my family and friends from the West Coast. It's not that we aren't as happy as them, it's just the way we talk. So if one were to count the number of 'f-bombs' in a statement to measure misery, for example, they would greatly misjudge a Philadelphian's excitement or unabashed joy.

    Complain about this comment

  • 13. At 09:17am on 26 Jul 2010, Aidy wrote:

    @AndrewMaus #12

    Tracking profanity will be the next bit of ground-breaking research (in case you don't know...no-one has ever done a keyword search on Twitter feeds to "gauge" something...this is all brand new and exciting) where they track the literary intelligence of people state by state :)

    Complain about this comment

  • 14. At 09:27am on 26 Jul 2010, John_from_Hendon wrote:

    #1. Fwd079 wrote:

    "A big hmmm..."

    Are you suggesting that this 'research' stinks? If so, I agree!

    Complain about this comment

  • 15. At 09:33am on 27 Jul 2010, jayshree wrote:

    In your second sentence you've written that this is a collaboration between scientists from Northwestern and Harvard. It's Northeastern and Harvard, not Northwestern. Just so you know!

    Complain about this comment

  • 16. At 9:08pm on 28 Jul 2010, gstewart75hotmailcom wrote:

    Very Unique project! I liked how you showed the pictures, graphs, and charts. I found it interesting that the day most people are grumpy is Thursday not Monday. The mood chart definitely showed large increases on Friday, Saturday, and Sunday in happiness while drastically decreasing to a bad mood on Thursday. On the second chart I was surprised how someone can go from being so happy one minute like at six o’ clock and then be so low and NOT happy at 12:30 or so. To see with the green and blue how the west coast and east coast can be so alike yet so different on the chart was very cool. Their moods are so different in one area but completely opposite in another. I thought it very fascinating and I never would have guessed these facts.

    Julia, 11

    Complain about this comment

  • 17. At 9:18pm on 28 Jul 2010, gstewart75hotmailcom wrote:

    This comment was removed because the moderators found it broke the House Rules.

  • 18. At 4:58pm on 29 Jul 2010, Philly-Mom wrote:

    This is hysterical! I love it!
    I just tuned in briefly from vacation and found this poignant and accurate. I've lived in the North East US as well as the South West US and find this altogether true.
    The relaxed attitudes of the SW are totally different than the go-getter punch-clock mindsets of the NE -- and the Old SE may be relaxed, but is often stubbornly up-tight. (This seems to be changing, though.)
    There are many MANY reasons for all of this: cultural habits, climate, industry standards, work requirements, diet/sleep paterns... blah blah blah.
    Suffice it to say that I love the central east coast and have settled in because it's not too slow & not too fast for my personal preference. Oh... and there's lots of camping, hiking, history & art, too. A pleasant mix of fun and work, IMHO.

    Complain about this comment

  • 19. At 5:02pm on 29 Jul 2010, Philly-Mom wrote:

    This comment was removed because the moderators found it broke the House Rules.

  • 20. At 8:49pm on 01 Aug 2010, webbod wrote:

    You've missed the point of the exercise - it's the techniques they used to extract the data that are interesting, not the fact that the mood swings of a bunch of self-important micro-bloggers can be shoe-horned into a map representing the attitudes of a continent.

    Maggie - these blogs are getting worse, you've should consider writing for CNet or ZDNet; it's either social-fruity-loopery or apples with you - I'm done complaining to the BBC, I'm sending a link to "New Scientist".


    Complain about this comment

  • 21. At 11:06pm on 02 Aug 2010, marcdraco wrote:

    Let me guess, Webbod (@20) your complaints have been met with the dubious sound of crickets.

    Complain about this comment

  • 22. At 11:45am on 04 Aug 2010, ajay wrote:

    do this in england and it'll be red all the time! why are people so moody in england???

    Complain about this comment

  • 23. At 12:37pm on 04 Aug 2010, Aidy wrote:

    @webbod #20

    > it's the techniques they used to extract the data that are interesting

    That's the thing...they're not. Trust me, there have been many many many identical uses of the twitter api.

    Complain about this comment

  • 24. At 11:47pm on 04 Aug 2010, marcdraco wrote:

    And even MORE coverage of Twitter ALL over the BBC... this is bizarre, you'd think the BBC had shares in it.

    Complain about this comment

  • 25. At 10:18am on 05 Aug 2010, dukeofearl wrote:

    We've had the new RIM Torch and some more bad news about Apple (PDF viruses) and not a word on the Maggie blog - is bad news about Apple and good news about competitors not worth a comment??

    Complain about this comment

  • 26. At 2:48pm on 05 Aug 2010, marcdraco wrote:

    Don't complain here - write to the BBC complaints and then to the Trust (if they let you get that far).

    If enough of us raise this issue it will have to be addressed.

    Complain about this comment

View these comments in RSS

BBC iD

Sign in

BBC navigation

BBC © 2014 The BBC is not responsible for the content of external sites. Read more.

This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.