How do game companies share massive files?

Battlefield 4 Developing Battlefield 4 proved to be a technical challenge for its publisher Electronics Arts

Electronic Arts' multi-player shooter "Battlefield 4" exploded onto the computer games scene earlier this month, giving players a realistic taste of military combat in the 21st Century.

Technology of Business

But the logistics needed to develop a complex game like Battlefield 4 were certainly not child's play.

In particular, handling the massive amounts of data that makes up the game - huge graphics files, sound clips, and the game code itself - proved to be a severe technical challenge for the California-based games developer and publisher.

That's because some of the complete game files were as large as 50GB, and future games with more advanced graphics for the new Xbox One console and Sony's PlayStation 4 are likely to be even bigger.

During development these files have to be distributed around the world, explains Steve Scivally, EA's technical director.

Start Quote

The game builds got larger and took longer. There was only one choice: transfer less content”

End Quote Steve Scivally EA's technical director

"EA is made up of a large group of development sites and a few dedicated QA (quality assurance) test centres. New game builds need to be transferred to the test centres as quickly as possible to locate defects and improve quality."

The problem was that sending a 50GB game file between sites over EA's own network often took hours - or even days.

In some cases it would have taken less time to move EA's staff to the data than to move the data to the sites where those people worked, and all this had the effect of slowing down project cycles and the overall pace of the game's development.

More selective

Solving this problem involved abandoning the use of its own computer network for large file distribution, relying instead on the internet to send data to its sites around the world from a central data repository in the cloud.

It also involved being more selective in which data was actually moved around the world, Mr Scivally explains: "The game builds got larger and took longer. There was only one choice: transfer less content."

Battlefield 4 Developers will produce different versions of their games for different games consoles

To do this EA installed devices called cloud storage controllers, supplied by a California-based company called Panzura, in each of its sites.

These cloud storage controllers download and store copies of "master" game files which live in the cloud. The clever part is what happens whenever a change is made to a file at any site anywhere in the world.

When this happens the cloud controller spots that a change has been made to its copy of the file, and transmits just this change, or "delta", back to the cloud so that the master copy can be updated.

This delta - which is often less than 5% of the entire file - is then transmitted from the cloud to all the controllers at the other sites, and their local copies are updated.

Multiple game versions
Alasdair Monk Alasdair Monk says using a system like Dropbox would be too expensive for his firm

To minimise the amount of data that needs to be sent around the world two other technologies are also used: compression - which reduced the size of a file - and something called deduplication.

Deduplication involves spotting chunks of data that are identical to other chunks of data stored at the same location, and then storing only a single copy.

It is useful for EA because the company produces versions of its games for different game platforms such as Sony's PlayStation and Microsoft's Xbox, and because each version is almost, but not quite, identical. Deduplication means that one batch of data can be transmitted to remote sites to update multiple versions of the game.

Using this system EA's developers and testers around the world can access the latest game files in a matter of minutes rather than hours, helping to ensure that Battlefield 4 was ready for release on schedule, Mr Scivally says.

Start Quote

The understanding of BitTorrent has been maturing”

End Quote Christian Averill BitTorrent director of communications

EA's story highlights a widespread problem many companies and individuals are encountering. Files often need to be shared, but increasingly the size of these files are becoming too large share using simple methods like email.

It's the reason that cloud-based services such as Dropbox and others like it have become so popular.

Files encrypted and reassembled

But for small companies that need to share large files frequently these types of services can be too expensive, and for that reason some are turning to more innovative solutions.

"I work with 50-200MB design files and I am constantly sharing stuff with our team of 15 developers," says Alasdair Monk, design director at London-based payments start up GoCardless. "Using Dropbox to do this would be far too expensive."

BitTorrent Sync graphic BitTorrent says its Sync tool is ideal for large uncompressed files

Instead he turned to BitTorrent - a system for sharing large files using peer-to-peer technology that's usually associated with pirating films, music and video games.

BitTorrent offers a free product called BitTorrent Sync, which sends information between users' PCs and eliminates the need to upload data to a third party's computer servers. Mr Monk uses it to make large design files stored on his computer available to GoCardless's developers very rapidly, even if they are working on a laptop from home.

It works quickly because files are split up into encrypted chunks and sent out to multiple BitTorrent Sync users within the company.

When someone else needs one of Mr Monk's files, it is reassembled from chunks stored on some of these users' computers - whichever ones can supply the chunks most quickly. "Even when developers or I am are out on the road, the speed that they can download one of my large files is phenomenal," he says.

Reliability of cloud storage?

Christian Averill, BitTorrent's director of communications, says over one million people are using BitTorrent Sync, and that the association that many people have between BitTorrent and film piracy seems not to have been an issue.

"The understanding of BitTorrent has been maturing and most people know that we have no affiliation with such activity," he says.

He adds that a system like BitTorrent Sync means that there is no need to store data in the cloud, which can be a benefit for companies in some industries that are subject to strict regulatory requirements

There's also the fact that cloud storage is only as reliable as the companies that operate it - something that EA learned the hard way.

Its master game files used to be held in cloud storage supplied by a company called Nirvanix, but in September the company suddenly announced that it would be closing down at the end of the month.

That left EA - along with all Nirvanix's other customers - scrambling to retrieve its cloud data and move it to another cloud before the Nirvanix storage was switched off for good.

More on This Story

The BBC is not responsible for the content of external Internet sites

More Business stories

RSS

BBC Business Live

  1.  
    CO-OP BANK RESULTS 08:08: Radio 5 live

    Chris Wheeler, a banking analyst at Mediobanca, is on 5 live talking about the Co-op Bank results. He says "there are signs of progress" but still "problems they have to deal with," including a drop in deposits. A reduction in money set aside for bad debts "flattered" the profit figures, he said.

     
  2.  
    BANK OF AMERICA 08:01: Radio 5 live

    Bank of America can easily afford the record $16.7bn (£10bn) settlement with US authorities for misleading investors about the quality of loans it sold, US investment expert George Conboy tells Radio 5 live. He says the bank is likely to make $10-15bn over the next 12 months.

     
  3.  
    CO-OP BANK RESULTS 07:54:

    An offer of shares to the public via a stock market listing is "logistically unlikely" before the end of 2014, the Co-op Bank says. It is setting up a committee to work out when it will do that. The Prudential Regulation Authority "has indicated it would be concerned if an IPO were to distract focus from the primary goal of delivering the Bank's Turnaround Plan," it said.

     
  4.  
    ARGENTINA DEBT 07:46: BBC Radio 4

    Argentina risks becoming an "international financial pariah" again over its latest debt row, James Bevan, chief investment officer at CCLA Investment Management tells the Today programme. "The problem is that investors won't trust them, [so] they won't provide money," he says.

     
  5.  
     
  6.  
    LSE RIGHTS ISSUE 07:35:
    LSE sign

    The London Stock Exchange has said it plans to raise £938m through a rights issue. The money is to be used to part-fund the acquisition of Frank Russell Company. The LSE announced in June it was buying the US asset manager for $2.7bn (£1.6bn).

     
  7.  
    PPI SPENDING 07:25: Radio 5 live

    Consumers have about £23bn of payment protection insurance redress from banks, says James Bevan, Chief Investment Officer at CCLA Investment Management on Wake Up to Money. When that dries up, it may have an effect on spending, he says.

     
  8.  
    EDF ENERGY PENALTY 07:15:

    Ofgem said it found that between May 2011 and January 2012, "EDF Energy did not have appropriate procedures in place to properly receive, record and process all customers' complaints in accordance with complaints handling rules". However, it adds EDF took action quickly to rectify the problems. It also says EDF "has acknowledged that their customers were caused significant disruption" and has apologised.

     
  9.  
    CO-OP BANK RESULTS 07:09:
    bank

    Co-operative Bank has reported a loss of £75.8m for the six months to the end of June. That compares with a loss of £845m the year earlier. The bank says it has cut staff numbers by 21% from a year earlier to 5,860.

     
  10.  
    EDF ENERGY PENALTY Breaking News
    EDF bill

    EDF Energy is to pay £3m after an investigation by energy regulator Ofgem found that the company breached complaint-handling rules. The company will pay the money "to benefit vulnerable customers", Ofgem says.

     
  11.  
    BANK OF AMERICA 06:55: BBC Radio 4

    Chris Wheeler, an analyst at Mediobanca, has been discussing Bank of America's massive $16.7bn settlement with US authorities. He told the Today programme that while banks were at fault, investors were also "in a hurry" at the time, and failed to carry out proper credit analysis of what they were buying.

     
  12.  
    EUROZONE 06:44: Radio 5 live
    ECB headquarters

    James Bevan is chief investment officer at CCLA Investment Management and he's on Wake Up to Money talking about the eurozone's performance. "Interest rates are so low" in some of the countries that when you factor in the risk of investing the rates are negative, he says. Sustainable growth in the UK hasn't yet been seen either, he says. with house price growth and PPI repayments from banks driving consumer spending.

     
  13.  
    GAP IN INDIA 06:30:
    Gap logo

    US clothing store Gap is bringing its brand to India, aiming to open 40 outlets. It will launch its first two stores there early next year in Mumbai and the capital Delhi - India's biggest and busiest cities.

     
  14.  
    BANK OF AMERICA 06:20: Radio 5 live
    Bank of America

    Chris Orndorff, a fund manager at Western Asset Management is on Wake Up to Money talking about Bank of America's record $16.7bn settlement. The penalty dates back to the behaviour of a company it purchased, Countrywide. "The timing of the Countrywide purchase has to be one of the worst in history," he says. The fine wipes out the cumulative net income of the bank for the last three years. The bank is very safe though, he says.

     
  15.  
    ARGENTINA DEBT 06:06:

    Argentina's plan to exit its debt default by asking investors holding defaulted bonds to swap them for new locally issued debt has been ruled "illegal" by a US court. New York Judge Thomas Griesa said the plan was "lawless". Argentina was trying to get around an earlier court ruling banning it from paying interest to investors who had accepted restructured bonds.

     
  16.  
    OIL PRICE 06:00: Radio 5 live
    Oil barrels

    After rising sharply in June, oil prices have now slipped back to 14-month lows. Amrita Sen from Energy Aspects tells Wake Up to Money. She says that "the initial price rise was on the back of the fears should any disruptions come at peak demand". However, despite geopolitical risks, supply has remained undisrupted while demand has fallen.

     
  17.  
    06:00: Nick Edser, Business reporter

    Good morning. Get in touch with us via email: bizlivepage@bbc.co.uk or on twitter @BBCBusiness.

     
  18.  
    06:00: Howard Mustoe, Business Reporter

    Hello! We'll be bringing you all the latest business news, data and analysis. Stay with us.

     

Features

BBC © 2014 The BBC is not responsible for the content of external sites. Read more.

This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.