How do game companies share massive files?

Battlefield 4 Developing Battlefield 4 proved to be a technical challenge for its publisher Electronics Arts

Electronic Arts' multi-player shooter "Battlefield 4" exploded onto the computer games scene earlier this month, giving players a realistic taste of military combat in the 21st Century.

Technology of Business

But the logistics needed to develop a complex game like Battlefield 4 were certainly not child's play.

In particular, handling the massive amounts of data that makes up the game - huge graphics files, sound clips, and the game code itself - proved to be a severe technical challenge for the California-based games developer and publisher.

That's because some of the complete game files were as large as 50GB, and future games with more advanced graphics for the new Xbox One console and Sony's PlayStation 4 are likely to be even bigger.

During development these files have to be distributed around the world, explains Steve Scivally, EA's technical director.

Start Quote

The game builds got larger and took longer. There was only one choice: transfer less content”

End Quote Steve Scivally EA's technical director

"EA is made up of a large group of development sites and a few dedicated QA (quality assurance) test centres. New game builds need to be transferred to the test centres as quickly as possible to locate defects and improve quality."

The problem was that sending a 50GB game file between sites over EA's own network often took hours - or even days.

In some cases it would have taken less time to move EA's staff to the data than to move the data to the sites where those people worked, and all this had the effect of slowing down project cycles and the overall pace of the game's development.

More selective

Solving this problem involved abandoning the use of its own computer network for large file distribution, relying instead on the internet to send data to its sites around the world from a central data repository in the cloud.

It also involved being more selective in which data was actually moved around the world, Mr Scivally explains: "The game builds got larger and took longer. There was only one choice: transfer less content."

Battlefield 4 Developers will produce different versions of their games for different games consoles

To do this EA installed devices called cloud storage controllers, supplied by a California-based company called Panzura, in each of its sites.

These cloud storage controllers download and store copies of "master" game files which live in the cloud. The clever part is what happens whenever a change is made to a file at any site anywhere in the world.

When this happens the cloud controller spots that a change has been made to its copy of the file, and transmits just this change, or "delta", back to the cloud so that the master copy can be updated.

This delta - which is often less than 5% of the entire file - is then transmitted from the cloud to all the controllers at the other sites, and their local copies are updated.

Multiple game versions
Alasdair Monk Alasdair Monk says using a system like Dropbox would be too expensive for his firm

To minimise the amount of data that needs to be sent around the world two other technologies are also used: compression - which reduced the size of a file - and something called deduplication.

Deduplication involves spotting chunks of data that are identical to other chunks of data stored at the same location, and then storing only a single copy.

It is useful for EA because the company produces versions of its games for different game platforms such as Sony's PlayStation and Microsoft's Xbox, and because each version is almost, but not quite, identical. Deduplication means that one batch of data can be transmitted to remote sites to update multiple versions of the game.

Using this system EA's developers and testers around the world can access the latest game files in a matter of minutes rather than hours, helping to ensure that Battlefield 4 was ready for release on schedule, Mr Scivally says.

Start Quote

The understanding of BitTorrent has been maturing”

End Quote Christian Averill BitTorrent director of communications

EA's story highlights a widespread problem many companies and individuals are encountering. Files often need to be shared, but increasingly the size of these files are becoming too large share using simple methods like email.

It's the reason that cloud-based services such as Dropbox and others like it have become so popular.

Files encrypted and reassembled

But for small companies that need to share large files frequently these types of services can be too expensive, and for that reason some are turning to more innovative solutions.

"I work with 50-200MB design files and I am constantly sharing stuff with our team of 15 developers," says Alasdair Monk, design director at London-based payments start up GoCardless. "Using Dropbox to do this would be far too expensive."

BitTorrent Sync graphic BitTorrent says its Sync tool is ideal for large uncompressed files

Instead he turned to BitTorrent - a system for sharing large files using peer-to-peer technology that's usually associated with pirating films, music and video games.

BitTorrent offers a free product called BitTorrent Sync, which sends information between users' PCs and eliminates the need to upload data to a third party's computer servers. Mr Monk uses it to make large design files stored on his computer available to GoCardless's developers very rapidly, even if they are working on a laptop from home.

It works quickly because files are split up into encrypted chunks and sent out to multiple BitTorrent Sync users within the company.

When someone else needs one of Mr Monk's files, it is reassembled from chunks stored on some of these users' computers - whichever ones can supply the chunks most quickly. "Even when developers or I am are out on the road, the speed that they can download one of my large files is phenomenal," he says.

Reliability of cloud storage?

Christian Averill, BitTorrent's director of communications, says over one million people are using BitTorrent Sync, and that the association that many people have between BitTorrent and film piracy seems not to have been an issue.

"The understanding of BitTorrent has been maturing and most people know that we have no affiliation with such activity," he says.

He adds that a system like BitTorrent Sync means that there is no need to store data in the cloud, which can be a benefit for companies in some industries that are subject to strict regulatory requirements

There's also the fact that cloud storage is only as reliable as the companies that operate it - something that EA learned the hard way.

Its master game files used to be held in cloud storage supplied by a company called Nirvanix, but in September the company suddenly announced that it would be closing down at the end of the month.

That left EA - along with all Nirvanix's other customers - scrambling to retrieve its cloud data and move it to another cloud before the Nirvanix storage was switched off for good.

More on This Story

The BBC is not responsible for the content of external Internet sites

More Business stories

RSS

BBC Business Live

  1.  
    MARKET UPDATE 11:36:

    The main movers on the share market include Mothercare, shares down 6% on news a potential bid has gone away. Ryanair is up 4% after a good earnings report. Otherwise it's a typical high summer market atmosphere - overall torpid.

    • The 100 share index is 6.31 higher at 6797.86
    • The Cac in Paris is 11.54 higher at 4342.09
    • Germany's Dax is down 17,73 at 9626.28
     
  2.  
    YUKOS AWARD 11:25:

    A spokesman for GML group of Yukos shareholders said the $50bn award was the largest ever by an arbitration court. Yukos was disbanded after filing for bankruptcy. Its boss Mikhail Khodorkovsky was arrested in 2003 and spent ten years in jail after being convicted of fraud and tax evasion.

     
  3.  
    FRACKING LICENCES 11:17:

    It won't be easy for a company to use these new fracking licences, warns Robert Gatliff from the British Geological Survey: "It doesn't have a carte blanche to drill everywhere. It still has to go through all our regulatory requirements which are in place, which look at the environmental issues and they have to get local planning permission and things like that."

     
  4.  
    RECKITT BENCKISER 11:10:
    Nurofen packets

    "Keeping consumers delighted," is how Reckitt Bensicker introduces its brands on its website. "RB brands are household favourites all over the world, with products that delight as they make life's daily routine easier and better," it says. We have to concur. It also makes Nurofen, an especial delight for many on a Monday morning. (The BBC adds: other painkillers are available.)

     
  5.  
    BANKRUPTCY HELP 10:56:
    Cut up cards

    More help is needed for people who are too poor to go bankrupt, according to a report out today. The Centre for Social Justice says 300,000 people in the UK can't afford the £525 (to the official receiver) and £180 to the (courts) to be legally declared bankrupt. Andrew Tate, from R3 - the insolvency trade body - thinks people should be allowed to pay the fees off over time.

     
  6.  
    YUKOS AWARD 10:40: World Service

    The Court of Arbitration in The Hague said Russia must pay the compensation after finding it forced the company into bankruptcy. The claimant's lawyer, Emmanuel Gaillard, said there would be no opportunity for Russia to contest the ruling: "As to appeal, there is no appeal... The tribunal has listened to both parties... the Russian Federation had ample opportunity to be heard in this case, the judgement is there. After 10 years of battle, the tribunal says they violated international law."

     
  7.  
    HOUSE PRICES 10:24:
    A grpah showing house price growth in the last year

    Average house prices in London were only 0.1% higher in June compared with a month earlier. But that still puts them 16.4% higher than the same month a year ago. The average price of property in the capital in June was £437,608. That compares with £172,011 in the rest of England and Wales.

     
  8.  
    HOUSE PRICES 10:11:

    In fact, average house prices in seven out of 10 regions in England and Wales fell - yup fell- in June compared with the previous month. Only London, the South East and the West Midlands saw an increase in average property values. The West Midlands was far and away the star performer in June with average house prices up 1.9%.

     
  9.  
    HOUSE PRICES 09:58:

    On an annual basis, the West Midlands saw a 4.7% increase in average house prices, while the South East saw house prices jump 7.8%. At the other end of the scale, house prices in the North East are up just 0.8% since June last year.

     
  10.  
    HEADLINES
  11.  
    HOUSE PRICES 09:48:
    A street of colourfully painted houses in London

    Average house prices in the UK in June were unchanged - yes that's right unchanged - says the Land Registry. It is the only one whose index measures the actual sale price. It's not the first house price data to show a levelling off and one or two have even suggested slight falls. One reason could be April's tougher mortgage affordability rules but that's unlikely to be the only explanation.

     
  12.  
    TRINITY MIRROR 09:35:
    Trinity Mirror share price

    Loyal shareholders in newspaper group Trinity Mirror will be rewarded for their patience. The company said its full-year results would be better than expected and it hoped to pay a dividend for the first time since 2008. First half profits are down 2% at £48.2m.

     
  13.  
    09:26: Breaking News

    The arbitration court in The Hague tells Russia to pay Yukos shareholders $50bn in compensation, says main shareholder GML. Russia forced Yukos into bankruptcy and sold its assets to state-owned businesses for political purposes, says GML's lawyer.

     
  14.  
    RAIL FRANCHISES 09:24:
    Commuters at Kings Cross Railway Station

    Here's an interesting tip off from the Guardian about Labour's election pledge to let cooperatives and mutually-owned companies run Britain's rail services. The party's pledge to give public sector bidders the same rights as private companies to bid for rail franchises will be outlined on a visit to Glasgow later today. Labour will say that employees and passengers could take a far greater role in running the railways.

     
  15.  
    FRACKING LICENCES 09:14: BBC Radio 4
    Balcombe fracking

    More from Mr Hancoc, who tells the Today programme that "of course" there is some local opposition to fracking but the public in general understands the need for energy security and largely agrees with fracking. He couldn't name a local community pleased about the prospect of fracking, but points out there is a £100,000 grant for local communities and says they will benefit from the proceeds of fracking as well.

     
  16.  
    YUKOS AWARD 09:04:

    More from Reuters on compensation for Yukos shareholders. There's another case going on in the European Court of Human Rights (ECHR) in Strasbourg on behalf of all Yukos shareholders. That is expected to announce its decision on Thursday. They're claiming Yukos was unlawfully deprived of its possessions by the imposition of bogus taxes and a sham auction of its main asset.

     
  17.  
    MOTHERCARE FUTURE 08:48:
    Mothercare share graph

    Mothercare shares have fallen sharply - 11% - after the company released a statement saying it had had no contact with potential US bidder Destination Maternity since early June. It said firmly it wants an independent future. Shareholders can't be said to be universally delighted about that prospect.

     
  18.  
    MARKET UPDATE 08:36:

    European markets have started the day steadily enough. There was some positive data from China with industrial profits growing at a stronger pace in the first half of 2014 than a year earlier. The biggest rise on the FTSE 100 so far is Reckitt Benckiser - up nearly 3% to 5220p - following its positive trading update.

    • The FTSE 100 is 0.17% higher at 6802.99
    • Germany's Dax is lower by 0.05% at 9639.04
    • France's Cac-40 is 0.30% higher at 4343.41
     
  19.  
    FRACKING LICENCES 08:29: BBC Radio 4

    Energy minister Matthew Hancock tells the Today programme there is "potentially" quite a bit of shale gas under the ground. That would be good news for our energy security, he adds. Areas need to be explored to see if there is shale gas in the ground first. Companies currently exploring for shale gas will then automatically be allowed to exploit it.

     
  20.  
    HEADLINES
  21.  
    RYANAIR TRADING 08:24:

    Ryanair shares have jumped 5% in the wake of its statement earlier this morning, predicting higher profits.

     
  22.  
    MARKET UPDATE 08:10:

    Asian stock markets are hovering near three-year highs, with China taking the lead after data showed a robust rise in profits earned by industrial firms. Japan's Nikkei 225 rose 0.46% to 15529.4, while Hong Kong's Hang Seng is up 1% to 24,456.5.

     
  23.  
    YUKOS AWARD 08:04:
    Kordorkovsky

    Yukos was run by what was then Russia's richest man, Mikhail Khodorkovsky. The $50bn (£29.4bn) the Hague court is said by Reuters to be awarding the shareholders is half the $100bn they were asking for. The claim is from subsidiaries of Gibraltar-based Group Menatep. Menatep now exists as holding company GML and Khodorkovskyis no longer a shareholder either entity.

     
  24.  
    ARGENTINA DEBT 07:56: BBC Radio 4

    One more from Mr Caicedo on Argentina's debt woes. He says default will be very damaging for the country because it has been trying to normalise its relationships with capital markets and that will go out the window. He adds that will also mean Argentine provinces won't be able to go to the markets to raise cash by issuing bonds of their own - and that will start to feed into their ability to pay wages to government employees, among other things.

     
  25.  
    YUKOS AWARD 07:45:
    Logo

    Reuters has a report - citing unnamed sources - that the Hague's arbitration court has ruled in favour of a group of shareholders in former oil giant Yukos against Russia, awarding compensation of around $50bn. That's half the amount originally claimed.

     
  26.  
    RECKITT BENCKISER 07:37:
    A shelf full of Reckitt Benckiser products

    Reckitt Benckiser - which does a host of branded goods from condoms to cleaning fluid Cillit Bang - reports a 4% rise in sales for the past six months. Profits rose 16% to £1bn. The company has also decided to spin off its Suboxone pharmaceuticals business in the next 12 months, ending months of speculation over the shrinking division's future.

     
  27.  
    OVERPRICED HEALTHCARE 07:36: Radio 5 live

    Private healthcare is 15% overpriced, estimates Bupa. The healthcare group itself has been criticised in the past by regulators for lack of clarity around its policies. Bupa boss Damien Marmion says they're working on that: "We have addressed it substantially, but will continue [to deal with the matter]."

     
  28.  
    FRACKING LICENCES 07:27: Radio 5 live

    More on Fracking. David Hunter tells Wake up to Money the industry will obviously will need to regulated properly. "It is right that we are clear on how tight regulation should be," he says, but "it's also clear we should be spending more time assessing the potential of fracking" given energy security concerns, he adds.

     
  29.  
    MOTHERCARE FUTURE 07:20:
    Pushchairs on sale at a Mothercare store

    Mothercare has responded to the news US rival Destination Maternity has gone off the idea of bidding for it with a corporate shrug. A statement says it "notes" the offer withdrawal and also "notes" it has had no contact with Destination Maternity for about six weeks. It says under new boss Mark Newton-Jones it is now "fully focused" on its turnaround plan.

     
  30.  
    OVERPRICED HEALTHCARE 07:13: Radio 5 live
    Bupa sign

    Private healthcare is overpriced, says ... private healthcare provider Bupa. Damien Marmion, its managing director of UK Insurance. He says the whole industry is to blame, especially the price charged by hospitals. He says prices need to go down by about 15%.

     
  31.  
    RYANAIR TRADING 07:02:
    Ryanair plane landing

    Irish airline Ryanair reports better than expected first quarter earnings. Pre-tax profits rose to 223.6m euros (I£176.8m) in the three months to 30 June up from 88.5m euros a year earlier. Full-year profit guidance is up, too, to between 620m euros and 650m euros for the full year.

     
  32.  
    FRACKING LICENCES 06:49: Radio 5 live

    Fracking will not necessarily lead to lower energy prices, David Hunter from Schneider Electric tells Wake Up to Money. It is important that the UK assess the potential of fracking, especially given Europe's currently fractious [a-hem] relationship with Russia. It's more about energy security than lower prices, Mr Hunter says.

     
  33.  
    GSK BREAK-UP? 06:40:
    Horlicks jars

    The Financial Times goes with the boss of drugs giant GSK is considering breaking up the company, should its consumer healthcare division be worth more as a standalone company. And yes, that division owns Horlicks, as well as mega-brands like Aquafresh and Corsodyl.

     
  34.  
    ARGENTINA DEBT 06:32: BBC Radio 4

    More on Argentina's debt woes. It is risking falling into default for the second time in 12 years, says Carlos Caicedo, senior principal analyst on country risk at IHS Global Insight. In fact, this looks like the most likely outcome after the Argentine economic minister failed to arrive in New York for talks on Friday. Leading Hedge Fund NML Capital believes Argentina will definitely default on its loans, he adds, and the government's refusal to meet and talk with its creditors isn't helping matters.

     
  35.  
    ARGENTINA DEBT 06:23: Radio 5 live

    It is more than 10 years since Argentina announced it couldn't pay back its debts. Most bondholders accepted a deal to get back a small percentage in exchange for writing off those debt. But some bonds are owned by US hedge funds who want full repayment. Argentina says no, but there's a court deadline on Wednesday. Deborah Zandstra, a specialist in debt restructuring at Clifford Chance, tells Wake Up to Money the move will have wider repercussions: "Essentially Argentina is being put into a position of defaulting on its bonds. That will trigger cross defaults on its other debts."

     
  36.  
    FRACKING LICENCES 06:09: BBC World News
    Brenda Kelly

    Brenda Kelly from IG Index is reviewing the newspapers on World News. "It is interesting to compare the Guardian's coverage with the Telegraph. The Guardian says there will be drilling but the Telegraph says it won't. There are a huge amount of environmental issues - the amount of water used, the risk of small earthquakes, for example. All energy firms will have to provide an environmental statement if they are going to drill near sites of natural beauty."

     
  37.  
    FRACKING LICENCES 06:03: Radio 5 live
    Barton Moss Results from test drilling for shale gas at Barton Moss, Salford are being analysed

    Energy companies are being invited to bid for licences to extract oil and gas from large areas of Britain, using the controversial process known as fracking. The technique will be all but banned from national parks and other environmentally sensitive areas. Mike Bradshaw, professor of Global Energy at Warwick Business School, tells Radio 5 live people are concerned: "We've got very limited experience of fracking in the UK. So we don't know the answer to many of the questions that people have about what fracking will look like under UK and EU regulations which are very different from those that exist in the United States."

     
  38.  
    06:01: Matthew West Business Reporter

    Morning folks, as ever you can get in touch with us by email at bizlivepage@bbc.co.uk and on twitter @bbcbusiness.

     
  39.  
    06:00: Rebecca Marston Business reporter, BBC News

    Monday morning. Here we are. Fracking is under the spotlight as companies start to bid for licences to drill for shale gas. Stay with us for the pick of the business news from the BBC and elsewhere.

     

Features

BBC © 2014 The BBC is not responsible for the content of external sites. Read more.

This page is best viewed in an up-to-date web browser with style sheets (CSS) enabled. While you will be able to view the content of this page in your current browser, you will not be able to get the full visual experience. Please consider upgrading your browser software or enabling style sheets (CSS) if you are able to do so.