BBC.co.uk

Talk about Newsnight

Newsnight

Lancet Iraq survey methodology under fire

  • Paul Mason
  • 19 Oct 06, 04:53 PM

Over the last couple of days two new themes have emerged in the debate over the Lancet/Johns Hopkins University report which estimated an excess 601,000 violent deaths in Iraq as a result of the invasion...

The first, here, maintains that there was a flaw in the methodology because the number of clusters sampled was too low. The Lancet study sampled from 47 clusters of families; Steven Moore, a political consultant writes:

"Appendix A of the Johns Hopkins survey, for example, cites several other studies of mortality in war zones, and uses the citations to validate the group's use of cluster sampling. One study is by the International Rescue Committee in the Democratic Republic of Congo, which used 750 cluster points. Harvard's School of Public Health, in a 1992 survey of Iraq, used 271 cluster points. Another study in Kosovo cites the use of 50 cluster points, but this was for a population of just 1.6 million, compared to Iraq's 27 million."

A second criticism is set to emerge in tomorrow's Science magazine where academics from Royal Holloway and Oxford will say that the study suffers from "main street bias" - in that sampling the families near main thoroughfares leads to an unnaturally high number of casualties because - with car bombs, driveby shootings etc - that is where they occur.

Both these critiques will raise questions not just for the researchers but for the peer review process leading to publication. We will try to keep you informed as to the answers, and the debate as it unfolds, here on the Newsnight blog.

Comments  Post your comment

As a popular political blogger, I know you’ll be interested in learning more about our recent study on American attitudes toward current foreign policy and the nation’s place in the world. Please read on for more information! Feel free to contact us or to blog away on our intriguing findings.

Here at Public Agenda, we’ve created a new tool to track Americans’ opinions on foreign policy issues, providing a basis for political commentary. Similar to the Consumer Confidence Index, the Foreign Policy Anxiety Indicator provides policy makers, journalists and ordinary citizens with the public's overall comfort level with America's
place in the world and current foreign policy.

An essential tool updated twice a year, the Indicator will consistently provide much-needed information on the public’s perception of more than two dozen aspects of international relations.

In a world strewn with violence and highly-charged international issues, Americans are broadly uneasy about U.S. foreign policy. The September 2006 shows the Foreign Policy Anxiety Indicator at 130 on a scale of 0 to 200, where 0 is the most confident, 200 the most anxious and 100 neutral.

Eight in 10 Americans feel the world is becoming a more dangerous place for Americans, yet they're also skeptical about most of the possible solutions, such as creating democracies or global development. Only improved intelligence gathering and energy independence have substantial support, with energy firmly established as a national security problem
for the public.

In fact, the public lacks confidence in many of the measures being taken to ensure America’s security. Less than 33% of Americans give the U.S. government an “A” or a “B” grade for its execution of the following foreign policy issues: reaching goals in Iraq and Afghanistan, maintaining good relationships with Muslim countries and protecting U.S. borders from illegal immigration. And these are just a few of the findings of the survey.

These are some of the other startling findings:

- 83 percent say they are worried about the way things are going for the United States in world affairs (35 percent worry "a lot", with an additional 48 percent saying they worry "somewhat.")

- 79 percent say the world is becoming more dangerous for the United States and the American people

- 69 percent say the United States is doing a fair or poor job in creating a more peaceful and prosperous world

- 64 percent say the rest of the world sees the United States negatively

- 58 percent say U.S. relations with the rest of the world are on the wrong track

Want to learn more? Go to http://www.publicagenda.org/foreignpolicy/index.cfm to download the report.

Public Agenda is a nonprofit, nonpartisan group devoted to public opinion and public policy. The confidence in U.S. Foreign Policy Index is developed in cooperation with Foreign Affairs with support from the Hewlett and Ford foundations.

  • 2.
  • At 01:05 AM on 20 Oct 2006,
  • Bob Goodall wrote:

Dear Newsnight

There is no reason why the use of less cluster points would give an inaccurate result. Using the fact that other surveys used more cluster points as a way of attacking a survey that used less cluster points seems illogical.

The countries quoted are all very different and cannot be compared in terms of sampling and surveys.

It is a huge assumption that most killings occur near the main thoroughfares.

My main criticism of those who attack the lancet's findings is to question their personal set of values. It may be the sort of self indulgent intellectual guessing game played at the sort of dinner parties held by people who have never had to face the harsh realities of what it means to live (and die) in a country like today's Iraq but I suggest that instead of trying to attack a respected medical journals attempt to show the extent of the horror that is unfolding in Iraq, by reducing the number of dead by a hundred thousand or so, that they would be better advised to keep quiet and indulge themselves somewhere else.

Bob Goodall

  • 3.
  • At 01:30 PM on 20 Oct 2006,
  • vikingar wrote:

Q. Lancet Iraq survey methodology under fire?

Hardly surprising.

Its been touted by groups with vested interests to make the most of exaggerated figures to support their multi agendas (& not for the first time).

I have previously commented #22 on questionable motives of The Lancets Editor - Richard Horton - given his vitriol anti-war opinions - a complete lack of objectivity [1]

Believe far too many people amongst 'The Usual Suspects' far more interested in impact of such figures rather than the validity of the science which generated the figure.

Which in my book is tantamount to intentional misrepresentation if not fraud & on par with anti-war crowds concern about the 'Iraqi Dossier'.

vikingar

SOURCES:

[1] http://www.bbc.co.uk/blogs/newsnight/2006/10/601000_reasons_to_regret.html

  • 4.
  • At 03:15 PM on 20 Oct 2006,
  • vikingar wrote:

Ref Bob Goodall #2

"There is no reason why the use of less cluster points would give an inaccurate result"

Try applying arguement that to Belfast, surely Catholic opinion in Falls Park will vary from Protestant view in Orange Field - ref The Union [1]

How many you ask & where you ask has bearing.

"My main criticism of those who attack the lancet's findings is to question their personal set of values"

I refer you to my #3 - Richard Horton (Lancet Editor) has a complete lack of objectivity, given his outside pursuits.

vikingar

SOURCES:

[1] http://www.wesleyjohnston.com/users/ireland/geography/belfast.html

IMHO the following statement is incorrect: "sampling the families near main thoroughfares leads to an unnaturally high number of casualties"

However, the Lancet article did NOT do this!

They sampled AT RANDOM from minor streets.

I think vikingar's comments on cluster sampling miss the point too. I can't explain cluster sampling in one sentence, but it is an established statistical method - see Wikipedia for example.

  • 6.
  • At 07:19 PM on 20 Oct 2006,
  • z wrote:

"Another study in Kosovo cites the use of 50 cluster points, but this was for a population of just 1.6 million, compared to Iraq's 27 million."

Well, right off the top that disqualifies the authors of this critique as statistically innumerate. Not only does every working statistician learn in stats 101 that is it inarguably mathematically derivable that the relationship between accuracy/precision and number of clusters does NOT involve the size of the base population, that's something that everyone sort of knows in their gut; when you're sampling water in your fishbowl for chlorine you take a testtube full, when you're sampling a swimming pool for chlorine you don't therefore have to haul out a 55 gallon drum full, you still only need a testtube full. The underlying sampling math is the same, the size of the entire population of people or of molecules does not appear in the equation; if 50 clusters is adequate to sample 1.6 million to a certain accuracy, it is precisely as adequate to sample 27 million to the same degree of accuracy. As I said, this is basic knowledge for anybody in the stats biz, and any author who even suggests otherwise for any reason can safely be dismissed. (Of course, the critique was published in the WallStreetJournal, so it's not like the wrongness seems out of place.)

That said, obviously more clusters would be more precise, but the accuracy/precision is already reflected in the confidence interval. More clusters would result in a narrower interval, but both that interval and the currently reported interval (392,979–942,636) both have a 95% chance of containing the true number. (That's not precisely technically the definition, but there's no space to explain). But there are plenty of published studies that have on the order of 47 data points; it's when you get below about 20 that people dismiss your results. That's probably the reason why the Hopkins folks, who are after all widely celebrated as being good at doing this for a living, designed the survey a priori to have 50 clusters. (Unlike the liberation of Iraq, a professional survey starts with a design that indicates that it's adequate to achieve its goals.) "A sample size of 12000 was calculated to be adequate to identify a doubling of an estimated pre-invasion crude mortality rate of 5·0 per 1000 people per year with 95% confidence and a power of 80%, and was chosen to balance the need for robust data with the level of risk acceptable to field teams." That's 12,000 people; the actual number of people in the 47 clusters sampled was 12,801.

In fact, several papers (on cluster sampling generally, not on Iraqi deaths) have been published venturing that too small a number of clusters would be biased towards an **artificially low** result, expanding on the basic premise that tossing a few rocks into a minefield is much more likely to hit no mines and give you a false estimate that there are 0 mines than it is to hit a mine with every rock and give you a false impression that it is wall to wall mines. This, in addition to the authors assigning zero deaths to the three provinces which were accidentally not sampled, makes it in fact more probable that their estimate is an underestimate rather than an overestimate.

  • 7.
  • At 01:15 AM on 21 Oct 2006,
  • SMB wrote:

My understanding is that the John Hopkins team picked homes in residential streets that ran off main avenues. It has also been pointed out that main roads attract people from all areas. The same is true with regard to attacks on street markets, mosques, office blocks, police/army recruitment centers, police stations, etc. The victims are likely to be a random cross section of society. And so I am not convinced that this potential bias is entirely applicable.

  • 8.
  • At 06:30 AM on 21 Oct 2006,
  • Jenny wrote:

vikingar wrote: "...Richard Horton (Lancet Editor) has a complete lack of objectivity, given his outside pursuits."

No doubt The Lancet's editor would have quite a bit to say in refuting your slurs, but since the report was neither produced nor edited by the magazine's editor, but by a team at Johns Hopkins in Baltimore, your remarks about him are entirely irrelevant to the validity of the report's findings.

  • 9.
  • At 11:46 AM on 23 Oct 2006,
  • vikingar wrote:

Ref Jenny #8

Slur?

Anti-war editor of medical journal with political agenda #4 twice publishes anti-war 'research', with the aim of influencing the debate, giving numbers of alleged 'deaths' far outstripping all other estimates.

If the research stated figures below the deaths say as opposed to 43-48,000 of IBC [1] would The Lancet left wing editor be so anxious to go to press?

Q. do you think that The Lancet's editorial team & the team at Johns Hopkins ran into other by coincide?

Love to know who approached who first and/other other relationships between the individuals concerned :)

Watch his speech via #4 & then declare Richard Horton editor of The Lancet is not political or leftwing or anti-war … but all three.

i.e. complete lack of objectivity in such matters.

vikingar

SOURCES:

[1] http://www.iraqbodycount.org/database/

  • 10.
  • At 09:31 PM on 23 Oct 2006,
  • Katie wrote:

z claims that the small sample of clusters used should not affect the reliability of the results. This might be true when undertaking exmaination of natural phenomena which are more likely to remain constant, however, given that the authors were carrying out a social science survey, a higher sample of clusters would be necessary to ensure accuracy.

  • 11.
  • At 11:12 PM on 23 Oct 2006,
  • vikingar wrote:

'THE RESEARCH'

The Lancet under Richard Horton, has a history of published 'flawed research' which it latter retracts.

CASE [1] - MMR Vaccine

Richard Horton had previously been involved in controversy over claims he published in 1998 in The Lancet linking the MMR vaccine with autism [1]

These were later retracted. [1]

Horton acknowledge the 'flawed research' [2]

Therefore, readers would be rightly concerned about claims of war dead - how does this impact Lancets other publications?

CASE [2] - 1st Commissioned Research ref Dead in Iraq [3]

- Estimate: 100,000
- Released: 2004
- Commissioned by: The Lancet
- Conducted by: Iraqi field researchers led by America's Johns Hopkins Bloomberg School of Public Health
- In Their Own Word: 'The report's authors admit it drew heavily on the rebel stronghold of Falluja... Strip out Falluja, as the study itself acknowledged, and the mortality rate is reduced dramatically.'
- Published: soon before the US election of 2004, bringing accusations that the respected journal had become politicised

CASE [3] - 2nd Commissioned Research ref Dead in Iraq [4]

- Estimate: 655,000
- Released: 2006
- Commissioned by: The Lancet? Johns Hopkins Bloomberg School? Other?
- Conducted by: the same researchers from America's Johns Hopkins Bloomberg School of Public Health who conducted 2004 research.
- In Their Own Words: "Their new study, they say, reaffirms the accuracy of their survey of two years ago and moves it on"
- Published: just before US elections of 2006

RICHARD HORTON:

"Richard Horton - Editor of The Lancet - "one should openly acknowledge science is political and not be afraid to get stuck into the debate,' Horton said in his first interview since the report appeared. 'To me that's one of the failures of science. It sees itself as being very apolitical, and that's just nonsense" 2004 [3]
Richard Horton - Editor of The Lancet - speech at anti war demo on 1st Day of Labour Conference 23rd September 2006 [5a]

Richard Horton - Editor of The Lancet - speech at anti war demo on 1st Day of Labour Conference 24th September 2006 [5b] *

* btw - if Horton is going to extend criticism of British government beyond Iraq to Lebanon & our relations with US & Israel, esp regarding use of 'Cluster Bombs' would like Horton to extend this to the criticism of Hezbullah esp their use of cluster bombs also [6]

If Horton rational about the use of science for political means does he have someone in common those who apply convenient liberal application of science for a 'good' i.e. German scientists who worked on V2 & other programs for the Nazis [7a] though not alone about danger of using science for political purposes [7b]

SUMMARY:

Mr Richard Horton & clique - who do you think you are you kidding?

Not only a lack of objectivity revisited but now of credibility.

With a track record of releasing inflated & speculative figures to influence & manipulate foreign elections (recall Alqaeda did the same via Madrid Bombing).

Seem to recall the Left always accusing the Right of manipulating other countries elections - 'kettle calling pot - over'

Another politicised discredited dubious 'study' placing all deaths by any manner/cause into one body count, whilst ignoring which groups are actually responsible [8] for murdering Iraqi civilians & continuously killing largest amount of Iraq civilians i.e. sectarian groups, terrorists, insurgents *

* btw - not ignoring coalition responsibilities here - but if people are intent on continuing historic Islamic fracticide amongst between communities [4] - very little they can do about it unfortunately, but the least we can do is acknowledge it not deny that the killings are happening & who is doing the killing.

Journalist Michael Fumenton of the US-based TCS website called The Lancet 'Al-Jazeera on the Thames'. [1]

Unfortunately, The Lancet has become nothing more than a discredited manipulated tool of left wing pressure politics.

If I had a subscription to The Lancet - it would have been cancelled.

vikingar

SOURCES

[1] http://www.newscientist.com/article.ns?id=dn4743
[2] http://briandeer.com/mmr/lancet-bbc.htm
[3] http://observer.guardian.co.uk/iraq/story/0,,1345442,00.html
[4] http://www.guardian.co.uk/Iraq/Story/0,,1892888,00.html
[5a] http://www.youtube.com/watch?v=v7BzM5mxN5U
[5b] http://www.youtube.com/watch?v=WwLp5P_i53c
[6] http://news.bbc.co.uk/1/hi/world/middle_east/6068154.stm
[7a] http://en.wikipedia.org/wiki/Wernher_von_Braun
[7b] http://www.csmonitor.com/2005/0927/p11s02-bogn.html
[8] http://news.bbc.co.uk/1/hi/world/middle_east/4750320.stm

  • 12.
  • At 12:35 AM on 24 Oct 2006,
  • Paul D wrote:

What William Hallowell (post 1) fails to point out is that fewer than 50% of US citizens have passports, suggesting that over half it's citizens have no intention of finding out anything about the real world. Opinion is one thing, informed opinion is something else.

  • 13.
  • At 11:10 AM on 24 Oct 2006,
  • vikingar wrote:

Ref Paul D #12

" .... fewer than 50% of US citizens have passports ..."

So only approximately 150 million people then :)

btw - does that include the 7 million+ illegal immigrants who have crossed US borders? [1]

vikingar

SOURCES:

[1] http://www.cis.org/topics/illegalimmigration.html

  • 14.
  • At 07:16 PM on 25 Oct 2006,
  • Paul D wrote:

Re: Vikingar No.13.

Doubtful if the 7 million illegals have passports but neither do they have the vote so you can discount them. If the holding of passports extrapolates across the voting population, it remains valid that fewer than 50% of those who vote for the world's most powerful government have any first hand experience of the impact the choice they make has on the rest of us.

  • 15.
  • At 08:34 PM on 25 Oct 2006,
  • Would like to know more wrote:

Paul, are you the chap who was blogging about Budapest? And if so would you be so kind as to repost your statement for us somewhere, perhaps in the my one day in history blog? - http://www.bbc.co.uk/blogs/newsnight/2006/10/my_one_day_in_history_06000800.html#c197297

  • 16.
  • At 09:40 AM on 27 Oct 2006,
  • Paul D wrote:

Paul.

Yes I am. I have replied by email.

  • 17.
  • At 06:33 PM on 30 Oct 2006,
  • Katie wrote:

Could we get back to discussing the methodology of the Iraq report please and not the democratic process in the US?

  • 18.
  • At 09:34 AM on 06 Nov 2006,
  • Shekhar Mande wrote:

Well....even if we were to assume that the sampling was biased in this study, there must be a lower and upper limits to the estimate. With biased sampling we get ~600000 deaths, and if it were unbiased, would it be say 60,000? To make a political statement, sure enough, the numbers are large enough to put the mankind to shame.

  • 19.
  • At 04:11 PM on 05 Mar 2007,
  • scientist wrote:

I agree wholeheartly with the comment by vikingarna (no. 11). I can't see why Horton is allowed to remain as Lancet's editor. The MMR incident, alone, should have made Horton's position untenable. The Iraq 'studies', clearly not suitable for a journal of Lancet's format, exposed the true nature of Horton's agenda in my eyes.

This is sad, Lancet used to be a very trustworthy journal.

  • 20.
  • At 10:02 AM on 06 Mar 2007,
  • vikingar wrote:

"Could 650,000 Iraqis really have died because of the invasion?" [1]

It goes full circle - March 2007

This is the 3rd Lancet report which has been criticised by its peers [2]

The editor, Richard Horton, has not only questionably politicised The Lancet but undermined its reputation with dodgy research.

This dubious research with dubious motives, is on par with anti-war brigade objections to the Iraq Dossier.

Looking forward to NN re-examining this issue, esp given how such figures have been used by the propaganda machines of both extremists & anti-war brigade.

vikingar

SOURCES:

[1] http://www.timesonline.co.uk/tol/news/world/iraq/article1469636.ece
[2] #11 http://www.bbc.co.uk/blogs/newsnight/2006/10/lancet_iraq_survey_methodology_under_fire.html

  • 21.
  • At 11:12 PM on 06 Mar 2007,
  • al wrote:

Post 11 is wrong to imply that the 98000 estimate in the 2004 paper would go down if Falluja were "stripped away". The 98 000 figure is the one without Falluja (and this is mentioned even in the Summary). The Falluja data on its own "indicates a point estimate of about 200 000 excess deaths in the 3% of of Iraq represented in this cluster" (p.1861 of 2004 Lancet paper), so if you want a number that would go down without Falluja, try 300 000.

"Over the last couple of days two new themes have emerged in the debate over the Lancet/Johns Hopkins University report which estimated an excess 601,000 violent deaths in Iraq as a result of the invasion..."

So they disagree with the figures the Lancet/Johns Hopkins University report has estimated...Well they would wouldn`t they. Nothing like Mudding the Waters as if that will take the heat off them (I don``t think so). It should not go amiss how many died in the Bush/Blair plan to steal there oil...

On August 6, 1990 the U.N. Security Council adopted Resolution 661 which imposed stringent economic sanctions on Iraq, providing for a full trade embargo, excluding medical supplies, food and other items of humanitarian necessity, these to be determined by the Security Council sanctions committee.

Click on my name to find out the true extent of all this slaughter.

  • 23.
  • At 08:18 PM on 07 Mar 2007,
  • vikingar wrote:

Ref al #21 & George Dutton #22

What can I say .. ref The Lancet ... The Iraq Body Count [1]

Intentional misrepresentation of 'research' esp at times of elections is tantamount to fraud, by The Lancet [2]

Such flawed propaganda, is the anti-war * version of The Iraq Dossier

* but with this rag tail bunch of agenda opportunists ... no one is really surprised (hence waning support & believability) [3]

vikingar

SOURCES:

[1] http://www.iraqbodycount.org/press/pr14.php
[2] #11 http://www.bbc.co.uk/blogs/newsnight/2006/10/lancet_iraq_survey_methodology_under_fire.html
[3] http://struggle.ws/stopthewar.html

Ref#23 vikingar

vikingar what can I say to you?.You must be one of the few that still Believe "Such flawed propaganda" do you still believe there is WMD in Iraq as well?.Click on my name and read it all.One more thing whatever the body count is add at the end in big letters.
AND COUNTING.

  • 25.
  • At 08:43 PM on 09 Mar 2007,
  • vikingar wrote:

Ref George Dutton #24

"One more thing whatever the body count is add at the end in big letters. AND COUNTING"

1) yes

2) & by whose hand? perchance Islamic fracticide local Muslims & international Muslims travelling to kill each other *

* continuation of 1,500 year old Sunni v Shia conflict, predates formation of US or British Empire

Have you never quoted either of The Lancet flawed & fictitious figures to justify your objections to the war [2]

Today the only real credible source the IBC [1] high end figure of 64,000 is tremendously different to The Lancet mid range figure of 650,000 [3] **

** in 2006 when 2nd Lancet report release stating 650,000 the high end IBC estimate was approx 53,000

Ref sanctions, they were a sick joke, that is one of the reasons an inept UN could not be further trusted after 10 years failed sanctions causing undue suffering to innocent Iraqis (by corruption & abuse of supplies by Saddam regime & members of the UN, including head of Food for Oil & Secretary Generals son) the madness had to end some time ***

*** regardless of whether Saddam was eventually assassinated, a victim of another coup attempt, died of natural causes, or by hand of the coalition.

vikingar

SOURCES:

[1] http://www.iraqbodycount.org/
[2] http://www.timesonline.co.uk/tol/news/world/iraq/article1469636.ece
[3] http://www.iraqbodycount.org/press/pr14.php

"Iraqi deaths survey was robust"

The British government was advised against publicly criticising a report estimating that 655,000 Iraqis had died due to the war, the BBC has learnt.

http://news.bbc.co.uk/1/hi/uk_politics/6495753.stm

For downloadabale questionnaires you might want to check out the Survey Questionniare Archive at: http://www.cadsr.udel.edu/sqa

The Survey Questionnaire Archive is an open collection of survey questionnaires used in social sciences and public policy making.

The archive is created, hosted and maintained by the Center for Applied Demography & Survey Research at the University of Delaware.

This digital collection allows users to browse, search, store and share survey instruments over the web.

The goal of the archive is to collect, capture, disseminate and preserve a wide variety of survey instruments.

This post is closed to new comments.

The BBC is not responsible for the content of external internet sites