US senators have rejected attempts to open a debate on a bill which included a provision allowing the repeal of the ban on openly gay military personnel.
Just 56 senators voted in favour of debating the defence authorisation bill, four short of the 60 required.
Gay people can serve in the military, but face expulsion if they reveal their sexuality. US President Barack Obama has promised to scrap the policy.
Democrats could still try again later this year to pass the legislation.
Reacting to the vote - which split largely along Republican-Democrat party lines - White House spokesman Robert Gibbs said: "We're disappointed at not being able to proceed on the legislation, but we'll keep trying."
The BBC's Paul Adams, in Washington, says the vote is a setback for Mr Obama, who had hoped to deliver on a campaign promise to repeal the law - known as "don't ask, don't tell".
The Pentagon is conducting a study into how repeal might be implemented, but Republicans, and many in the military, fear that it could damage morale at a time when the armed forces are fighting two wars, our correspondent adds.
Earlier, the only Republican senator to support repealing the law, Susan Collins, said she was withdrawing her support.
Her vote was seen as the crucial 60th vote needed to limit debate and advance the bill in the 100-seat Senate.
In the event Democratic senators Blanche Lincoln and Mark Pryor both sided with Republicans to block the bill, while Senate Majority Leader Harry Reid, a Democrat, also voted against the measure as a procedural tactic.
It is not clear how the Democrats intend to respond to this setback in the Senate, but it seems highly unlikely that President Obama will get his way on gays serving openly in the military before November's mid-term elections.
If the Republicans retake control of one or more houses of Congress, then the president may never get his way, our correspondent says.
Defence Secretary Robert Gates and the Chairman of the Joint Chiefs of Staff, Admiral Mike Mullen, have said they support a repeal but want to move slowly on it so as to not damage morale.
Mr Gates has asked Congress not to act until the Pentagon finishes a study on the impact on the military of lifting the ban.
The Obama administration has said any repeal of the law would not be implemented until after the study, which is due on 1 December, is completed.
Most Americans now accept openly gay service personnel, according to a recent Gallup poll.
Mr Obama's nominee to take over command of the Marine Corps said on Tuesday that he opposed repealing the "don't ask, don't tell" policy.
In testimony to a Senate committee, General James Amos said letting gay people serve openly in the force could disrupt unit morale.
"I'm concerned that a change now will serve as a distraction to marines who are tightly focused at this point on combat operations in Afghanistan," he said in written testimony.
"In my personal view, the current law and associated policy have supported the unique requirements of the Marine Corps, and thus I do not recommend its repeal."