Amir Khan's rematch with Lamont Peterson has been called off by his promoter following the American's failed drugs test.
Peterson tested positive for synthetic testosterone after both fighters agreed to random testing before their 19 May light-welterweight bout in Las Vegas.
"This has upset me because I wanted to settle the score," Khan told BBC Sport.
"They should at least give those world titles back [to me]. I'm upset for the [5,000 travelling] fans."
The issue was referred to the Nevada State Athletic Commission, which was due to hold a meeting on Tuesday, 15 May.
However, Khan's promoters, Golden Boy Promotions, felt that date was too close to the fight and decided to call it off.
The International Boxing Federation will now speak to Peterson about his failed test before deciding whether to strip him of their light-welterweight title and potentially return it to Khan.
An IBF spokesman said: "We are still waiting to receive all the information from the Nevada State Athletic Commission, as well as to speak with Lamont Peterson."
Khan, who lost his belts when the pair met in December, wants to find another opponent to fight in June or July and says that, even after the news of Petersen's failed drugs test, he still wanted it to go ahead.
"The truth's come out now and it just proves that Lamont Peterson was a cheat really," he added.
"It hurt me more than anything because I really did want to go in there and put on a performance and show the world how good Amir Khan is.
"I was so fit and looking forward to this fight. I wanted this fight even when I heard he failed his drugs test.
"I did everything to get the fight on, but my parents and my team said to me 'Look, at the end of the day Amir, it could be a risk to your life. It's not going to be fair to step into the ring with someone who is gonna be on drugs and gonna be cheating'."
Golden Boy Promotions chief executive Richard Schaefer - Khan's promoter - confirmed earlier this week Peterson tested positive from the A and B samples, but then gave a negative test in follow-up samples.
Schaefer said he was disappointed he had not been made aware earlier of key information regarding Peterson's drugs test, leaving little time for an alternative opponent to be found.
"For Amir, it's a big blow," he said. "It's a big blow for fight fans and for the sport. Team Khan are all here in Los Angeles.
"What is shocking is that Team Peterson and the agency that conducted the drug testing - the Voluntary Anti-Doping Agency - did not give us the information that Peterson had tested positive back in April - 12 April to be exact.
"They waited until this Monday to disclose that to us. That put everybody in an extremely unfortunate position, because it does not give us enough time to find a proper replacement with little more than a week to go."
Peterson's team defended their boxer and pledged to clear his name.
"Team Peterson is very disappointed and distraught by the decision to cancel the 19 May rematch against Amir Khan," a statement read.
"This is an extremely difficult decision to accept. We did everything that was asked of us in efforts to comply.
"To support our stance we provided the Nevada State Athletic Commission with a significant amount of factual medical data in response to these allegations.
"Lamont did a battery of tests this week and saw a number of independent board-certified physicians. They all had the exact same conclusion as the doctor that initially treated and diagnosed Lamont's medical condition.
"He did nothing wrong and he was more than ready to go through with the fight.
"We will vigorously pursue the truth with regards to this matter and continue to fight to protect this young man's character, credibility and all he has accomplished. Once all the facts have been reviewed we have no doubt that he will be vindicated."
Vada, however, said it could not release the results of Peterson's failed test because documents needed to be signed by both fighters, saying they were happy to have their results disclosed.
Robert Smith, general secretary of the British Boxing Board of Control, believes it vindicates drug-testing in the sport.
"In sport these things are put in place to try and make sure we keep the cheats away and he's been found out," Smith said.
The fighters were scheduled to meet for a second time after the World Boxing Association ordered a rematch following Peterson's controversial victory over Khan.
The cancellation of the bout also spells the end for British fighter Gary Sykes' title hopes. The 28-year-old Yorkshireman was due to challenge unbeaten Adrien Broner for the WBO super-featherweight title in a support fight on the card.