Turkey has expelled Israel's ambassador and suspended all military agreements over its refusal to apologise for last year's raid on a Gaza-bound flotilla.
This came as the UN published a report saying that Israeli commandos used excessive force when they boarded an aid ship. Nine Turkish activists died.
Turkish Foreign Minister Ahmet Davutoglu said it was "time Israel pays a price for its illegal actions".
Israel has refused to apologise and said its troops acted in self-defence.
"Israel, like any other country, has a legitimate right to protect its citizens and soldiers," an Israeli government official told the BBC.
The report was leaked to the New York Times on Thursday, the day before it was delivered to UN Secretary General Ban Ki-moon's office.
"The secretary general's idea was to help bring these two countries back together again. He deeply regrets the fact that this has not been possible through this report," said deputy UN spokesman Eduardo del Buey.
"I imagine he will want time to read the report, to discuss it with his officials and then take a decision as to future steps."
US state department spokeswoman Victoria Nuland said Washington hoped Turkey and Israel "will continue to look for opportunities to improve their longstanding relationship, and we will encourage both to work towards that end", according to the AFP news agency.
'Null and void'
The nine pro-Palestinian activists who died were on board the Turkish-flagged ship, Mavi Marmara, when it was intercepted by the Israeli navy in international waters as sailed towards Gaza's coast on 31 May 2010.
At the time, the Israeli military said its commandos fired live rounds only after being attacked with clubs, knives and guns. But activists on board said the commandos started shooting as soon as they hit the deck.
The UN inquiry chaired by former New Zealand Prime Minister Geoffrey Palmer found the Israeli troops faced "significant, organised and violent resistance from a group of passengers" and were therefore required to use force for their own protection.
But it said Israel's decision to board the vessels "with such substantial force at a great distance from the blockade zone and with no final warning immediately prior to the boarding was excessive and unreasonable".
The report noted "forensic evidence showing that most of the deceased were shot multiple times, including in the back, or at close range".
The inquiry also found Israel's naval blockade "was imposed as a legitimate security measure in order to prevent weapons from entering Gaza by sea and its implementation complied with the requirements of international law".
The report was completed months ago, but its publication was delayed several times as Turkey and Israel attempted to repair their diplomatic relations, which have been frozen since the flotilla incident.
But on Friday, hours before the report was released by the UN, Turkey' announced that diplomatic relations had been reduced to the level of second secretary and all military agreements had been suspended.
"The time has come for Israel to pay for its stance that sees it above international laws and disregards human conscience," Mr Davutoglu said. "The first and foremost result is that Israel is going to be devoid of Turkey's friendship."
"As long as the Israeli government does not take the necessary steps, there will be no turning back," he warned.
Mr Davutoglu said the report "displayed the violence committed by the Israeli soldiers", but added that some of its findings were questionable.
"Turkey does not recognise Israel's blockade of Gaza. It will secure the study of this blockade at the International Court of Justice."
He added that Turkey would take all necessary measures to protect its shipping in the eastern Mediterranean.
Asked to comment on the UN panel's decision not to recommend that Israel apologise, Turkish President Abdullah Gul said: "To be frank, the report is null and void for us."
Turkey's embassy in Washington later said in a statement that it was "determined to defend the rights of its citizens and will take legal actions against all those responsible for the crimes committed".
Hamas, whose decision to seize control of the Gaza Strip in 2007 led to Israel imposing the blockade, applauded the Turkish move.
"This is a natural response to the Israeli crime against the flotilla," spokesman Sami Abu Zuhri told the BBC.
The Israeli official said the government accepted with "reservations" the UN report, which it considered "professional, profound and serious".
"During the events of the Mavi Marmara flotilla, Israeli soldiers boarded the boat with non-lethal means. They had no intention to hurt anyone," the official said.
"Once the Israeli soldiers were viciously attacked by dozens of violent IHH activists, armed with batons, knives and steel pipes, the soldiers had to defend themselves," the official added, referring to the IHH Humanitarian Relief Foundation, which Israel has banned for supporting Hamas.
"As recommended in the report, Israel again expresses regret about the loss of life but will not apologise for acts of self-defence by its soldiers."