The EU says it could impose sanctions on Turkey over "provocations and pressures" in a row with Greece over energy resources and maritime borders.
European Commission President Ursula von der Leyen called on Ankara to "abstain from unilateral actions" in the eastern Mediterranean.
She spoke early on Friday during a meeting of EU leaders in Brussels.
Earlier, Turkey and Greece set up a military hotline to try to reduce the risk of clashes in the region.
Tensions rose earlier this year when Turkey sent a ship into a disputed area to search for potentially rich oil and gas deposits.
What has the EU said?
Mrs von der Leyen told reporters that the EU wanted "a positive and constructive relationship with Turkey and this would be also be very much in Ankara's interest".
"But it will only work if the provocations and pressures stop," she said. "We therefore expect that Turkey from now on abstains from unilateral actions. In case of such renewed actions by Ankara, the EU will use all its instruments and options available. We have a toolbox that we can apply immediately."
After their late-night meeting, EU members agreed to review Turkey's behaviour in December and impose sanctions if "provocations" had not stopped.
Austrian Chancellor Sebastian Kurz, posting on Twitter after the meeting, said: "The EU issues a clear threat of sanctions against Turkey should it continue to violate international law."
European Council President Charles Michel said the EU was offering Turkey closer relations on trade and other areas, but holding out the threat of sanctions if tensions in the Mediterranean did not de-escalate. "We very much want to give political dialogue a chance to move toward greater stability," he said.
Arriving for the second day of the EU summit in Brussels, Greek Prime Minister Kyriakos Mitsotakis said the EU had sent "a message of unity, solidarity and determination" to Ankara, which made clear the consequences of its "aggressive behaviour".
But a Turkish official said it was hoped talks with Greece would resume soon and that sanctions would put dialogue at risk. "If the EU applies sanctions, this will not deter us. On the contrary this would increase our resolve," the official told Reuters news agency.
What's the background?
The European Union and Turkey have long held a fragile relationship.
Turkey has been a long-term candidate for membership of the EU, but efforts have stalled. EU leaders have criticised Turkey's record on human rights and the rule of law, in particular in the wake of the 2016 failed military coup.
Despite the strains, Turkey remains an important partner for the EU. Turkey hosts millions of migrants and struck a deal with the EU that limited the numbers arriving in Greece.
Greece and Turkey are both Nato members, but have a history of border disputes and competing claims over maritime rights.
Tensions flared in August when Ankara sent the research ship into an area south of the Greek island of Kastellorizo which is claimed by Greece, Turkey and Cyprus.
Greece called the move a "new serious escalation" and the EU has backed its members Cyprus and Greece against Turkey.
Tensions eased somewhat when the research ship returned to Turkish waters last month and both sides said they were prepared to resume talks.
Why the military hotline?
The announcement of the hotline on Thursday followed talks between Turkey and Greece at the Nato headquarters in Brussels.
"I welcome the establishment of a military de-confliction mechanism, achieved through the constructive engagement of Greece and Turkey, both valued Nato allies," said Nato Secretary-General Jens Stoltenberg.
"This safety mechanism can help to create the space for diplomatic efforts to address the underlying dispute and we stand ready to develop it further."
Such mechanisms enable direct communication between two sides. Russia and the US set one up during the Cold War and it has been in operation ever since.
Last month, France - which is also at odds with Turkey over the crisis in Libya - deployed two Rafale fighter jets and a naval frigate in the Eastern Mediterranean because of the tensions between Greece and Turkey.