The UK has raised the threat to British shipping in Iranian waters in the Gulf to the highest level - where the risk of attack is "critical".
The step was taken on Tuesday, amid growing tensions in the region.
On Wednesday, Iranian boats tried to impede a British oil tanker in the region - before being driven off by a Royal Navy ship, the MoD said.
Iran had threatened to retaliate for the seizure of one of its own tankers, but denied any attempted seizure.
The Department for Transport said it regularly provided security advice to UK ships in high-risk areas.
The threat level means British ships are advised not to enter Iranian waters, BBC defence correspondent Jonathan Beale said.
Boats believed to belong to Iran's Islamic Revolution Guard Corps (IRGC) approached the British Heritage tanker and tried to bring it to a halt as it was moving out of the Gulf into the Strait of Hormuz.
HMS Montrose, a British frigate shadowing the BP-owned tanker, was forced to move between the three boats and the ship, a Ministry of Defence spokesman said.
He described the Iranians' actions as "contrary to international law".
Guns on HMS Montrose were trained on the Iranian boats as they were ordered to back off, US media reported. The boats heeded the warning and no shots were fired.
Last week, British Royal Marines helped the authorities in Gibraltar seize an Iranian tanker because of evidence it was carrying oil to Syria in breach of EU sanctions.
A spokesman for the Royal Gibraltar Police said they had arrested the captain and chief officer of the Iranian tanker on Thursday, on suspicion of breaching EU sanctions, but neither had been charged.
The BBC has been told British Heritage was near the island of Abu Musa when it was approached by the Iranian boats.
Although Abu Musa is in disputed territorial waters, HMS Montrose remained in international waters throughout.
Defence Secretary Penny Mordaunt said the government was concerned by the incident and urged the Iranian authorities to "de-escalate the situation".
Foreign Secretary Jeremy Hunt added the UK would monitor the situation "very carefully".
Prime Minister Theresa May's official spokesman said the government was "committed to maintaining freedom of navigation in accordance with international law".
A spokesperson for the US State Department condemned Iran's actions and said that Washington would continue to work closely with the UK.
Morgan Ortagus said: "We commend the actions of the Royal Navy in ensuring freedom of navigation and the free flow of commerce through this critically important waterway."
Commander of the US Fifth Fleet Vice Adm Jim Malloy described the incident as "unlawful harassment" and said the fleet would continue to work closely with the Royal Navy to defend "the free flow of commerce".
What does Iran say?
The navy of Iran's Islamic Revolution Guard Corps has denied claims it tried to seize the tanker, Iranian news agencies reported. IRGC's navy said there had been no confrontation with any foreign vessels in the past 24 hours.
Iranian foreign minister Mohammad Javad Zarif said the UK made the claims "for creating tension".
"These claims have no value," Mr Zarif added, according to the Fars news agency.
Why are UK-Iran tensions escalating?
The relationship between the UK and Iran has become increasingly strained, after Britain said the Iranian regime was "almost certainly" responsible for the attacks on two oil tankers in June.
Tensions grew after the seizure of an Iranian oil tanker by authorities in Gibraltar, assisted by British Royal Marines.
On Thursday an Iranian official told the BBC the seizure was "unnecessary and non-constructive escalation by the UK" and called for the tanker, Grace 1, to be released.
An Iranian official previously said a British oil tanker should be seized if Grace 1 was not released.
On Wednesday, Iranian President Hassan Rouhani called the UK "scared" and "hopeless" for using Royal Navy warships to shadow another British tanker in the Gulf.
"You, Britain, are the initiator of insecurity and you will realise the consequences later," Mr Rouhani said.
The Royal Navy has a frigate, four minehunters and a Royal Fleet Auxiliary support ship already stationed in a permanent Naval Support Facility in the region, at Mina Salman in Bahrain.
This is enough to provide reassurance, but probably not to deal with a crisis, BBC defence correspondent Jonathan Beale said.
Ministers would now have to consider sending another Royal Navy ship to the region - but this could further escalate tensions with Iran, our correspondent said.
Foreign Office officials said they were keeping the UK's military posture in the region under constant review but insisted they did not want to see tensions escalate.
By Paul Adams, diplomatic correspondent
The government says it absolutely doesn't want to get sucked into a confrontation with Iran, but cannot ignore the fact that life for British-flagged ships in the Gulf is becoming increasingly precarious. Hence the decision to raise the ship security level to its highest category - essentially a warning to British ships not to enter Iranian waters.
This is an unusual but not unprecedented step, a reflection of the government's mounting concern.
With limited military resources in the Gulf (one frigate and four mine countermeasures vessels), there's a limit to how much protection the Royal Navy can offer to British shipping. On any given day, there are between 15 and 30 large UK ships, including oil and gas tankers, in the Gulf, with between one and three transiting the Strait of Hormuz.
The United States is looking to put together a coalition to ensure freedom of navigation through the Gulf, but has yet to share details or make requests of possible partners. With the US and Iran locked in a bitter dispute over the future of the 2015 Iran nuclear deal, some countries who already participate in a 33-nation Combined Maritime Force, based in Bahrain, may be reluctant to get more involved.
Can ships in the area avoid Iranian waters?
The Strait of Hormuz, through which all ships must pass to enter the Gulf, is so narrow - just 21 nautical miles (39km) at its narrowest - that Iranian and Omani territorial waters meet in the middle, BBC Security correspondent Frank Gardner says.
So instead of sailing through international waters, ships must pass through Iranian or Omani territory which both extend 12 nautical miles out from their coasts.
Ships do this under something called Rights of Straits Passage - part of a UN convention which gives ships free passage through the world's chokepoints like the Strait of Gibraltar and the Malacca Strait.
In the case of the Strait of Hormuz, shipping is channelled through two lanes heading in opposite directions, each one two nautical miles wide.
This is called the Traffic Separation Scheme.
Both Iran and the US Navy deploy warships to patrol this area and have narrowly avoided confrontation on several occasions.
Once ships pass through the Strait of Hormuz and enter the Gulf they need to be wary of a contested area around the islands of Abu Musa and the Greater and Lesser Tunbs.
These are claimed by both Iran and the UAE but occupied solely by Iranian forces.
What we know about British Heritage
It is understood British Heritage was not carrying cargo at the time of the incident with the Iranian boats.
The vessel is registered at the port of Douglas, in the Isle of Man.
What are US-Iran tensions about?
The US has blamed Iran for attacks on six oil tankers in May and June.
On Wednesday, the chairman of the US military's Joint Chiefs of Staff said it wanted to create a multi-national military coalition to safeguard waters around Iran and Yemen.
However, the UK and the rest of Europe were more cautious, fearing this would escalate tensions, BBC diplomatic correspondent James Landale said.
It follows the Trump administration's decision to pull out of an international agreement on Tehran's nuclear programme and reinforce punishing sanctions against Iran.
Tehran has begun to nudge the levels of its enriched uranium beyond the limits of a nuclear deal agreed with a group of world powers.
Iran's ambassador to the UN Majid Takht-Ravanchi told the BBC Europeans must do more to compensate Tehran for economic losses inflicted by US sanctions - or Iran would continue to step up its nuclear programme.