The family of Nazanin Zaghari-Ratcliffe says Boris Johnson's resignation as foreign secretary has put their bid for her release "back to square one".
The 39-year-old was detained in Iran in April 2016 and given a five-year jail sentence for spying.
Her sister-in-law, Rebecca Ratcliffe, said Johnson's resignation had put Mrs Zaghari-Ratcliffe's case "at the bottom of the pile".
The Foreign Office said it is working to secure the "outcome we all want".
Speaking on BBC Radio Wales, Dr Ratcliffe, a GP from Cwmbran, Torfaen, said Johnson developed a "genuine concern" for Mrs Zaghari-Ratcliffe, from Hampstead, north London, and his resignation is a "setback".
In November, while serving as foreign secretary, Johnson was forced to apologise after contradicting the claim that the mother was on holiday in Iran when she was arrested.
However, Dr Ratcliffe said progress was being made after he visited Iran and to discuss the case.
"It came a bit out of the blue. We weren't expecting it and it's a bit of a setback.," she said.
"Poor Nazanin has been detained for two years and this is her third foreign secretary in that time."
Jeremy Hunt was appointed foreign secretary in the wake of Johnson's departure, but the family are concerned his attention will be elsewhere.
Mrs Zaghari-Ratcliffe was close to securing diplomatic protection, but the change means a decision is now not expected for several months.
"Every time there's a change in foreign secretary and anything happens in the government to destabilise things, her case just gets put to the bottom of a very long list of priorities for the foreign secretary and foreign office," Dr Ratcliffe added.
"It felt like we had momentum. It felt like we had a foreign secretary who was taking her very seriously and was going to try and get her out.
"Nazanin gets more and more disheartened and it's like she's been forgotten.
"It's a heart-breaking situation all round. Her daughter has just turned four and that's her third birthday she's had without her parents. She won't carry on like this much longer."
A Foreign Office spokesman said: "We will continue to approach our consular cases in Iran in a way that we judge is most likely to secure the outcome we all want. Therefore we will not be providing a running commentary on every twist and turn."