South Korea has accused the North of blocking the return of 27 North Koreans who strayed across the border.
The North has said Seoul is keeping hostage four North Koreans, from a boatload of 31 which drifted over the sea border in heavy fog on 5 February.
The refugee return row comes while South Korea is hosting large-scale military drills with the United States.
Both North and South have called for talks to ease tensions; the last effort ended without agreement last month.
South Korea's Unification Minister, Hyun In-taek, told parliament he was respecting the free will of the four people who had chosen to stay in the South.
When South Korean officials took the 27 others to the Panmunjom border village there were no North Koreans prepared to accept the group, Seoul said.
"We are still waiting for North Korea's response but we cannot wait forever," said a ministry spokeswoman Lee Jong-joo.
"We will make a decision if North Korea keeps silent," she told AFP.
For its part, North Korea has accused the South of "despicable unethical acts".
"This cannot be interpreted otherwise than a grave provocation to the DPRK (North Korea)," said a statement attributed to a spokesman for the North's Red Cross.
This comes in the same week that North Korea threatened to turn the South's capital, Seoul, into a "sea of flames" because of large-scale military drills being carried out by US and South Korean forces.
The joint command says the drills involving more than 200,000 troops as defensive - the North says they are a rehearsal for invasion to topple its communist government.
Relations have been fraught since the North shelled a South Korean island in November, killing four people.
The attack came just months after 46 sailors were killed when a South Korean warship was sunk. An international investigation blamed a North Korean torpedo attack but Pyongyang denies any involvement.