A North Korean patrol boat violated a sea border with the South several times late on Monday, officials in Seoul say.
The ship spent a few hours south of the border, nearing a South Korean border island, before returning after repeated warnings from the South, they added.
North Korea disputes the maritime border and has sent boats across it in the past.
The incident comes amid joint military drills between the US and South Korea which are opposed by Pyongyang.
The North Korean patrol ship crossed the Northern Limit Line (NLL), which South Korea considers the maritime border between the two sides, at around 22:46 local time (13:46 GMT), South Korea's defence ministry said.
The South broadcast warnings 10 times before the ship returned, at around 02:25 local time (17:25 GMT), the ministry added.
'Testing the South'
The NLL was drawn unilaterally by the United Nations Command at the end of the 1950-1953 Korean War.
However, North Korea has disputed the NLL and drawn its own border further south of the line.
"The North Korean ship's NLL violation is seen as part of military drills," defence ministry spokesman Kim Min-seok said. "It is believed that [the ship] intended to test the South Korean military."
The ship was said to have come within 23km (15 miles) of South Korea's border island of Baengnyeong.
In 2010, a South Korean warship sank near Baengnyeong, leaving 46 sailors dead. South Korea said North Korea torpedoed the ship, but Pyongyang denied this.
Monday's incident comes on the same day joint annual US-South Korea military exercises began.
More than 12,500 US troops will take part in the exercises, which include Key Resolve, a computer-based simulation, and Foal Eagle, which involves air, ground and naval drills.
Pyongyang is opposed to the drills and has previously called them "exercises of war".
Tuesday is also the last day of rare family reunions for North and South Korean relatives separated after the Korean War.
A few hundred North and South Korean relatives met briefly for the first time in decades at North Korea's Mount Kumgang resort, amid an apparent thaw in inter-Korean ties.