Danes have rejected adopting EU rules on cross-border policing in a referendum that could have seen the country take closer ties with the bloc, according to final results.
Denmark's centre-right government had wanted to abandon some Danish opt-outs from EU home affairs legislation.
But with all votes now counted, more than 53% said No to the proposals.
The vote comes weeks after the Paris attacks and as Europe struggles to deal with record numbers of migrants.
"It is a clear no," Prime Minister Lars Lokke Rasmussen said, adding he had "full respect" for the voters' decision.
The government, backed by the opposition, had campaigned for Yes, saying it would help Danish authorities in the wake of the Paris attacks.
Analysis: Gavin Lee, BBC News Copenhagen
Ultimately, voting No means Denmark remains exempt from large parts of the EU's criminal justice and home affairs system, a position it negotiated in 1993.
It risks losing access to Europol, Europe's crime and intelligence-sharing agency, a service frequently used by Denmark.
The confusing wording of the referendum question seems to have been a factor.
One voter described it as "the most baffling in the history of the EU", and on the foggy, wet streets of Copenhagen, that sentiment seemed to be shared by voters as they left polling stations, saying the question was too complicated and technical, and that explanations from politicians were not comprehensive.
For Denmark's government, urgent talks will now take place between Copenhagen and Brussels, to work out the ramifications of what the No vote means.
The result is likely to have been monitored by British politicians, interested to see with the upcoming UK referendum battle about to play out, how the gut instinct of the voters can be tied to current European events, and how the particular phrasing of the question can discourage or influence how tied-in to the European project Britons want to be.
Several of the Paris attackers were French nationals who had been living in neighbouring Belgium. At least one surviving gunman, Salah Abdeslam, is thought to have fled across the French border in the aftermath of the shootings and suicide blasts in Paris on 13 November.
The result means Denmark will have to negotiate a special agreement to stay inside Europol, the EU's law enforcement agency which tackles organised crime and terrorism.
"We will work very hard for the Danes to get the best possible agreement. But it will be difficult," Soren Gade of the governing Venstre party told the Ritzau news agency.
The anti-immigration Danish People's Party (DPP), which props up Mr Rasmussen's administration in parliament, had urged voters to say No to avoid giving away further sovereignty to Brussels.
Although a Yes vote would not have affected Denmark's opt-out on immigration, the DPP argued that it could eventually have led to immigration policies being dictated by the EU.
Unlike Denmark, the UK and Ireland have opt-ins on justice and home affairs legislation, which enable them to choose whether to accept or reject legislation on a case-by-case basis.
The result of the referendum is likely to be of interest in Britain, whose government is trying to renegotiate its relations with the EU before holding a vote on whether to remain in the bloc.