German Chancellor Angela Merkel said her party had a "difficult day" after right-wingers made gains in state elections.
The anti-immigrant AfD party had campaigned against what it called Mrs Merkel's "catastrophic" decision to accept a million migrants and refugees.
However, a German government spokesman said the country's immigration policy would not change.
The chancellor said a European solution was needed for the crisis.
"The fact is that in the eyes of the people no appropriate and satisfactory solution has yet been found and this had a very great impact on the vote," she said.
Germany and other countries have reinstated some border controls in response to the flow of migrants.
Despite making gains in all three states where voting took place, the AfD remains a minor force, especially in western Germany.
Mrs Merkel's Christian Democrats (CDU) lost votes but remained the strongest party in the poor eastern state of Saxony-Anhalt, where the AfD made its biggest gains, taking a quarter of the vote. That made it the second-biggest party there.
The CDU lost a third of its vote in Baden-Wuerttemberg, reaching a historic low of 27%. The Green party came first, while the AfD gained about 15% of the vote.
In Rhineland-Palatinate the Social Democrats maintained their position as the strongest party, the CDU came second - in a state they had hoped to win - and the AfD came third with 12.5%.
Analysis: Damien McGuinness, BBC News, Berlin
The unprecedented success of the populist AfD shows just how vehemently some German voters disagree with Chancellor Merkel's pro-refugee stance. Many are conservative voters disenchanted with her centrist policies, and swayed by the robust populist narrative offered by AfD.
So is this a blow to Mrs Merkel's leadership? Not necessarily. It was also a historic victory for the Greens, who support the chancellor's position on refugees.
And during the campaign in Baden-Wuerttemberg and Saxony-Anhalt the CDU candidates saw their voters fall away as they sporadically tried to undermine the chancellor's message and regain the populist ground from AfD.
Remarkably, in those states it was the CDU's left-wing opponents who backed Mrs Merkel's refugee policies most consistently - and tonight reaped the rewards for it. These results are proof that Germany is becoming increasingly polarised between voters who passionately support Mrs Merkel's pro-refugee stance and those angered by it.
Already represented in five of Germany's 16 regional parliaments, the AfD has campaigned on slogans such as "Secure the borders" and "Stop the asylum chaos".
In Berlin on Saturday, about 2,000 right-wing demonstrators carrying German flags chanted "Merkel must go!" and "We are the people!".
The poor results could put additional pressure on Mrs Merkel, just as she is trying to push through an EU deal with Turkey to reduce the numbers of migrants and refugees entering western Europe.
As Europe's largest economy, Germany has a leading role in policy-making for the European Union.
At a summit earlier this week, the chancellor promoted a last-minute draft of the deal and demanded the support of other European leaders.
Ms Merkel still needs to complete that deal at another summit at the end of this week.
Meanwhile, thousands of migrants have massed in muddy camps in the Greek border town of Idomeni after countries across the Balkans closed their borders.
- Founded in 2013 by Bernd Lucke, Alexander Gauland and Konrad Adam to oppose German-backed bailouts for poorer southern European countries
- Mr Lucke, seen as a moderate, wanted Germany out of the euro but his colleagues were unhappy that he wanted to focus exclusively on euro-related issues
- He quit the party in early July 2015, arguing it was becoming increasingly xenophobic
- Right-winger Frauke Petry replaced him as party leader
- It became the first anti-euro party to win seats in a German regional parliament, receiving almost 10% of the vote in the eastern German state of Saxony in 2014, and went on to win seats in four other states' parliaments in 2014 and 2015
- The party had seven MEPs elected in the 2014 European elections (including Mr Lucke), but only two remain party members
- AfD was part of the European Conservatives and Reformists Group, like the UK's ruling Conservatives, but its two MEPs look set to be expelled from the group over comments on shooting refugees