The US is marking nine years since 9/11 amid controversy over plans for an Islamic centre near Ground Zero and a threat to burn the Koran.
In New York, relatives read out the names of those who died when hijacked airliners hit the World Trade Center.
At the Pentagon, also hit by a hijacked plane on 9/11, President Barack Obama said the US was not at war with Islam.
Earlier, the pastor behind the threat to burn Korans in Florida said the event had been cancelled permanently.
"We will definitely not burn the Koran, no," the Reverend Terry Jones told NBC's Today show. "Not today, not ever."
Later on Saturday, rival protests began close to Ground Zero, by groups opposed to or supporting the building of a controversial Islamic cultural centre in the area.
Speaking at a memorial event at the Pentagon, President Obama paid tribute to those who died on 11 September 2001.
He said that while it was tempting to dwell on their final moments, the memorial events were taking place "to remember the fullness of their time on Earth".
Mr Obama also repeated his recent calls for unity, saying: "It was not a religion that attacked us that September day. It was al-Qaeda."
"We will not sacrifice the liberties we cherish or hunker down behind walls of suspicion and mistrust."
US Vice-President Joe Biden and his wife were among the dignitaries who attended the commemoration in New York.
Relatives read out the names of the dead. For the first time, they were joined by people working to build the Ground Zero memorial.
"No other public tragedy has cut our city so deeply, no other place is as filled with our compassion, our love and our solidarity," said New York Mayor Michael Bloomberg.
Bells tolled across the city to mark the moment when the first plane struck the North Tower of the World Trade Center. There was also a minute's silence, with further pauses at the time the South Tower was hit and the times at which each tower fell.
This year's anniversary is likely to be the most contentious and fraught yet, says the BBC's Laura Trevelyan in New York.
As the city's official commemorations wound down, rallies began both for and against the proposed Islamic community centre and mosque near Ground Zero.
Both sides want to use the emotion of the day to highlight their causes, our correspondent says.
President Obama's wife Michelle, the First Lady, was present at a commemorative event in Shanksville, Pennsylvania, where a fourth plane crashed nine years ago.
In London, 67 roses - one for each British victim - were laid at a memorial in Grosvenor Square, not far from the American embassy.
A small group of Muslim demonstrators set fire to an American flag outside the US embassy in protest at Mr Jones's now-cancelled plan to burn the Koran.
Victims' families divided
Some relatives of the 9/11 victims oppose the cultural centre in New York because they say it disrespectful to have a reminder of Islam by the place where so many died.
However, other families support the project as an expression of America's commitment to freedom of religion.
Rosaleen Tallon, who lost her firefighter brother on 9/11, told the BBC this was the moment to speak out against the Islamic centre.
"It's just a reminder that in the name of that religion, my brother was murdered," she said. "I feel very strongly that on the anniversary of 9/11, we have to defend that and honour those people."
But Myrna Bethkey, who also lost a brother in the attacks, said she supported the proposal.
"I have absolutely no problem with it, I support it wholeheartedly as a place to continue to build bridges, create understanding," Ms Bethkey said. "The more we walk in each others' cultures and each others' shoes and learn about one another, the less possibility there is for misunderstanding."
Imam Hafiz Chowdhury, who leads a mosque in Lower Manhattan, said the planned Islamic centre should move, if that would relieve the tension.
The anti-mosque rally is expected to hear taped messages from Bush-era diplomat John Bolton and conservative blogger Andrew Breitbart.
Dutch anti-Islam politician Geert Wilders, who advocates banning the Koran, plans to address the rally in person, as do a handful of Republican congressional candidates.
Saturday saw new protests in mainly Muslim countries over the Koran-burning proposal, with rallies reported in Somalia and Afghanistan.
Pastor Terry Jones, the head of a small church in Florida, called off his plans to burn the Koran after attracting criticism from around the world.
He told NBC: "We feel that God is telling us to stop."
President Obama and the top US commander in Afghanistan, Gen David Petraeus, had warned that Mr Jones's plan would put the lives of US soldiers abroad at risk.