Millions of Pakistanis live in a "human rights-free zone" in the country's north-west, Amnesty International says.
Residents of tribal areas face Taliban abuse and get no protection from the government, the rights group alleges.
In a report, it says the Taliban secured their rule by killing elders and torturing teachers and aid workers.
A Pakistani foreign office spokesman rejected Amnesty's findings, saying his government was "fully committed" to improving human rights in tribal areas.
"We are not denying that there are problems there," spokesman Abdul Basit told reporters in Islamabad, but he said the government was "sparing no effort" to ensure people's rights were protected.
The 130-page Amnesty report, As if Hell Fell on Me, was based on nearly 300 interviews with residents of the Federally Administered Tribal Areas (Fata) and the surrounding areas.
It says more than a million people have been displaced by fighting between the Pakistani military and the Taliban in tribal areas bordering Afghanistan.
The report also says available information suggests that at least 1,300 people were killed in the conflict during 2009.
"Nearly four million people are effectively living under the Taliban in north-west Pakistan without rule of law and effectively abandoned by the Pakistani government," said Claudio Cordone, Amnesty's interim secretary general.
The report quotes a teacher, who fled the Swat valley with his family in March 2009, describing how the Taliban operated.
"[The Taliban] took over my school and started to teach children about how to fight in Afghanistan. They kicked out the girls from school, told the men to grow their beards, threatened anybody they didn't like."
The teacher said the government failed to protect them.
"What's the point of having this huge army if it can't even protect us against a group of brutal fanatics?"
Last year the Pakistani army declared the Swat valley to be free of militants after completing an anti-Taliban operation in and around Mingora, the main city in the valley.
Pakistani military spokesman Athar Abbas dismissed the report, describing it as "factually incorrect". He said that tribal elders supported the army's action to bring stability, particularly to the Swat region.
"Local journalists as well as foreign journalists who have been visiting Swat, talking to the people, the elders, the notables, the locals there, they have endorsed the contribution of the military in bringing normalcy and protecting the people," he told the BBC.
Amnesty has documented what many civilians in north-west Pakistan have often been scared to openly say, the BBC's Aleem Maqbool reports from Islamabad.
The report talks of systematic human rights abuses by the Taliban and accuses militants of increasing the likelihood of civilian casualties by dispersing themselves among civilians during clashes with government forces.
But it also accuses the Pakistani army of not doing enough to avoid civilian casualties in its operations against militants, and the government of neglecting the basic needs of the millions of people living in the frontier regions close to Afghanistan.
Deal with that, Amnesty says, and many of the conditions that have led to the considerable unrest in these areas, would be removed.
The group has appealed to both the Taliban and the Pakistani government to end human rights abuses in north-west Pakistan.
It has also called on Islamabad to reform the Pakistani constitution, which excludes the Fata from the legal and parliamentary system of the country.