Rwandan police are arbitrarily detaining vulnerable people picked up on the streets of the capital, according to Human Rights Watch (HRW).
The US-based campaign group says Kigali's reputation for being tidy comes at a price.
HRW says it has documented the illegal detention of thousands, including street children and sex workers.
Rwanda's Justice Minister Johnston Busingye denied there were unofficial detention centres in the country.
Mr Busingye added HRW "seeks to spread falsehood and speculation" but that the allegations will be investigated.
He said the centre where people were being taken was for short stays only before long term rehabilitation.
HRW says in its report, based on research carried out over four years, that people are transported from the capital's streets to a vast former garage which was used to interrogate genocide suspects after 1994.
The organisation says the rounding up of street vendors, beggars, homeless people, street children, sex workers and petty criminals reflects an unofficial policy of keeping those considered undesirable away from the public eye.
HRW says detainees are held there in deplorable conditions for periods ranging from a few days to several months.
Former detainees told HRW of routine beatings for actions as trivial as talking too loudly or not standing in line to use the toilet.
Carina Tertsakian, who put together the report, says the Rwandan authorities should immediately close the detention centre and release all the detainees.
"If some of these detainees are suspected of committing a criminal offense they should be brought to justice according to the law," she added.
Rwanda's justice minister told HRW in 2014, and said again on Thursday, that the place where people were taken provided rehabilitation and avoided "unnecessary incarceration".
"While HRW insists that Rwanda should charge drug addicts and other criminals with serious crimes that carry jail terms, the country has instead chosen to focus on rehabilitating and reintegrating them to offer the chance for a better life," he said.