Amnesty International says it has evidence showing Australian officials paid people smugglers to turn back boats and threatened asylum seekers.
In a new report, the rights group says asylum seekers' lives were put at risk in two incidents in May and July.
In response the Australian government denied the allegations, as it has done ever since they first emerged in June.
The country has a controversial policy of zero tolerance towards migrant boats approaching its territory.
No migrants or asylum seekers are allowed to reach Australia's territories by boat. They have been instead intercepted at sea and turned back or taken to detention facilities in neighbouring Pacific countries.
Amnesty is now calling for a government-appointed investigation, known as a Royal Commission, into Operation Sovereign Borders, the name for Australia's controversial security operation to stop such boats.
In the first incident on 17 May, 65 passengers and six crew were allegedly intercepted by Australian officials and subsequently turned back to Indonesia.
Amnesty, which says it has interviewed all on board, claims that officials handed over US$32,000 (£20,900) to the crew.
The passengers were then transferred from their boat to two smaller rickety boats - one of which sank near an island in Indonesian waters. Passengers managed to swim to safety.
The report also speculates that officials may have paid another crew of people smugglers to return to Indonesia in a second incident on 25 July, based on testimony from 15 asylum seekers.
Their group of asylum seekers and people smugglers was intercepted by Australian officials and held separately on an Australian vessel for several days, before being put on another boat and told to head for an Indonesian island.
The officials allegedly gave two large bags to the smugglers and told the asylum seekers not to open the bags. They also threatened to shoot them if they returned.
Their boat ran out of fuel before reaching the island and they were eventually picked up by Indonesian officials.
In response to the report, the Ministry for Immigration and Border Protection said "people on intercepted vessels are held lawfully in secure, safe, humane, and appropriate conditions by the personnel of the Australian Border Force (ABF) and the Australian Defence Force (ADF)".
"To suggest otherwise, as Amnesty has done, is to cast a slur on the men and women of the ABF and ADF," the statement said.
Australia's 'stop the boats' policy:
- The previous Labor government reintroduced offshore processing in Nauru and Papua New Guinea in September 2012 - a policy it had ended in 2008 - whereby it pays outsourced contractors to operate and provide security at temporary detention camps for asylum seekers on the Pacific islands
- It also reached a deal with Papua New Guinea that any asylum seekers judged to be genuine refugees would be resettled in Papua New Guinea, not Australia
- The current Liberal-National coalition government adopted Labor's policies and expanded them, introducing Operation Sovereign Borders, which put the military in control of asylum operations
- Under this policy military vessels patrol Australian waters and intercept migrant boats, towing them back to Indonesia or sending asylum seekers back in inflatable dinghies or lifeboats
- In 2013 some 300 boats carrying illegal migrants reached Australia. In 2014 the number was just one