The government plans to launch a new workers' watchdog to take over protecting the rights of UK workers.
The new body will be responsible for tackling modern slavery, enforcing the minimum wage and protecting agency workers.
Currently these tasks are spread across three different bodies.
It said it was looking at extra curbs targeting the garment sector including banning sales if brands' behaviour does not improve.
The government is looking at these specific measures after reports of serious problems in the industry, the Department for Business, Energy and Industrial Strategy said.
One of the leading online fashion retailers, Boohoo, came in for heavy criticism after concerns were raised that workers at its Leicester suppliers were being underpaid and not being protected against Covid-19 during the pandemic.
It was alleged last year that some factories in the UK working for the firm were paying staff as little as £3.50 an hour and had working conditions that did not meet Covid guidelines.
After an internal review, it cut its supplier network from an estimated 500 firms to under 100 following the allegations.
Boohoo said earlier this year that if it were to discover any suggestion of modern day slavery it would "immediately disclose this to the relevant authorities."
The government said it was looking into either creating a Garment Trade Adjudicator to investigate companies' supply chains, or it could extend an agricultural licensing scheme.
Under that scheme, businesses who provide agricultural workers must have a license, and can be inspected to make sure they are meeting employment standards.
If brands' behaviour doesn't improve, the government warned it could introduce harsher measures, including bans on goods made in factories where workers have been underpaid.
Business minister Paul Scully said: "This government has been absolutely clear that we will do whatever we can to protect and enhance workers' rights.
"The vast majority of businesses want to do right by their staff, but there are a minority who seem to think the law doesn't apply to them. Exploitative practices like modern slavery have no place in society."
He added that the new watchdog would enable the government to "take action against companies that turn a blind eye to abuses in their supply chains".
The government also says the new watchdog will make it possible for vulnerable workers to get the holiday pay and statutory sick pay they are entitled to, without having to go through a lengthy employment tribunal process.
Previously, the rights of UK workers were split into three bodies - the Gangmasters and Labour Abuse Authority, the Employment Agency Standards Inspectorate and HMRC's National Minimum Wage Enforcement.
The new workers' watchdog will provide guidance to businesses on best practice, complementing existing work already being carried out by independent bodies like the Advisory, Conciliation and Arbitration Service (Acas).
It will also have enforcement powers and will continue the government's existing Naming and Shaming scheme run by the Low Pay Commission, which in December 2020 named 139 businesses for failing to pay the national minimum wage.