Sunday trading laws will be suspended by the government on eight weekends from 22 July during the Olympics and Paralympics.
In his Budget, George Osborne proposed emergency legislation so large shops in England and Wales can trade for more than six hours.
"When millions of visitors come to Britain... we don't want to hang up a closed for business sign," he told MPs.
But campaign group Keep Sunday Special called the plan "profoundly worrying".
And shopworkers' union Usdaw warned its members "vehemently opposed" the idea.
General secretary John Hannett said: "The government's own consultation just last year showed there is no widespread support from either retailers or the general public for change.
"Deregulation would have a very detrimental impact on the lives of millions of shopworkers.
"Shopworkers are entitled for their views to be heard before any decisions of this importance are made."
Analysts, though, predict the move could lead shoppers to spend £200 million more, with shopping comparison website Kelkoo saying almost half of the extra money would be spent in London.
But ministers hope to see the proposal passed by Easter.
Speaking to BBC One's The Andrew Marr Show on Sunday, Mr Osborne said: "We've got the whole world coming to London and the rest of the country for the Olympics.
"It would be a great shame - particularly when some of the big Olympic events are on Sunday - if the country had a closed for business sign on it."
Shadow chancellor Ed Balls said there should be proper consultation and careful consideration before implementing change.
"George's people have told the newspapers this is an experiment to change it for the future, breaking up centuries of tradition," he said.
Church leaders acknowledged the Olympics would be a special time, but said they were likely to oppose the move.
Rev Sally Hitchiner, of St John's Church in Ealing, west London, told the BBC: "We're concerned it could become a precedent, we could lose some of the specialness of Sunday.
"Sunday should be a time for relationships, a time when we put some boundaries on consumerism, so you can go to the park and play football with the kids, and take your mum breakfast in bed."
The Sunday Trading Act 1994 states that shops over 280 square metres in England and Wales are restricted to any six hours of continuous trading between 10:00 and 18:00 on Sundays.
And they cannot open at all on Easter Sunday.
The law also includes measures to protect the rights of shop workers who wish not to work on Sunday.
By temporarily suspending these rules, the government hopes visitors heading to London for the Olympics will take advantage of longer opening hours, boosting flagging retail figures.
The Treasury is expected to closely monitor the suspension's effects and has not ruled out a permanent change.
Tory MP Nadine Dorries predicted that Mr Osborne would "face a barrage of criticism" as a result of the move.
She tweeted: "Arrogant to impose without debate and vote of whole house.
"Is the coalition government secretly implementing an anti-Christian agenda. And if so, who is driving it, Cameron and Osborne or the LDs?"
Last year, Conservative MP Therese Coffey warned a temporary change for the Games might become permanent.
On Sunday she said: "I wouldn't stand in the way of shops being open in the Olympic Park during this time.
"But I just remind people that small, independent stores are allowed to open all day and this is an opportunity to celebrate them rather than having big stores open as well."
But party colleague Mark Menzies said he was "absolutely delighted" the plan was being taken up, saying it would "send out a very powerful message that Britain is open for business".
The MP for Fylde, whose background is in retail, said he had "no desire at this stage" to see longer opening hours beyond August.
The director of the Confederation of British Industry, John Cridland, backed the plan, saying: "I think it's going to do a lot for the spirit of Britain and for the businesses for Britain."
ButKeep Sunday Special said the rumours were "profoundly worrying"and that it totally opposed the "unnecessary" move.
'Undermining the principle'
The Association of Convenience Stores warned the move could set an unwelcome precedent.
Chief executive James Lowman said on Sunday: "Any relaxation, even just for London during the Olympics, would erode our existing, popular, Sunday trading rules."
Mr Lowman added that the temporary concession could open the door for the big retail lobby to press for all sorts of exemptions for other events, "undermining the whole principle" of Sunday trading restrictions.
In Scotland, Sunday trading has long been deregulated with shops deciding their own hours.
In Northern Ireland, legislation introduced in 1997 allows large shops over 280 square metres to open between 13:00 and 18:00 on Sundays.