Sky Bet is to pay £1m for "failing to protect vulnerable customers", the Gambling Commission has said.
It did not stop problem gamblers even after they had asked to be banned from its websites, the watchdog said.
Sky Bet chief executive Richard Flint said the firm accepted that it "needed to do more" to stop self-excluded gamblers from opening duplicate accounts.
He added that Sky Bet had tried to return the money in their accounts.
People who feel they are having trouble controlling their gambling can ask betting firms to refuse their service.
But 736 self-excluded Sky Bet customers were able to open and use duplicate accounts, the Gambling Commission said.
In addition, about 50,000 people who had excluded themselves received marketing emails, texts or push notifications through a mobile app.
And 36,748 customers did not get the balance on their account returned after self-excluding.
Richard Watson, Gambling Commission programme director, said: "This was a serious failure affecting thousands of potentially vulnerable customers and the £1m penalty package should serve as a warning to all gambling businesses.
"Sky Bet reported the issues to us quickly, co-operated with us and has taken this investigation seriously."
Matt, a trainee accountant from London, started gambling as soon as he turned 18. He says he lost as much as £30,000.
"It's a 'lad culture' thing," the 22-year-old recently told the BBC's Victoria Derbyshire programme.
"You can't go to the pub on a Saturday afternoon without having the football on the TV, and then obviously there's adverts that come on and you get the urge to bet.
"I couldn't watch a game of football without having a bet on it.
"It was embarrassing to open up [to friends] and say, 'I'm really struggling with this gambling problem.' They were involved in gambling as well," he said.
Mr Flint said Sky Bet had notified the Gambling Commission when the firm had noticed the issue.
"In this case, we didn't make it sufficiently difficult for people to open up duplicate accounts, and for that, we've apologised, it's not good enough. We could and should have done more," he told the BBC. "It is embarrassing for us."
Customers were able to open up duplicate accounts by initially providing false information, then calling up the contact centre with accurate information, and verifying that information with accurate documentation.
"We at that time didn't have appropriate processes in place to check that there wasn't already an existing account that was a duplicate and was self-excluded," Mr Flint said.
The firm also didn't have "sufficient segregation" in its databases which led to inappropriate marketing, he said.
The firm will donate the £1m to charities for socially responsible purposes, Mr Flint said in a statement
"We want to reassure people that we have not made any profit out of this episode," Mr Flint said.
He added that Sky Bet had increased resources "and focus on helping our customers to gamble safely", including a TV and online advertising campaign.
Sky Bet also had "a team of [more than] 60 people monitoring accounts for unusual behaviour," he said.
The Leeds-based operation, which is majority-owned by CVC Capital Partners, runs sites including Sky Vegas and Sky Bingo.
According to figures released in November, more gamblers are opting to "ban" themselves from gambling, with more than a million requests in 2016.
The industry is planning to let people who want to self-exclude use a single website from spring 2018.
The scheme, which all operators will be required to offer, is called Gamstop.
At present, gamblers have to contact every company they hold an account with to ask to be excluded.