Ed Miliband has called David Cameron a "dodgy prime minister surrounded by dodgy donors", in a row over party grandees with Swiss bank accounts.
Mr Cameron hit back by accusing the previous Labour government of being the "friend of the tax dodger".
The row comes amid allegations HSBC's Swiss private bank may have helped wealthy clients avoid or evade tax.
The Labour leader singled out Lord Fink in his attack - but the peer has challenged him to withdraw the remark.
Lord Fink, who stepped down as Conservative co-treasurer in 2010 and has given the party £3m, is among those named in documents seen by the BBC's Panorama programme, as is Labour donor Lord Paul.
At Prime Minister's Questions in the Commons, Mr Miliband called on Mr Cameron to say what steps he was going to take "to find out about the tax avoidance activities of Lord Fink".
'Local bank account'
Reading out a statement outside Conservative Central Office, Lord Fink said Mr Miliband's allegations, made under the protection of Parliamentary privilege, were "untrue and defamatory".
He said he had opened an HSBC account in Switzerland when he worked there for four years because he needed "a local bank account to do simple things like receive my Swiss franc salary and pay grocery bills".
He said he chose HSBC because he already banked with them in the UK.
He accused the Labour leading of "playing the man not the ball", suggesting "this had become something of a habit for him."
He added: "I challenge Ed Miliband to either repeat his allegations outside the House of Commons or perhaps to withdraw them publicly."
But Labour sources said "serious allegations" about Lord Fink's "complex arrangements to minimise tax" had been made in The Guardian and he had still "not justified the reasons why he made these arrangements".
In a separate development, Lord Paul said he had held an account with HSBC for quite some time in Switzerland. There was "nothing unsavoury" about it, he said.
He has been a Labour donor and a Labour peer, but now sits as an independent in the House of Lords.
The Guardian published names of Swiss account holders. It reports the Conservatives have raised more than £5m from HSBC clients with Swiss accounts.
Labour have benefited, the Guardian says, from cash and gifts in kind worth well over £500,000 and received a loan for £2m.
Holding a Swiss bank account is not illegal and does not prove either tax evasion or tax avoidance.
The individuals named may have held the accounts for a wide variety of reasons and there is no suggestion they did anything wrong.
In a heated clash at Prime Minister's Questions, Mr Miliband asked Mr Cameron how he would explain the "revolving door between Tory Party HQ and the Swiss branch of HSBC".
He said none of the Labour donors named in the list had "given a penny on my watch" and claimed Mr Cameron was "up to his neck in this".
Mr Cameron said Conservative donors did not choose the party's policies and candidates - unlike Labour who, he said, were controlled by trade union donors.
Shouting across the despatch box at the Labour leader, he said: "The only reason he is sitting there is that a bunch of trade union leaders decided he was more left wing than his brother."
In a separate exchange Mr Miliband said: "You took the money, you gave a job to the head of HSBC and you let the tax avoiders get away with it.
"There's something rotten at the heart of the Conservative Party and it's you."
Mr Cameron replied: "For 13 years they [Labour] sat in the Treasury, they did nothing about tax transparency, nothing about tax dodging, nothing about tax avoidance.
"This [present] government has been tougher than any previous government. That's why they [Labour] are desperate, that's why they are losing."
He said Labour had welcomed the appointment of former HSBC chairman Lord Green as a trade minister, and had even held meetings with the peer as recently as 2013.
Conservative MP Zac Goldsmith is among those being linked to a HSBC Swiss bank account.
In a statement, the Richmond Park MP said: "To be clear, my very numerous family members around the world who are beneficiaries of the same Trust do not own or control it, but like me, they receive income from it.
"I have never had a Swiss bank account, and do not control any Swiss bank accounts. I have never sought or been given tax minimisation advice by HSBC, directly or indirectly. The media commentary around this is therefore wrong."
He said he had paid taxes in full on his income and the "vast majority" of his donations to the Conservative Party were "notional" and "relate to use of offices over the past 10 years".
'Change of behaviour'
Meanwhile, the boss of HM Revenue & Customs has rejected claims from MPs that she should have done more to share "relevant information" about HSBC's activities with David Cameron before he nominated Lord Green to be a Conservative peer.
Lin Homer said the agency had a "very limited responsibility to give a limited amount of information" about individuals' personal tax affairs in relation to potential honours.
"I just do not think what you are asking HMRC to do would be possible in all circumstances," she told the Public Accounts Committee.
"Your assumption is based on an assertion that there is something wrong going on in HSBC UK. I am not making any comment on that assertion."
The bank helped 100,000 individuals - 7,000 UK nationals - with tax affairs.
The BBC's business editor Kamal Ahmed said the HMRC was set to expand the scope of its inquiry and was likely to meet the police and the Serious Fraud Office next week.
HMRC was passed a hoard of documents in 2010 from France about clients of HSBC's Swiss operation, but has said it could not share information it had about the bank because of an international agreement.
Offshore accounts are not illegal, but many people use them to hide cash from the tax authorities. And while tax avoidance is perfectly legal, deliberately hiding money to evade tax is not.
HSBC's Swiss accounts in numbers
clients with Swiss bank accounts
$118bn total assets held in Swiss accounts
11,235 clients from Switzerland held $31.2bn
9,187 clients from France held $12.5bn
7,000 clients from UK held $21.7bn