UPDATE AT 1805:Regrets? He had a few. But, in the first instance, decidedly too few to mention for his opposition critics.
But by this evening - after Mr Swinney's closing speech in the debate - the atmosphere has changed somewhat.
Early talk of a motion of no confidence has not gone away entirely - but seems clearly to be slipping down the agenda.
That is because he closed by repeatedly stressing his regrets - and by apologising twice.
For the critics, the attention now shifts to digging further into the episode. Expect a committee hearing, a demand for the publication of all relevant documents.
Expect the Scottish government to comply fully with any such hearings and demands.
That, they say, is what parliament is for. Ministers are adamant that they acted with honour and in Scotland's interests at all times.
Expect Labour and the LibDems to urge a referral to the two former presiding officers, David Steel and George Reid, on the issue of whether Mr Swinney misled Parliament when he stated last week that he chose not to use a tax power (when, say critics, it was effectively suspended.)
UPDATE AT 1514: In the debate, a strong performance by John Swinney, insisting that the tax power was not in a suitable state of readiness when his party took power in 2007 - while regretting that he had not informed Parliament as the issue developed.
Re the readiness issue, was that because of neglect by the previous Executive, as Mr Swinney implied or was it rather because HMRC had conducted a review in the light of changes to its own IT structure?
The document produced by the Scottish Government dates from 14 May 2007, just after Mr Swinney entered office.
From that, we learn that the Tartan Tax would not be ready until April 2009 - but could be prepared by April 2008 at higher cost.
In other words, it was feasible, at a price, to re-establish the 10 month readiness (from a decision to implement the tax in June 2007.)
In response to Mr Swinney's strong performance, equally strong counter-attacks from Iain Gray, Annabel Goldie and Tavish Scott.
While Mr Scott was fiery, Mr Gray was coldly precise. Mr Swinney's apology, he said, had been grudging, his overall behaviour unacceptable.
As things stand, the SNP will lose tonight's vote - because, in terms of simple arithmetic, the Opposition parties will combine.
But further, in terms of momentum, the finance secretary may have more to do to persuade parliament to let the matter drop.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Unaccountably, it hasn't yet become a "gate".
I suppose "HMRC-gate" doesn't quite cut it. And yet the political challenge to John Swinney has perceptibly strengthened.
The issue? The Scottish government's decision to suspend payments to Her Majesty's Revenue and Customs which led to the effective lapsing of Holyrood's power to vary the standard rate of income tax.
Where are we this morning, ahead of this afternoon's debate? Opposition parties have tabled an amendment to the government motion defending Mr Swinney's actions.
The amendment accuses ministers of "an abuse of power" and of misleading Parliament. Given that this is backed by all the Opposition parties, it will carry.
Labour goes further, noting in a statement that Mr Swinney should "consider his position" - the standard code for "think about quitting."
Offstage, the SNP's opponents are preparing a motion of no confidence in the event that they are not satisfied following the debate.
So is John Swinney a goner? No. His tone this afternoon may shape the next phase.
There were indications from his appearance before the finance committee yesterday that he recognises the need for an element of contrition.
Further, this remains a party political row, a Holyrood issue. It has, I would suggest, minimal salience at this point with the wider public.
Further still, nobody has been harmed. The tax power has not been used. No major party is proposing to use it.
Further still, it is at least arguable that HMRC's demands were unreasonable: that Ministers were right to resist.
I suspect, however, that Mr Swinney appreciates that he should have updated Parliament on the continuing negotiations with HMRC.
That is because decisions on taxation rest with parliament, not an individual minister.