Ken Clarke has declined to apologise after he appeared to suggest that some rapes were less serious than others.
It followed a BBC interview about sentencing proposals in which he referred to "serious rape".
The justice secretary later returned to TV studios to stress that "all rape is a serious crime" and that he had used the "wrong choice of words".
Labour leader Ed Miliband had said he should quit for effectively suggesting there were "other categories of rape".
The row began on Wednesday morning with remarks Mr Clarke gave in an interview with BBC Radio 5 live about proposals under consultation to halve jail terms for people who plead guilty early, including rapists.
Mr Clarke has not apologised for his remarks in general, but he has written to a victim of attempted rape, who also featured on the show, saying: "I have always believed that all rape is extremely serious, and must be treated as such.
"I am sorry if my comments gave you any other impression or upset you."
At present, a defendant entering an early guilty plea can earn up to a third off their sentence. But proposals to halve sentences are outlined in a Green Paper on sentencing in England and Wales.
In an interview with Victoria Derbyshire on BBC Radio 5 live, Mr Clarke argued that pushing for an early guilty plea would stop rapists denying charges and would relieve the victim of "going through the whole ordeal again and of being called a liar" in court.
He dismissed suggestions rapists could be out in 15 months - calculated by halving the average sentence of five years, then allowing for the time someone would be allowed to serve on licence - as "total nonsense".
On being told that the five-year figure came from the Council of Circuit Judges, Mr Clarke said: "That includes date rape, 17-year-olds having intercourse with 15 year olds...
"A serious rape with violence and an unwilling woman - the tariff is longer than that."
When BBC interviewer Victoria Derbyshire interrupted to say "Rape is rape, with respect", Mr Clarke replied: "No it's not, if an 18-year-old has sex with a 15-year-old and she's perfectly willing, that is rape. Because she is under age, she can't consent... What you and I are talking about is we are talking about a man forcibly having sex with a woman and she doesn't want to - a serious crime."
He also said date rapes were included in the figures adding: "Date rape can be as serious as the worst rapes but date rapes... in my very old experience of being in trials [from his time as a practising lawyer]... they do vary extraordinarily one from another, and in the end the judge has to decide on the circumstances."
He later admitted he had confused "date rape" with sex with a willing but underage girl.
BBC home affairs correspondent Danny Shaw said Mr Clarke had, in any case, not been correct to suggest consensual sex with a 15-year-old would be rape - under the Sexual Offences Act 2003 children under 13 are presumed to be incapable of giving their consent to sex. Sex with a 15-year-old would amount to another sexual offence which carries a lower penalty.
In a separate interview with Sky News, Mr Clarke denied he was cutting sentences. He said the proposal applied to every single criminal offence, adding: "Rape has been singled out as an example mainly to add a bit of sexual excitement to the headlines."
At Prime Minister's Questions, Mr Miliband said Mr Clarke's comments had implied there were "serious rapes and other categories of rape" adding: "The justice secretary can't speak for the women of this country when he makes comments like that."
David Cameron told MPs rape was "one of the most serious crimes that there is and it should be met with proper punishment" and the "real disgrace" was that only 6% of reported rape cases ended in a conviction.
He said there was already a plea bargaining system in the UK and the government was only consulting on whether to extend it - and had not yet decided what crimes it should include.
The prime minister said he had not heard the interview but Mr Miliband told him to go back and listen to it, adding: "The justice secretary should not be in his post at the end of today."
Shadow home secretary Yvette Cooper said Mr Clarke "has to go if he stands by these comments because they are absolutely appalling".
In later interviews Mr Clarke said the "most extraordinary spin" had been put on his comments and he had been responding to average sentence figures that were put to him.
But he told BBC political editor Nick Robinson: "My view is all rape is a serious crime and if I have given the impression that is not my view then that is wrong, a wrong choice of words."
Asked if he had been ordered to apologise following the row, Mr Clarke said he had not - and he had not apologised: "I apologise if an impression has been given which is not my view and which I don't think I stated."
He added: "Nobody has had to tell me anything - I have always believed from the days I was a young lawyer that rape is a very serious crime - all rape."
The prime minister's spokesman said it was "clearly regrettable" if anybody had been offended by Mr Clarke's comments but said the PM had confidence in him.
He added there had been some misconceptions about what has been said, but added: "The prime minister has not spent the day watching Ken Clarke interviews."
He said both Mr Cameron and Mr Clarke had thought it important he "went out and clarified the position".