Labour has called for Culture Secretary John Whittingdale to withdraw from press regulation decisions after news of his relationship with a sex worker.
Four newspapers knew about the relationship, which ended in 2014, but decided not to publish the story.
Shadow culture secretary Maria Eagle said it had left him "vulnerable" to pressure from the press.
Mr Whittingdale, who says he had not known the woman was a sex worker, said it had not affected his decisions.
Downing Street said Mr Whittingdale was "a single man entitled to a private life" and had the full confidence of the prime minister.
As culture secretary, Mr Whittingdale's job is to regulate newspapers and he is currently overseeing a new regulatory framework under consideration in the wake of the Leveson Inquiry into press standards.
Ms Eagle said the Conservative minister should "recuse" himself from any further involvement in decisions relating to the inquiry.
Mr Whittingdale told BBC's Newsnight: "Between August 2013 and February 2014, I had a relationship with someone who I first met through Match.com.
"She was a similar age and lived close to me. At no time did she give me any indication of her real occupation and I only discovered this when I was made aware that someone was trying to sell a story about me to tabloid newspapers. As soon as I discovered, I ended the relationship.
"This is an old story which was a bit embarrassing at the time. The events occurred long before I took up my present position and it has never had any influence on the decisions I have made as culture secretary."
At the time of the relationship, Mr Whittingdale was chairman of the Commons Culture, Media and Sport select committee.
He was made culture secretary in May 2015, more than a year after the relationship ended.
Downing Street told Newsnight it was not aware of Mr Whittingdale's relationship before his appointment to the cabinet job.
David Cameron first knew about the story "about ten days ago", a spokesman added.
Mark Easton, BBC home editor
New ministers routinely sit down with their department's most senior civil servant, the permanent secretary, to talk through anything that might cause a problem.
We don't know whether John Whittingdale told Sue Owen, at the Department for Culture, Media and Sport CMS, about the embarrassing personal story that had been hawked around Fleet Street for over a year.
Given that Downing Street says it only learned about it 10 days ago, if he did say anything, it would seem it was not passed on.
Having ended the relationship 15 months earlier, Mr Whittingdale may argue there was no longer anything that could look like a conflict of interest, nothing that might appear to place him under an improper obligation.
Four newspapers - the People, part of the Mirror Group, the Mail on Sunday, from the Daily Mail group, the Sun, part of News UK, and the now online-only Independent - had investigated the claims against Mr Whittingdale.
They did not publish the story, concluding it was not in the public interest.
Tom Newton-Dunn, the Sun's political editor, told Sky News: "We didn't publish the story because, quite frankly, there is no story.
"He is a single man, a divorced man. You can pretty much have a relationship with anyone you want."
However, James Cusick, a former Independent reporter who looked at the story for five months, said the public had a right to know the story as Mr Whittingdale was making decisions on how newspapers and the BBC could conduct themselves.
He said: "If this individual is making these decisions... you have a right to know about this man's private life and whether there is something in it he is trying to hold back from you."
BBC political editor Laura Kuenssberg said the question for Mr Whittingdale was not about his relationship, but about his role in regulating the press when the newspapers had a story about his private life.
Who is John Whittingdale?
- Secretary of State for Culture, Media and Sport since May 2015
- Chairman of the House of Commons Culture, Media and Sport select committee for a decade
- Conservative MP for Maldon since 1992
- Educated at Winchester College and studied economics at University College, London
- Has two children, and enjoys television, films and music
Shadow culture secretary Ms Eagle accused Mr Whittingdale of changing his stance over the Leveson Inquiry, having previously supported a second stage - which was set to look into ties between newspapers and the police.
Earlier this year, it was reported in the Times that the second half would be "quietly" shelved.
Ms Eagle also called on him to clarify why he had not yet implemented key recommendations made by the inquiry, including some involving the awarding of costs against publishers in legal disputes.
The government says it has not decided whether to pursue a second stage of the inquiry. A No 10 spokesman said: "We've always said that criminal investigations and legal processes relating to Leveson need to have fully concluded before we consider part two."
Shadow international development secretary Diane Abbott said it had long been her view that Mr Whittingdale's close relationship with the family of Rupert Murdoch - whose company News UK publishes the Sun - made him an "unsuitable person to deal with press regulation".
However, shadow foreign secretary Hilary Benn said Mr Whittingdale "ought to get on and do his job", including pressing ahead with the second stage of the Leveson Inquiry.
Media commentator and former newspaper editor Roy Greenslade said newspapers would have been wary about covering such a story in the aftermath of the Leveson report on press standards.
"I think it is a bit much to castigate the newspapers for doing the right thing for once," he added on Newsnight.
Journalist and media commentator Steve Hewlett said there was not "a single shred of evidence" that any form of direct contact, influence or attempted influence had actually happened but there remained "the question of perception".
However, Brian Cathcart, from the Hacked Off campaign group, insisted Mr Whittingdale was "compromised".
He said the newspapers "stocked up" the story for future use and as a way to possibly put "pressure on him".