The Co-op Group chairman Len Wardle has resigned, citing "serious questions" raised by the scandal surrounding its former banking chairman, Paul Flowers.
Mr Flowers, 63, apologised after he was filmed allegedly buying drugs.
Mr Wardle, who led the board that appointed Mr Flowers, said he felt it was "right" that he stepped down now.
It has also emerged that Mr Flowers stepped down as a councillor in Bradford in 2011 after "adult content" was found on a computer he had used.
The council said the content was "inappropriate but not illegal".
Questions have been raised about Mr Flowers's appointment as chairman of the Co-op Bank in 2010, and banking regulators will be questioned by MPs on the issue.
The Co-op Group said Mr Wardle had resigned "with immediate effect". He had held the position since 2007 but announced last month that he would retire in May 2014.
In a statement, he said the "revelations" about Mr Flowers had raised "serious questions for both the bank and the group".
"I led the board that appointed Paul Flowers to lead the bank board and under those circumstances I feel that it is right that I step down now," he added.
Mr Flowers, who was chairman of the bank from April 2010 until June this year, was filmed allegedly ordering cocaine and boasting about his use of other banned substances in a video published by the Mail on Sunday.
The paper said he had been filmed buying the drugs days after being grilled by the MPs earlier this month.
The Methodist minister and former Bradford councillor said his actions were "stupid" and "wrong". He has been suspended from both the Labour Party and his church.
Mr Wardle will be replaced by Ursula Lidbetter, who is the Co-op Group's deputy chairwoman and chief executive of the Lincolnshire Co-operative Society.
Mrs Lidbetter said the stories about Mr Flowers were "shocking" but she could not comment further because police were looking into the issue.
West Yorkshire Police are making inquiries and are believed to be trying to obtain the video footage at the centre of the allegations.
Mrs Lidbetter said Mr Wardle, who launched a major review of how the Co-op Group is run before stepping down, had brought forward his retirement because he had felt a "fresh start" had been needed.
The Co-op group must now devise a system of governance "fit for the future", but its "ethos" would remain unchanged, she added.
When Mr Flowers appeared before a committee of MPs on 6 November he appeared to have "no grasp" of "basic" facts about the bank, according to BBC business editor Robert Peston.
He said Mr Flowers had never worked in the banking sector in "any senior capacity", but had been appointed chairman of the Co-op Bank as a result of a "power struggle within the co-operative movement".
The BBC has learned the Treasury Select Committee will not recall Mr Flowers.
But it is understood the committee plans to recall City regulators from the time of Mr Flowers's appointment as chairman.
The scandal has prompted pressure on regulators to introduce tougher checks on candidates for senior banking roles.
Geoff Reid, a Lib Dem councillor in Bradford and retired Methodist minister who has known Mr Flowers for 30 years, told BBC Radio 5 live: "He can be very generous, very gifted as a speaker either in the pulpit or the council chamber, and at the same time he can be subject to incredible lack of judgement.
"I don't think the Co-operative Bank got the political nous they thought they were paying for."
Shadow chancellor Ed Balls' office has said that he will not give back a donation from the Co-operative Group of £50,000 made in March 2012.
Conservative Party chairman Grant Shapps has written to Ed Miliband asking a series of questions about the "conduct and behaviour of Mr Flowers" and suggesting the Labour leader's answers will help determine his "credibility to govern".
Mr Shapps stated the revelations had "shocked and appalled the public" and "raised serious questions about the Labour Party to which you have not yet adequately responded."
His questions include whether Mr Miliband knew why Mr Flowers resigned from Bradford Council, and what advice Mr Flowers had given him on "banking and economic policy".