The first "tangible" complaint about Carl Sargeant behaving inappropriately towards two women was made in the weeks before he died, an inquest has heard.
The former communities minister, 49, was found dead in November 2017, four days after he was sacked by First Minster Carwyn Jones.
The Labour AM from Flintshire had denied allegations of inappropriate behaviour towards women.
He left a letter telling his family "I have failed you", his inquest heard.
Matthew Greenough, senior special advisor to the first minister, told the inquest in Ruthin, Denbighshire, there had been two separate allegations of inappropriate behaviour in 2016, one via an anonymous letter received by Carwyn Jones saying Mr Sargeant was "not fit to be around women".
The first minister also made Mr Greenough aware of "an incident involving a woman".
However, it was in October 2017 that the special advisor said there was a first "tangible" complaint about Mr Sargeant, made to him via a phone call from David Hagendyk, former general secretary of the Labour Party.
He said Mr Hagendyk told him allegations had been received that Mr Sargeant had behaved inappropriately towards two women at a political event.
The inquest heard the women were known as Miss A and Miss B.
Mr Greenough told the inquest he spoke to the first minister about the allegations, and then contacted Miss A. She divulged the details of the allegations to him. He said she "wanted to consider her options" on how to progress the matter.
He told the inquest Miss A asked "that I would only discuss the matter with the first minister and nobody else".
He said he relayed the information of the conversation to Mr Jones. The first minister, he said, believed that unless the complainants specifically wanted to make their complaint official, or put it in writing, then it was not a formal complaint.
The next day Mr Greenough contacted Miss B, and discussed allegations of inappropriate behaviour. Both alleged incidents involving Miss A and Miss B took place in the summer of 2017.
Both their names were later given to the first minister, on 26 October.
The inquest heard that around this time, the first minister instructed him to contact another complainant, Miss C.
Mr Greenough said the first minister still believed something had to be put in writing to treat it as a formal complaint.
By 1 November, a written statement was received from Miss A. No letter came from Miss B, but she was happy for it to be referenced in the statement by Miss A. No written complaint was received from Miss C.
Prior to these events, Mr Sargeant's name had been in the frame for a job move when a cabinet reshuffle was being discussed, possibly to leader of the house.
Coroner John Gittins said: "I struggle a little with this in that there are these rumours... and yet his name is in the frame potentially for what is a very senior and responsible role."
Mr Greenough replied that the first minister had "never been interested in rumour". He said rumours in politics are rife, and you simply can't "operate that way".
Mr Greenough, quoting from his hand-written notes from a meeting between Carwyn Jones and Mr Sargeant on 3 November, said Mr Sargeant was upset.
"He shook his head as Carwyn was speaking and said he didn't recognise what he was being told," Mr Greenough told the inquest.
He said Mr Jones advised Mr Sargeant the matter could become public and that he should be "prepared" if that happened. He also advised Mr Sargeant it was a "difficult time".
Mr Greenough said the first minister offered advice to Mr Sargeant, including to work with the party.
During the meeting where the sacking occurred, Mr Jones reminded Mr Sargeant that he had been warned about his conduct in 2016, according to Mr Greenough
He added the reshuffle was treated differently from previous ones, because it "wasn't appropriate" to brief the media on new appointments as the Welsh Government normally would.
The coroner said later that day the "stakes were raised" when Mr Sargeant tweeted.
Mr Greenough said the tweet was "along the lines" of his sacking being an agreed position and the door to government being left open in the future.
It was decided to correct the inaccuracies "through a short statement," he said.
Mr Greenough said he was surprised by Mr Sargeant's tweet because "at that point there was a lot of information in the public domain about the reshuffle... the focus of the media reporting at that point was about the reshuffle as a whole".
Phone records examined
Earlier, the inquest heard telephone records are to be examined after it emerged a phone call made by Mr Sargeant just hours before he died on Tuesday, 7 November 2017 did not come from his usual phone.
On Monday, future generations commissioner Sophie Howe, who once worked as a special advisor to Mr Sargeant and was a friend, had told the inquest she received a call from Mr Sargeant at about 07:30 BST on the morning he died and believed she may have been the last person to speak to him.
On Tuesday morning, counsel for the first minister requested Ms Howe's phone records be examined as there was no log of the call being made from Mr Sargeant's usual number.
Cathy McGahey QC said: "Mrs Howe's evidence yesterday gives a good indication he called her and yet they can't find the phone."
Ms McGahey said it was "overwhelmingly likely" there was a missing phone.
Coroner John Gittins agreed Sophie Howe's records could be looked at.
'Emotional health not right'
Leslie Thomas QC, representing the Sargeant family, also requested audio of interviews given by Carwyn Jones on Monday, 6 November 2017 be played, saying it was relevant because it may show "what may have been operating on Mr Carl Sargeant's mind on the morning of the 7th".
The inquest heard evidence from Welsh Government minister Ken Skates, who said he believed Carl Sargeant had been suffering mental health issues for a long time.
Mr Skates's statement was read by the coroner. In it, Mr Skates said Mr Sargeant "did not explicitly tell me" he had a mental health condition, but he worked it out for himself.
Mr Skates said: "As someone who has had mental health problems in the past myself, I came to believe Carl's emotional health was not right."
He said Mr Sargeant began to "ask me more and more" about his own experiences.
He also said Mr Sargeant had taken up knitting to help with "mindfulness".
First minister 'furious'
The inquest heard a statement from Steven Jones, former special advisor and head of media to Carwyn Jones.
He said he received a text message from Carl Sargeant on Friday 3 November 2017, following the re-shuffle, which said: "Bad shit."
He said when they spoke later, Mr Sargeant told him allegations of improper conduct had been made, but he did not know what the allegations were, or who made them.
Mr Sargeant had said Carwyn Jones told him "not to go public" over the allegations, and that they would handle it over the weekend. He took "they" to mean the Welsh Government, Steven Jones told the inquest.
On 8 November - the day after Mr Sargeant's death - Mr Jones said he went to Carl Sargeant's constituency office, and Ken Skates was present. Mr Jones said he asked Mr Skates about Carwyn Jones' reaction to Mr Sargeant going public.
He reported Mr Skates had said that Carwyn Jones was "furious", using an expletive for emphasis.
The inquest also heard Mr Skates emailed the coroner, John Gittins, twice last November, disputing the accuracy of Steven Jones's statement.
He also believed he would not have used expletives in the conversation with Steven Jones, which he said was not in his nature.
'Easy to work with'
Mr Sargeant's senior private secretary Imelda Francombe described him as "very easy to work with".
She said he had confided to her some of his mental health issues in late 2016 or early 2017, saying he had not been taking his tablets of late and felt "really blue".
She had spoken to her line manager Peter Greening about that conversation after he made comments about Mr Sargeant's diary not being very full of commitments, as she did not want a negative impression of Mr Sargeant to be held.
After the reshuffle, he had texted her saying: "Sorry I let you guys down. You were magic."
Asked by the coroner whether she ever witnessed any evidence of Mr Sargeant displaying inappropriate behaviour, she replied: "Absolutely, unequivocally, no."
Asked whether she had heard any rumours, she replied: "Never."
Peter Greening confirmed the conversation with Ms Francombe had taken place but said he had not been asked by anyone to speak to her.
He was shocked to hear Mr Sargeant had health issues - he was "pretty certain" the word depression was not used - because the AM was "very outgoing, sociable".
He said it changed the way he thought about Mr Sargeant's diary and did not share the conversation with the first minister.
Mr Greening said he saw Mr Sargeant leaving the cabinet meeting following the reshuffle that saw him lose his post.
"He walked past me... I couldn't tell his demeanour, whether he was upset... I really couldn't tell," he told the court.
He also said there had been no support in place for ministers being removed that he knew of, adding: "I think there are lessons to learn."
Questioned by Leslie Thomas QC, Mr Greening said he had no qualifications in terms of employee well-being, and no professional experience of dealing with mental health.
Pushed on whether his comment about Mr Sargeant's diary being "light" was a negative judgement, he said: "I would never pass judgement on a minister's performance." He said it was not his role to criticise ministers or exercise any negativity on a minister's performance.
Asked what it meant that Mr Sargeant's diary looked light, he said: "It means that his workload is not as extensive as some of his cabinet colleagues." He added that a diary could be "light" because of either the minister, or his team.
Speaking about Mr Sargeant's illness, Mr Thomas said: "You did not appraise yourself of the nature or extent of the mental illness that may have been affecting Carl. Is that fair?"
Mr Greening replied: "That's fair."
Mr Thomas also asked if Mr Greening had considered going back to speak to Ms Francombe about Mr Sargeant's mental health once he knew Mr Sargeant was to lose his job in the reshuffle, which he could have done without breaking confidence.
Mr Greening said: "No."
On Monday, the court heard details of Mr Sargeant's death, including that he was found hanged at his home in Connah's Quay by his wife Bernadette. The letter was found next to his body.
It also heard evidence from former minister Leighton Andrews, who called Carwyn Jones' behaviour following Mr Sargeant's sacking "irresponsible".
Sophie Howe described to the court how his health began to deteriorate in the summer of 2017, several months before his death.
Mr Sargeant had been treated for depression on and off for a number of years prior to his death.
He had told her after the sacking: "I'm in a dark place."
The hearing was adjourned until Wednesday.
Information and advice
If you or someone you know is struggling with issues raised by this story, find support through BBC Action Line .