The new first minister turned down a one-to-one meeting with Theresa May to get to a Labour celebration in Cardiff.
Mark Drakeford was to meet the PM at 14:00 GMT on Wednesday, but her office said it had to be "pushed back".
He was offered an early evening slot but BBC Wales understands he declined as he had already agreed to attend a party for leadership supporters.
Rival parties' criticism of Mr Drakeford's decision was branded "utter nonsense" by the Welsh Government.
Mr Drakeford and Mrs May did meet at the Joint Ministerial Committee, which also included the Scottish first minister and Northern Irish officials, where the implications of a no-deal Brexit were amongst the issues discussed.
But the one-to-one meeting would have been the first between the two leaders since Mr Drakeford became first minister of Wales last week.
A source who was at the party has told BBC Wales that Mr Drakeford told them: "Jeremy Corbyn knows about you as you ran a fantastic campaign to get me elected.
"Theresa May didn't know about you but she does now, as I told her I had to come home to meet you all.
"She offered me a one-to-one but I said I had a prior engagement with you all."
Plaid Cymru MP Jonathan Edwards called it "a shocking abdication of responsibility".
"On the day that marked a hundred days to Brexit, the first minister thought it more appropriate to quaff champagne with his mates rather than make Wales' voice heard in Westminster," Mr Edwards said.
"To put your political party before your country is shameful, to put a celebratory party before your country is madness."
Leader of the Conservatives in the Welsh Assembly Paul Davies said it was "not a good start" for Mr Drakeford.
"It is crucial that we have a committed individual who will go that extra mile to ensure that Welsh people are represented at the highest level," he said.
"I appreciate that he had other commitments but surely he could have rearranged his party as discussions with the prime minister should take precedence."
A Welsh Government spokesman said: "This is utter nonsense.
"The first minister met the prime minister yesterday afternoon and also had a detailed discussion with her over the phone earlier in the week.
"A bilateral meeting had been scheduled for earlier in the day yesterday, but the prime minister had to rearrange at short notice due to other commitments.
"The meeting will be rescheduled."
A Number 10 spokesman confirmed Mrs May and the first minister's phone conversation but said Wednesday's one-to-one meeting was not possible because of diary clashes.
He said a meeting was now planned for "early in the new year".
"It didn't happen yesterday but they are going to meet soon," the spokesman added.