More than 60,500 households are on social housing waiting lists in Wales, according to the latest figures.
Shelter Cymru describes it as a "housing crisis" and wants more investment in affordable homes.
Community Housing Cymru says housing associations aspire to build 75,000 homes in the next 20 years.
The Welsh Government said it was a "fundamental priority" and it was committed to building 20,000 new affordable homes by 2021.
Most people in Wales can apply for affordable or social housing and some of those waiting will already be in social housing but might need a bigger property, for example.
At least 60,569 households are on waiting lists but only 20 of the 22 councils have the information so the total figure is likely to be higher.
John Puzey, director of Shelter Cymru, said the number of households waiting was not a surprise.
"I think it shows the gap that's grown between the number of people in Wales who need decent, affordable homes and the number that are actually provided.
"I think the Welsh Government recognises that and they are investing more in housing but we need even more investment if we're going to meet that demand."
He said fewer private landlords were accepting people on benefits and that some people were living in overcrowded conditions while they waited for decent, affordable properties.
"We've had a shortage of affordable homes for some time now. There is an attempt to address that but in the meantime we have got [a housing] crisis.
"Our service helped more than 18,000 people over the last 12 months - that's a record number of people."
He said Shelter Cymru's telephone advice service had thousands of people phoning in "in desperate need of help" while its website had a quarter of a million unique visits from people in Wales.
"That's a huge amount of demand for help and I would say that's very indicative that we've got a housing crisis in Wales."
Gareth James has been waiting since August for a bungalow to be ready in Merthyr Tydfil so his father Maxwell, 68, can move from a nursing home.
The current family home in Ynyswen, Rhondda Cynon Taf is not suitable for Maxwell, who is in a wheelchair after his leg was amputated following an infection.
"You need a stair-lift, you can't get up and downstairs," he said.
"We need a walk-in shower, a disabled toilet - there's nothing like that here. For me it's like my parents have been divorced. One's up here, one's there."
Stuart Ropke, chief executive of Community Housing Cymru, which represents housing associations in Wales, agreed that there was a crisis.
He said while the Welsh Government's commitment to build 20,000 new affordable homes by 2021 was positive, it was not enough.
"It's not meeting the need identified by independent research, but it is an increase on what's been done previously - we're seeing a ramping up of delivery," he said.
"What we now need to do is go further. We've got an aspiration as a housing association sector to build 75,000 in the next 20 years - that's 50% higher than we're currently doing [and] it would represent 90% of the need for social housing in Wales across that period."
The Welsh Government said it knew there was a huge demand for affordable housing.
"We are investing a record £1.4bn in the housing sector, to tackle a range of issues, including building homes, improving existing housing stock and to help prevent people from becoming homeless," said a spokesman.
"We have passed legislation which abolished the right to buy social housing to protect the stock, and incentivise the building of new homes for people in Wales."
He said the Welsh Government had been able to achieve and exceed its affordable homes target during the last term of government, working with housing associations and councils.