There are more than 27,000 empty privately-owned homes in Wales, up more than 40% since 2010, figures show.
The scale of the problem has been described as a "wasted resource" when so many people need affordable homes.
Shelter Cymru said councils have powers to take over some homes to bring them back into use, but do not out of a fear of "getting it wrong".
The Welsh Government said it had given councils £40m to bring empty properties back into use.
Shelter Cymru wants a simpler process introduced - such as in Scotland where councils can take over and auction empty homes.
Homes often remain empty because authorities are unable to contact the owner, the owner cannot afford to renovate when needed or are waiting for the property market to improve.
Empty homes can attract vandalism, drug use and anti-social behaviour, while flats above shops are particularly difficult to return to use.
John Puzey, director of housing charity Shelter Cymru, said: "We know when you reduce empty homes, you reduce crime and vandalism.
"We also know there are a lot of people who are desperately seeking affordable homes.
"So if we can put empty homes in an appropriate place and the right conditions together with people who need them, it's a win-win situation."
Mr Puzey said there was a lot of variation in policy and practises of local councils.
"It's complicated. There are things called empty dwelling management orders and compulsory purchase orders but actually there has been no empty dwelling management order issued in Wales for at least the last three years, if not longer.
"There's these powers there that could be used but not being used. Why is that? Possibly a lack of expertise or concern that they might get it wrong."
Wales has 27,213 empty private homes, figures from Data Cymru for 2018-19 showed, compared to the earliest set of available StatsWales figures - 18,980 in 2009-10 - a rise of 43%.
Allan Morris, a councillor in Newport which has more than 7,000 households looking for affordable homes and 1,199 empty private homes, said: "When you're desperate and see a premises just degenerating it's very difficult to understand why you can't put two and two together, why homeless families can't use those premises.
"It's absolutely frustrating, it's heartbreaking in many cases."
Stuart Ropke, the chief executive of the housing charity Community Housing Cymru said housing associations were "well placed to work with local authorities and the private sector to bring homes back into use".
However, in evidence submitted to a group of AMs on the Equality, Local Government and Communities committee, Community Housing Cymru said: "Even if all 27,000 of the estimated empty homes were returned to use, this would only meet housing need and demand in Wales for just over two years."
The Welsh Government - which aims to build 20,000 affordable homes by 2021 - said: "We have provided councils with £40m to help bring empty properties back into use and we expect the number of empty homes to fall over the next two years.
"We are also improving our Houses to Homes scheme to simplify our grants and loan process and have established a new enforcement team to help councils tackle empty homes."