Labour AM David Rees and MP Stephen Kinnock have called on their party colleagues in the Welsh Government not to sell land selected for a new prison in Port Talbot.
The UK government announced in March that it plans to build a new prison on undeveloped land.
The site in Baglan is owned by the Welsh Government.
A spokesman for the Welsh Government said a new prison would bring economic benefits to the area.
Mr Rees, the AM for Aberavon, is not opposed to the idea of a prison in the area but said: "This is not the right site.
"It's in the heart of the town, it's in the heart of a residential area tied to an industrial estate which is in the economic enterprise growing area."
Mr Rees said there were rumours that Cardiff and Swansea prisons were likely to close as a consequence.
If that happened, "then this will not create economic opportunities because the jobs from those prisons will be transferred here", he said.
Mr Kinnock said: "The Baglan site is inappropriate, being too close to schools, a care home and residential properties, as well as being a greenfield site."
He added: "The land is part of an industrial park and the enterprise zone. It should be used to help businesses grow and not for a new prison."
Officials at the Welsh Government put forward 20 potential sites for a new prison but the list was whittled down to three before the land on the Baglan Industrial Park site was chosen.
As part of the UK government's £1.3bn commitment to create up to 10,000 modern prison places by 2020, the Ministry of Justice (MoJ) asked Welsh Government officials to provide a list of potential sites in south Wales that met certain criteria.
Of the 20 sites submitted, nine were privately owned and 11 publicly owned.
BBC Wales understands the proposed prison will be designated Category C with space for up to 1,600 prisoners, although that has not been confirmed by the MoJ.
The proposed site is located in one of the three areas designated as part of the Port Talbot Waterfront enterprise zone set up by the Welsh Government last year.
It is one of eight enterprise zones established by the Welsh Government to provide business support including financial help.
Plaid Cymru AM Bethan Jenkins, who represents South Wales West, said the enterprise park is "meant to be dedicated to improving the private sector economy of Port Talbot".
She also called on the Welsh Government to tell the "MoJ that the site in Port Talbot is not for sale".
A firm based next to the Baglan Moors site has made a formal offer to buy it from the Welsh Government.
In its letter to the Economy Secretary Ken Skates, Envases, which produces aluminium packaging such as aerosol sprays, said the bid was made to protect the company and the community's interest.
Another neighbouring firm, Cultech, is concerned about the potential impact a "facility of that size" could have on the local transport infrastructure.
But Sue Plummer, co-founder of the company that manufactures pro-biotic supplements, said: "Personally, I've got no real problem in that it's located very close to us. We would just be able to continue manufacturing as is."
A Welsh Government spokesman said: "We recognise the concerns of local residents but are confident that if the UK government were to secure planning permission for a new prison this would bring economic benefits to the area, both through the prison itself and in the local supply chain."