A warning that there were "too many near misses in which railway workers have had to jump for their lives" was issued only three months ago.
Now the Rail Accident Investigation Branch (RAIB) is looking into how two workers were killed by a train near Port Talbot on Wednesday.
The pair were struck near Margam by the Swansea to London Paddington train at about 10:00 BST and a third person was treated for shock.
Unions have demanded answers.
Network Rail has said it is cooperating with the British Transport Police and Rail Accident Investigation Branch.
The RAIB's annual report in April issued the warning and said every near miss should be regarded as a "failure" to deliver safety.
An incident in London in November 2018 was the first in about four years where a track worker was killed.
That was the only death of a track worker on mainline railways - excluding trams, undergrounds and private lines - included in the RAIB's annual report for 2018.
There were 6,661 injuries, of which 164 were major.
Prior to this:
- A worker was killed in Newark, Nottinghamshire in 2014, according to Network Rail
- From 2009-14 there were four Network Rail fatalities
- From 2014-19 there were no Network Rail fatalities and two contractor deaths
Rail investigator Simon French said the risk to workers had fallen since the organisation was created 13 years ago.
But, in the RAIB report, he said: "I am concerned that, despite much effort and many initiatives, we are not seeing the hoped-for improvements in safety for track workers.
"Every near miss, however caused, should be viewed as a failure of the system to deliver safety."
Unions have said Wednesday's tragedy should not have been allowed to happen.
Manuel Cortes, Transport Salaried Staffs' Association general secretary, said: "It's too early to speculate about what has happened here but clearly something has gone badly wrong.
"There must now be a full investigation because it is simply not acceptable that in the 21st Century people go out to work and end up losing their lives.
He added: "Safety on our railways is paramount and sadly, as today's tragic events show, it can never be taken for granted."
Mick Cash, general secretary of the Rail, Maritime and Transport (RMT) union, described the deaths as "shocking".
"RMT is attempting to establish the full facts but our immediate reaction is that this is an appalling tragedy and that no-one working on the railway should be placed in the situation that has resulted in the deaths that have been reported this morning," he said.
"As well as demanding answers from Network Rail and a suspension of all similar works until the facts are established, the union will be supporting our members and their families at this time."
Martin Frobisher, safety director at Network Rail said: "We take all safety incidents, especially near misses with track workers, extremely seriously.
"Huge strides have been made in railway safety over recent years. Britain has a good safety record and Network Rail expects that all our people should come home safely every day.
"The tragic accident in south Wales has been shocking for all our people and we will make further improvements," he said.