Less than 1% of fathers take advantage of additional paternity leave up to a total of 26 weeks, figures suggest.
The TUC study for 2011/12 found 1,650 out of 285,000 partners took the leave at the statutory rate of £136 a week.
The trade union organisation said this was because they could not afford to live on such a rate, which is not normally supplemented by employers.
The Department for Business said a new system of shared parental leave would be introduced from 2015.
The system which allows the father, or husband or partner of a child's mother, to take up to 26 weeks additional paternity leave - and receive additional statutory paternity pay - was only introduced in April 2011 and the TUC figures are based on its first year.
Fathers and partners had already been entitled to two weeks ordinary paternity leave.
They can only claim the extra weeks in their child's first year if the mother returns to work before taking her full maternity leave entitlement. The additional paternity leave can also be taken when adopting a child.
TUC general secretary Frances O'Grady said: "A good gift for fathers this Sunday would be for ministers to increase statutory paternity pay rates and for employers to top it up for longer, so that new dads can spend more time with their children.
"Poor levels of financial support are preventing new dads from taking extra time off and are particularly affecting low-paid fathers who simply cannot afford to take leave.
"Extending paternity pay from two to six weeks and paying a better statutory rate would make a massive difference, as has been shown in other countries."
The TUC said in contrast, the first two weeks of paternity leave was taken by nine out of 10 fathers. But the difference is that although the statutory rate is the same - £136 a week - many employers typically choose to top this up to full pay throughout that short period of leave.
A spokesman for the Department for Business Innovation and Skills said: "The current system for parental leave is old-fashioned and too rigid.
"This is why we are introducing a system of shared parental leave from April 2015 so that fathers can take more leave if they want to in the early days of a child's life.
"We want to challenge the myth that it is the mother's role to stay at home and care for children.
"Men will be more able to get better involved with the caring of their children from the earliest stages and evidence shows this sort of involvement has significant benefits for children's educational and emotional development in later life."
A separate report by NatCen Social Research and the University of East Anglia has found that men with a partner and children at home work longer hours than other working men.
Three in 10 men in this family situation worked 48 hours a week and one in 10 worked more than 60 hours.