Growing up in chaotic homes is leaving some children "actively harmed" and unprepared for starting school, Education Secretary Michael Gove warns.
Mr Gove acknowledged some young children started school in nappies and unable to speak in full sentences.
He said while it was important to "exercise a degree of restraint" with state intervention, there were cases where the state must intervene.
His comments came in a speech to the Politeia think tank in London.
He was asked by Graham Allen, Labour MP for Nottingham North, about his stance on early intervention and how prior attainment affected pupils' education.
Mr Allen said: "I work with my secondary heads and they say that the raw material they are sent at age 11 is not good enough, therefore I go on to my primary heads and they say children arrive at school unable to read or recognise the difference between a letter or a number, they arrive sometimes in nappies, unable to speak in a sentence."
Evidence demonstrates that there is a need to intervene "much earlier" in life, Mr Allen said, so that the "raw material is better" to work with.
Mr Gove responded: "I think it's true.
"One of the reasons why we've shifted the focus of the academies programme to take in primary schools as well as secondary is not because we believe it's done improving secondary schools, far from it, but because there are many secondary schools doing a good job which are inherently impeded by the quality of the education offered in primary schools, and you are absolutely right, there are significant numbers of children who, because of their home environment arrive at school simply incapable of learning.
"As you can probably tell, I think it's important that we exercise a degree of restraint when we think about state intervention, but I think we both agree that there are a group of children for whom the state has to intervene because they will grow up in circumstances so chaotic that it's not just a case that they are neglected, it is the case that they are actively harmed by the failure to be in a nurturing environment where their brain can develop and where they can learn the sorts of habits which allow them to not just succeed academically at school but are effectively socialised."
In his speech to Politeia, Mr Gove said it was crucial to intervene in failing schools, particularly in poorer neighbourhoods.
"We need to move even faster, extending the frontiers of opportunity, providing more excellent school places for more children than ever before.
"And I am determined that we concentrate our efforts particularly on the children in greatest need - those in the weakest schools - overwhelmingly in the most disadvantaged areas."
Mr Gove said there were hundreds of under-performing primary schools and said he would be writing to MPs asking them to support his plans to turn poor primaries into academies.
"Children in those schools are not receiving the education they deserve. And today I want to invite the MPs in those communities to work with me to open up the education system in their areas to the new providers who can raise standards."
In total, 310 poor primaries have now become, or are in the process of becoming academies, he said.
Schools are considered to be falling short if fewer than 60% of pupils reach Level 4 in their national curriculum tests, known as Sats, taken at the end of primary school.
The latest figures show that in 2011, 1,310 primary schools fell below the standard.
Shadow education secretary Stephen Twigg said: "Michael Gove's education policies are putting standards at risk - 10,000 teachers have left the profession, we've seen the biggest cuts to education budgets since the 1950s and thousands were let down by his GCSE fiasco.
"While Labour would deliver for all young people, the Tories' divisive and narrow approach is not fit for the challenges of the 21st century."