Delays in young children's ability to speak and listen properly are leaving them struggling to learn to read and write, Ofsted has warned.
At one nursery visited by inspectors, about 30% of three-year-olds started with a marked speech delay.
Although phonics work helps with early reading, some schools said their pupils were not yet ready for it, Ofsted said.
A focus on speaking and listening and high expectations of pupils often helped tackle the problems, it added.
Ofsted's "Removing Barriers to Literacy" report looked at the factors that stop children from gaining good reading and writing skills.
It said that systematic phonics work, where children are taught to blend letter sounds to form words, is vital in tackling the issue.
But it also suggested that barriers in children's communication skills needed to be overcome as well.
It highlighted how many children who struggled with literacy skills had experienced a "disturbed start to life".
It added: "In one nursery visited, most of the two-year-olds had already had some form of social care intervention by the time they joined the nursery."
Inspectors visited schools mainly in poorer areas for the report. These schools had a clear idea of the reasons why some pupils struggled with literacy.
These included low aspirations in a child's home with few set routines or boundaries for behaviour, and poor attendance.
A reluctance by parents to engage with the school and limited experience of life beyond the immediate community was also highlighted.
'Monitor white boys'
Schools that were good at tackling the impact of a difficult start in life focussed on speaking and listening.
They also had high expectations for their pupils, setting them stringent targets.
Ofsted chief inspector Christine Gilbert said: "Despite some major initiatives in recent years to improve reading and writing, the standards being reached by some groups of children and young people, including those from low-income families, certain ethnic groups and looked-after children, still fall far below that of the rest of the population."
Inspectors recommended that schools closely monitored the progress and attainment of groups who tend to struggle with reading, including white British boys and those from the poorest homes.
They also urged ministers to consider how the progress, especially in literacy, of children from a disadvantaged background, could be reflected in school league tables.
England's Communication Champion for Children Jean Gross said: "Ofsted is right about the importance of speaking and listening skills for literacy.
"These core skills underpin all learning, including reading and writing. Phonic skills are vital for children, but they need to sit alongside good speaking and listening skills.
"If children can't express themselves or understand spoken language, all the phonics in the world won't help them to become good readers."
Schools Minister Nick Gibb said the simple fact was that England was sliding down international league tables in reading and writing.
"Poor literary skills hold children back in all subjects and cause pupils to lose interest and become disillusioned with school. Every child must have a decent grasp of the basics from a young age."
"That's why it is a priority for this government to raise expectations and aspirations for all pupils. Improving literacy levels at all ages is one of the keys to achieving this.
"This is why we are increasing support for phonics teaching and are introducing an age six reading progress check, so that we identify children who are falling behind before it is too late."
This year is the National Year of Communication and a government-backed campaign called "Hello" is being run by The Communication Trust, a coalition of over 35 leading voluntary sector organisations.
It is seeking to raise awareness of speech, language and communication issues among everyone that works with children and young people.