China's Education Ministry says that about 400 million people - or 30% of the population - cannot speak the country's national language.
Of the 70% of the population who can speak Mandarin, many do not do it well enough, a ministry spokeswoman told Xinhua news agency on Thursday.
The admission from officials came as the government launched another push for linguistic unity in China.
China is home to thousands of dialects and several minority languages.
These include Cantonese and Hokkien, which enjoy strong regional support.
Mandarin - formally called Putonghua in China, meaning "common tongue" - is one of the most widely-spoken languages in the world.
The Education Ministry spokeswoman said the push would be focusing on the countryside and areas with ethnic minorities.
For decades, the ruling Communist Party has promoted Mandarin in an attempt to unite the most populous nation in the world.
But government efforts have been hampered by the sheer size of the country and a lack of investment in education, particularly the rural areas, says the BBC's Martin Patience in Beijing.
The government's policies have also long been contentious, particularly among ethnic minorities, our correspondent adds.
In 2010, there were protests in Tibet about the use of Mandarin in schools. At the time, protesters said it was eroding their culture and language.