Teachers' "entrenched" attitudes could lead them to write off Gypsy, Roma and Irish Traveller children, "enabling prejudice to continue", MPs have said.
Throughout her time in education, one woman told the Women and Equalities Committee, teachers had said: "You're a Gypsy - are you going to leave school?"
These children were underachieving, it said, and more likely to drop out or be temporarily or permanently excluded.
And almost half of them were missing at least a month of lessons every year.
The MPs' report said: "Some Gypsy and Traveller children are taken out of school as early as the end of primary school, some persistently do not attend and some never register at school at all.
"Where these children end up is unclear, although we have heard of successful and unsuccessful home education, children starting work at as young as 10 years old, and children who simply stay at home without any formal education.
"Local authorities must serve a notice on parents they believe are not educating their children."
But the MPs acknowledged that many local authorities, despite their best efforts, were unable to reach children who may be missing from education.
"Consequently, there is very little, if any, oversight of how Gypsy and Traveller children are being educated at home," the report said.
And, with three out of four of their families living in settled accommodation, most of them were not missing lessons because of time spent actually travelling.
"Parents have told us that they take their children out of education for reasons ranging from bullying that they experience in school, schools not taking their children's needs into account, and not seeing the relevance of education, to, most worryingly, feeling that schools do not educate their children in a way that they would find acceptable," the MPs said.
Maria Miller, who chairs the committee, said these children had been comprehensively failed by policy-makers and public services.
She said: "The government must stop filing this under 'too difficult' and set out how it intends to improve health, education and other outcomes for these very marginalised communities who are all too often 'out of sight and out of mind'."
Association of School and College Leaders general secretary Geoff Barton urged all agencies to redouble their efforts over the education of Gypsy, Roma and Irish Traveller children.
But he said the reports of teachers making inappropriate remarks based on assumptions about pupils' backgrounds were "anomalous and does not reflect the normal practice in schools, which is highly attuned to promoting an inclusive environment to children of all backgrounds".
"Similarly, schools have robust policies in place to prevent bullying and discrimination and to act upon any incidents that occur."