It is a source of "national shame" that as many as 800,000 people in England are "chronically lonely", Health Secretary Jeremy Hunt has said.
In a speech, Mr Hunt highlighted the "problem of loneliness that in our busy lives we have utterly failed to confront as a society".
"Some five million people say television is their main form of company," he said.
Labour accused of him trying to blame families for government failures.
In a speech at the National Children and Adults Services (NCAS) conference, Mr Hunt said: "Each and every lonely person has someone who could visit them and offer companionship.
"A forgotten million who live amongst us - ignored to our national shame."
'Reverence and respect'
He added: "According to the Campaign to End Loneliness, there are 800,000 people in England who are chronically lonely."
He also told delegates that the 112,000 cases of alleged abuse in care homes referred by English councils in 2012-13, the majority involving over-65s, indicated that "something is badly wrong".
But the regulation of care in both the private and public sector was improving, he argued.
The Conservative MP said the new chief inspector of social care, Andrea Sutcliffe, would start to give ratings to care homes from April 2014 with a view to organising inspections of all 25,000 care homes by 2016.
Mr Hunt said he was "particularly worried" about the 400,000 people in care homes, some of whom get regular visits but others who were just "parked there".
He said that 46% of people aged 80 or over reported feeling lonely "some of the time or often".
He warned that loneliness was as "bad for you" as "smoking 15 cigarettes a day", was "worse than obesity" because of the risk of blood clots, heart disease and dementia, and warned that lonely people "drink more" and were more prone to early admission in residential or nursing care.
Mr Hunt also said he believed the UK should learn from Asian cultures where there was "reverence and respect for older people" and "residential care is a last rather than a first option".
"The social contract is stronger because as children see how their own grandparents are looked after, they develop higher expectations of how they too will be treated when they get old," he explained.
"If we are to tackle the challenge of an ageing society, we must learn from this - and restore and reinvigorate the social contract between generations.
"And uncomfortable though it is to say it, it will only start with changes in the way we personally treat our own parents and grandparents."
England should aim to become "the best place in the world to grow old in", he concluded.
But Labour said the "real national shame" was "Jeremy Hunt's attempt to shift the blame for dealing with the very real problems of loneliness amongst elderly people on to families alone".
Shadow health and social care minister Liz Kendall said: "He seems completely unaware that there are over six million unpaid family carers in Britain today, one in five of whom provide more than 50 hours care a week for their loved ones.
"These unsung heroes save the taxpayer billions of pounds yet often get precious little support in return.
"Families, friends and neighbours need a decent care system to back up their efforts to look after elderly people, but the reality is our care system is in crisis and has been pushed to the brink of collapse.
"Jeremy Hunt and David Cameron need to take responsibly for their government's actions rather than trying to turn the clock back and say it's down to families alone."
Shadow health secretary Andy Burnham said on Twitter that Jeremy Hunt was "right to highlight isolation of older people".
He added: "But he can't get away from fact that Govt cuts to care have made it much worse."
But speaking on the BBC Radio 4's Today programme, Anthea Tinker, professor of social gerontology at King's College in London, said it was a "myth" that eastern Asian families placed greater reverence on families and older people than in the West.
"The reality is that with one-child-families, children have often moved to a city or emigrated," she said.
"So it's just not practical for families to depend on their children. One of the largest nursing homes in the world is about to open in China, for 5,000 people, which is amazing."
Prof Tinker added: "We've got to look at the reality rather than the myth."
Mr Hunt's speech came as a BBC poll found that almost half of all adults said they experienced feelings of loneliness.
The survey of more than 2,500 adults in England was commissioned for BBC Radio 2 and BBC Local Radio's Faith In The World Week.
It also showed that people who practised a religion felt lonelier than those who did not.
London was identified the loneliest place with a figure of 52% compared with 45% in the south west of England.