Prof Noriko Arai has spent years training a robot to pass prestigious University of Tokyo's entrance exams.
And in 2015 and 2016, her Todai robot outperformed 80% of high-school pupils and was in the top 1% for maths.
But Prof Arai, a mathematician at the National Institute of Informatics, is not happy about how well it is doing.
At the Ted (Technology, Entertainment and Design) conference, in a session called Our Robotic Overlords, she said the results were "alarming".
"You might think I was delighted, but I was alarmed," she said.
"This robot, which could not read or understand, was able to outperform thousands of high-school children."
This led Prof Arai to investigate the reading and writing skills of high-school students, in conjunction with Japan's ministry of education.
"Most of the students pack in knowledge without understanding, and that is just memorising," she said.
"AIs can do that better, so we need a new type of education."
Stuart Russell, a professor of computer science at Berkeley, University of California, told the session machines would soon be reading and understanding very well.
"And very soon afterwards, they will have read everything that has ever been written," he said.
Humans needed to start devising rules for how robots related to them, he told the Ted audience, and proposed three basic principles:
- The robot's only objective is to maximise the realisation of human values
- The robot is initially uncertain about what these values are
- Human behaviour provides information about human values
But Prof Russell acknowledged there might be teething troubles.
"If a robot's job is to feed hungry kids and it sees the family cat but doesn't see that the sentimental value of the pet is greater than its nutritional value, that could single-handedly destroy the market for home robots," he said.
The idea of altruistic robots is one that Tom Gruber, the man who designed Apple's voice assistant Siri, believes in.
And while some experts, including tech tycoon Elon Musk and Prof Stephen Hawking, worry machines will overtake and destroy mankind, he maintains an optimistic vision of "humanistic" AI.
"The purpose of AI is to empower and augment us," he told the Ted audience.
"Imagine if AI remembered every person you ever met or could retrieve everything you had ever read or seen.
"Not only would it make us better at remembering people at social occasions - but for those with dementia or Alzheimer's, it would mean the difference between a life of isolation and one of dignity."