The board of Tata Sons, the holding company of the Tata Group, has unexpectedly replaced Cyrus Mistry as chairman.
Ratan Tata has been named as interim chairman while a selection committee finds a permanent replacement.
Mr Mistry has led the company, which owns Tata Steel, since late December 2012.
He was the first chairman in nearly 80 years to come from outside the Tata family.
No details were given by the company as to why the board had decided to make the change.
Mr Mistry was only the sixth chairman of the group, which is nearly 150 years old.
Ratan Tata was the previous chairman and was in charge of the company for more than two decades until he stepped down in 2012 at the age of 75.
Mr Mistry was appointed more than a year before Mr Tata retired.
Tata Sons owns Jaguar Land Rover, as well as Tetley Tea. It is one of India's oldest conglomerates and is made up of more than 100 companies, including Tata Motors, Tata Power and the IT giant, Tata Consultancy Services.
Tata employs almost 7,000 workers around Wales, including more than 4,000 at its troubled plant in Port Talbot, south Wales.
Analysis, Yogita Limaye, Mumbai correspondent
This is perhaps the biggest upheaval that corporate India has seen in recent years. It's a sudden move from a company that is usually very private and conscious of its public image.
Analysts are divided about Mr Mistry's performance as chairman of the Tata group, but many of the challenges he faced were inherited.
It was during Ratan Tata's tenure that the group made many bold acquisitions, among them the takeover of Corus Steel and Jaguar Land Rover. Some of those decisions paid off, while others have cost the company a lot of money.
JLR has turned into a profitable company, but Tata Steel has been suffering heavy losses in the UK. Speculation has already begun about why the decision to replace Mr Mistry was taken by the board, but the person who replaces him has their job cut out for them.
The members of the selection committee charged with finding the permanent replacement are Ratan Tata, Venu Srinivasan, Amit Chandra, Ronen Sen and Lord Kumar Bhattacharyya.
The committee will complete the selection process in four months.
Mr Mistry is the youngest son of Pallonji Mistry, an Indian construction tycoon.
The Shapoorji Pallonji Group owns 18% of Tata Sons, making it the largest shareholder.