Sai Baba, who was admitted to hospital in his hometown of Puttaparti a month ago, died of multiple organ failure, media said.
His followers, estimated to number six million, included top Indian politicians, business tycoons and Bollywoods stars.
Sai Baba, with distinctive frizzy hair and always clad in his trademark saffron robes, ran schools and hospitals through trusts in numerous countries. Questions are likely to arise over the management of his substantial assets.
He was credited with mystical powers including conjuring objects out of thin air.
Prime Minister Manmohan Singh said Sai Baba's death was "an irreparable loss" to all.
Legions of followers, including top judges and bureaucrats, were expected to take part in his last rites, media said.
