IIT-Madras faculty develop AI models to process text in 11 Indian regional languages

Read| Emerging courses to pursue: Virology | Actuarial science |  Pharma Marketing | FinTech | Coronavirus | Robotics | Healthcare Engineering | Cyber Security | Data Science | Petroleum and Energy | Design Strategy | Business analytics | Digital auditing | Digital marketing | Luxury management | Machine learning | Corporate Law | Product design

For the past one year, a team of researchers comprising students, faculty and volunteers from IIT-Madras and AI4Bharat worked on collecting data and training powerful models for processing text written in Indian languages, claims the IIT. The models take advantage of the similarities between Indian languages to make efficient use of data.

With these models, the researchers have been able to push for Indian language processing on several tasks such as document classification, sentiment analysis, semantic matching, paraphrase detection and so on.

Elaborating on this initiative, Mitesh M Khapra, assistant professor, department of computer science and engineering, IIT-Madras, said, “We have a very rich diversity of languages in our country. As we move towards a digital economy, it is important that our languages find a space online. This requires a lot of innovation in creating input tools, datasets, and AI models for Indian languages.”

Read | Ayodhya Ram Temple construction: L & T reaches out to IIT-M for expert help on design, concrete

“While such tools are available for English and other foreign languages, there are hardly any tools for Indian languages and this is the critical gap that we are trying to address through this initiative. These models are available free of cost as we want the entire country to benefit from them,” added Khapra.

The researchers from IIT Madras and AI4Bharat released AI models and datasets for the following languages: Tamil, Hindi, Malayalam, Telugu, Kannada, Punjabi, Bengali, Odia, Assamese, Gujarati, and Marathi. The multilingual AI models and datasets developed through this initiative will provide the essential building blocks to students, faculty, start-ups and industry to work on Indian language tools and push the frontiers of technology.

The resources are available in open-source and are free of cost, which can be accessed by anyone. These models are freely available and can be downloaded from a Github repository. An accompanying research paper describing the research methodologies and evaluation have been accepted at EMNLP-Findings (a companion publication at one of the top Natural Language Processing conferences).

Read | IIT-Delhi sets up School of Artificial Intelligence; to offer degree, professional courses

AI4Bharat is an initiative co-founded by Dr. Mitesh M Khapra and Dr. Pratyush Kumar from IIT Madras and works to solve India specific problems in a community-driven, open-sourced manner. Both Dr. Mitesh Khapra and Dr. Pratyush Kumar are also associated with the Robert Bosch Centre for Data Science and Artificial Intelligence.

Pratyush Kumar, assistant professor, Department of Computer Science and Engineering, IIT Madras, said, “This initiative is one of the few attempts in Academia to develop and publicly release such large scale multilingual AI models containing millions of parameters trained on billions of tokens from 11 Indian languages, completely free and open-source.”