Prime Minister Narendra Modi on Sunday told leaders of the Bharatiya Janata Party that Muslims should not been viewed as vote banks, and that they should focus on empowering the underprivileged. Modi was speaking at the party’s national council meet in Kozhikode, a bastion of the Muslim League and the left parties.

“Do not reward them, do not rebuke them [Muslims], but empower them. Consider them your own,” Modi said, invoking Jan Sangh leader Deendayal Upadhyay, whom the BJP considers an ideological mentor. The meet also marked the beginning of Upadhyay’s birth centenary celebrations by the BJP.

Modi also said the party has to reach out to other minority communities, saying “people at the higher level have to bend down and support those who have been exploited and neglected.” He added, “no one is untouchable in society”, in a reference to Dalits, who have been protesting all over the country over the past few months against incidents of caste violence. On Saturday, a pregnant Dalit woman in Modi’s home state of Gujarat was beaten up, along with her family, by upper caste groups, allegedly for “not picking up cow carcasses”.

A large number of the BJP’s Lok Sabha seats came from the so-called Hindi belt in North India, but it has been looking to increase its influence in newer states such as Kerala. Since it came to power, the party has faced increasing criticism for implicitly empowering right-wing vigilante groups that often target Dalits and Muslims.