The clashes, which also left almost 200 injured, were triggered by protests against a citizenship law seen by many critics as anti-Muslim and part of Modi's Hindu nationalist agenda.
They exploded into brutal violence on Monday and Tuesday, with residents forced to flee their homes after seeing dwellings destroyed, a mosque attacked and a tyre market and two schools set ablaze.
"I appeal to my sisters and brothers of Delhi to maintain peace and brotherhood at all times. It is important... calm and normalcy is restored at the earliest," Modi tweeted on Wednesday.
Delhi's chief minister, Arvind Kejriwal, a political opponent, called for the army to be deployed and for a curfew to be imposed.
"Police, despite all (their) efforts, (are) unable to control the situation and instil confidence," Kejriwal tweeted.
Delhi Police spokesman Mandeep Singh Randhawa told reporters 106 people had been arrested and police were scouring CCTV footage to look for the perpetrators of the violence.