Yadav is touchy about the issue. On Monday, he snapped at an NDTV reporter who asked about Muzaffarnagar at a press conference, seemingly because of the channel's negative coverage.
While he is probably hoping to avert political and media attention by dismantling the camps, this move may just backfire.
Trucks have arrived to transport refugees to nearby government buildings and administrators have begun bulldozing the temporary shelters to force residents to leave.
But the residents do not want to leave. The alternate accommodation provided by the government is not much better than their camps and leaves them just as exposed to the elements, they said.
Only 4,783 of the original estimated 40,000 refugees remain in the camps that were set up hastily after riots in September.
There is little sanitation or access to water. After the onset of winter, at least 34 children have died from exposure to the cold.
Although the camps are poorly equipped, residents have few options.
The government has offered a compensation of Rs 5 lakh on the condition that those who accept it never return to their home villages.
They will, however, retain ownership of their property and are free to rent it out as they see fit, at least on paper.
Several refugees who accepted the compensation have bought land in villages near the camps.
Even if everyone in the camp is given their promised compensation, three or four days is hardly adequate time to find places to live.
The Muzaffarnagar riots forced 40,000 people to relocate from their homes in mixed communities to villages with Muslim majorities.
After riots in Gujarat in 2002 and in Mumbai in 1993, Muslims were ghettoised into enclaves by fear and social circumstances.
This is the first massive reorganisation of a religious community in rural areas in years.
If the Samajwadi Party has an eye on the coming national election, exploiting a human tragedy and making refugees fend for themselves is likely to lose them votes of Muslims, who are a crucial part of their support base.