Prejudice against Muslims is not due to Muslim extremists. Muslim extremists represent literally 1% of 1% or so of the world’s 1.6 Billion Muslims, and most people are able to therefore realize that 99.99% of all Muslims are not extremists.
Anti-Muslim prejudice is due to things that people have made up about Muslims. Many of these efforts are professional. A relatively small cadre of people with an anti-Muslim agenda are involved.
The Southern Poverty Law Center recently published three articles on professional anti-Muslim efforts in the United States which outlines who the players are, and what they are doing.