Let's not just gloss over the fact there are plenty of far-right organisations whose language towards people of other races and nationalities is particularly hateful and discriminatory. Stopping (just) short of condoning violence against them doesn't make their opinions any less horrifying. If they actually gained the power to enforce laws that discriminated against people on grounds of race and was able to enact measures they want like forced deportations and what have you, that would be quite concerning, would it not?
If I've learned anything in my travels through various political viewpoints, one thing I have come to understand is that in most societies the state does hold the monopoly on violence - They're the only ones who are legally allowed to use violence against people and those people don't have the right to fight back. We see it everywhere, whether that's totalitarian states like North Korea, authoritarian states like Russia, or western "liberal democracies" like the US and UK. If there does become a risk the could fall into the hands of people who are happy to use that violence in a discriminatory manner then perhaps it's a bit easier to see why people feel they need to use violence to prevent that. I mean, were the participants in the Battle of Cable Street wrong for standing up to the BUF? What if they hadn't? The real reason the Americans have the right to bear arms is to allow them to ensure their country remains "a free state" by force of arms if necessary. In most of Europe we have no such option, as the weapons are almost entirely in the hands of the police and armed forces. Once the state is in the hands of people who want to do terrible things to a certain section of the populace, there's no-one to stop them.
I don't think ignoring or glossing over problems with immigration and integration is helpful to anyone, and is ultimately much more likely to foster animosity between communities than open and honest dialogue is. I do think some European leaders are guilty of this, but I think they are still very aware of the fact that fascism was actually able to gain power in some of their countries, and with WWII it certainly affected them all. They're scared of even talking about these things in case they find themselves branded a fascist and made a pariah. To steer this back somewhat to the topic of the EU, for a lot of mainland Europe a major reason for wanting to establish a union was to prevent conflict on the continent again, but they concentrated on preventing conflict between countries and largely ignored the possibility for internal conflicts to develop between their people in say, the way they did in former Yugoslavia. I think our leaders do need to be brave enough to address this risk, and they need to do it soon.