In a stable liberal democracy, the majority of citizens must consent to a shared vision of liberty.
As soon as some powerful minority imposes a vision of liberty that the majority experiences as unjust, the fragile alliance between democracy and liberalism begins to break down.
This will happen even if that minority is entirely correct about absolutely everything — that it really does have special access to the absolute moral truth, thanks to better education, purer motives, or more reliable techniques for counterbalancing its biases and neutralizing its own motivated reasoning — or any other similar claims that justify privileging one’s own judgment and enforcing one’s own convictions over the unconvinced.
If the unconvinced also think they have privileged access to the truth (and they do think that!), and however wrong they may be (and they are wrong about having special access to the truth!), the righteousness of the minority will not matter: the majority is going to assert its will. Angry majorities tend to have the advantage over righteous minorities in such conflicts.
One of the consolation prizes of vulnerability is you can never forget how dependent you are on persuasion for your very survival. This, and this alone, is what gives the marginal person some kind of intellectual privilege.