I have heard it suggested as obvious that some ideas are harmful. For instance, the idea that faith healing has a better chance of curing you of cancer than modern medicine could certainly cause people harm.
I think we need to draw the line between the concept that ideas or information may cause people to do harmful things, and the concept that ideas or information can be inherently harmful.
The first we see happening regularly. The second seems to be the main concern of people who seem to think that they are superior enough to everyone else to be unaffected by information, but that it is their responsibility to ensure no one else has access to it, especially children. I can still remember being a child, and I don't buy it for one second.
Perhaps there are a couple forces at work here. First is the erosion of the principle that people are responsible for their own actions. The second is that harmful ideas have much less power in a well educated and skeptically minded population.
I have an inclination to believe that free will is just an illusion, or is at least a chaotic system such that it is impossible to predict the future without taking into account what effect predicting the future might have on it. That is to say, it's deterministic, but not necessarily predictable. However, I don't think this means we should act like people are not responsible for their own actions. Ultimately, if they do not, then we also are not responsible for our reaction to their deeds, so I think we can just call this one even.
Regardless, what this comes down to is the "ruining it for everyone" problem. When I was in school and the class was granted some special privilege or freedom, occasionally some joker would abuse it and result in it being taken away from everyone, hence "ruining it for everyone." It seems that every time someone does something bad, like shooting people in a school or crashing airplanes into buildings, it's everyone else who gets their privileges or freedoms taken away. The idea of freedom is that some of these things never get taken away, regardless of how badly others abuse them, but that appears to be lost on some people.
As to ideas being "harmful to other people," there are a few forces at work there. Some people seem to believe that, just because they do not wish to see something, they must force everyone else not to see it as well. This despite the apparent lack of evidence that the mere existence of information could be harmful. Critical thinking does not seem to be a significant part of education, at least it is not a main focus, or something educators proudly proclaim to have accomplished with their students. Perhaps it's too hard to test, or too hard for them to teach. That would be a shame.
So it's easy then. We just need a critically-minded and educated population, and the will to stand up to people who would take away our freedoms when someone else does something wrong.
Somehow, I'm still cynical.