I think freedom of speech should be limited a ways before it "infringes upon the rights of others." For example, I recall seeing a protest on the news of this batshit woman who was preaching about her own religion. They protested near a military funeral, with signs an slogans such as "God hates fags," "Thank God for dead soldiers," "Thank God for 9/11" and other stuff that would generally start a riot.

And the police had to let them do it!

I draw the line at racism or anything like the horror story I told. Yes, free speech is fine... But if you are protesting specifically to offend or provoke a person, race, or party, then you should have to shut the hell up.