Is Free Speech Really Free?

Last Tuesday I went to a Giants Game and where I was sitting we were close to the visiting teams bullpen.
When the pitcher started to warm up some of the fans in our section began to yell rude things to them and make fun of their pitching abilities. Most of the comments were just in good baseball rivalry fun, but the guy sitting behind me and my group of friends took it to the next level. He started using foul language and personally attacking the pitcher from the other team by talking about his mom and saying things that some members in the crowd found "obscene". A few minutes later the ballpark security came to our section and asked the guy to step outside with them. He got very upset and started saying that he hadn't done anything wrong and that he had the right to say those things. The security guard thought otherwise and after some very loud confrontation between the guy and his group of friends and the security, he was escorted out of the section and ejected from the game. This got me thinking about is Free Speech really free? Other fans were poking fun at the pitcher too, but because foul language was involved with the other guy he was the only one ejected from the game. But people use foul language and profanity in public all the time, so why is it ok to do it on the street, but if you do it at a baseball game you can be ejected?
He was being very loud and distracting the other team's pitcher, but that is common in the major league baseball environment. It just doesn't seem right that he was the only one punished, when there were numerous other people doing pretty much the exact same thing. Something else that I thought was interesting is that security didn't even give him a warning, like " hey some of the things you are saying is offensive to other fans, so if you don't stop we will eject you from the game". Instead they didn't even give him a chance to explain himself or defend himself they just told him that he could take it up with the police outside.
This event just put some things into perspective for me that not all speech is free and that there are consequences for the speech we choose to say.

 

Baseball is democracy in action: in it all men are "free and equal," regardless of race, nationality, or creed. Every man is given the rightful opportunity to rise to the top on his own merits...It is the fullest expression of freedom of speech, freedom of press, and freedom of assembly in our national life.
~ Francis Trevelyan Miller