exsft: Vegetarians are the minority, and as always, minorities are looked down upon.
People who don't follow the herd are ridiculed. Most people in the US and Canada are omnivores. So they think it's odd if people don't eat meat. They think it's a frivolous life choice. Unfortunately many of the people who complain the loudest are the least educated. Remember, many of history's greatest creative and scientific minds chose the vegetarian or vegan lifestyle. Those people were great thinkers and chose the same lifestyle you have. Just because the average person may not agree with you, smart people who don't have a chip on their shoulder will leave you in peace because they will realize that it is your choice.
For example, when you tell an ignorant meat-eater your lifestyle choice, some of them become so defensive. They will try to point the finger at vegetarians/vegans to divert attention from the fact that meat consumption contributes to animal cruelty. They'll say things like "But you eat plants; those are living things!" Yeah, okay.