In fact there are MANY people (possibly MOST people) who take exactly the opposite perspective: If an idea isn't intimately connected with "religion" then it is highly suspect (or worse).

Here's an article that discusses a poll study showing that people generally trust atheists and rapists about equally.