I briefly considered joining the Freemasons, as it's somewhat a family tradition. There was a good mix of guys in that particular lodge, and not everyone was Christian. In fact, to my surprise, upon swearing in, you could choose any religious book you wanted, so long as it represented faith in a Supreme Being/Architect. One guy was sworn in using the Koran, and maybe another using the Rig Veda. Anyhow, I didn't join because meetings conflicted with my work schedule.
Sometimes guys need a space just to be a guy and talk about guy stuff – and there's nothing wrong with that. For example, when we I had a drinking club, I would tell stories to the guys that I wouldn't if my wife or another lady was there, because I would have come off wrong. And that's just how it is.