That sounds really cool man, hell ya. Follow your math moose wherever it wants to take you, just go with it. Learn diverse stuff too, and try to connect them with math, if not logically then broadly; that is, emotionally. Then you may see the most beautiful crags of logic and waterfalls of emotion and symbols and symbolism and vast frameworks of interconnectedness -- what does it all mean???
I've thought of doing something like this before, but then I know I would anyway. Only lately I don't have any discipline for studying math. I used to be great at it, almost enjoyed it even. Mind you, I just graduated high school, so I didn't get into crazy shit, just Calc BC.
Anyway, I'm not gonna ramble on. Do post your experience with this, and if you gain any insights into math or if you feel that studying math had some interesting effects during your trip.