I think it´s only toxic when your intake exceeds certain amounts taken at once. When you take small amounts on a regular basis i don´t think it´s gonna accumulate into any damage, just like when you don´t get drunk when you spread 20 beers over 40 days but you do get drunk when you drink them within half an hour.

There may be other reasons not to put teaspoons of the stuff in your smoothies, though. When it comes to nutmeg as flavoring spice it´s my personal opinion that less realy is more. A bit too much and you ruin every dish you can think of.