Not necessarily, and I would say most often it won't be the case, at least with MHRB. As an example, look at these pictures of very yellow but pure DMT:
Why is it yellow?
That's really up to you, it shouldn't make any difference when smoking it. But if you prefer white, you can set apart the yellow and recrystallize (re-x) with heptane. Or just recrystallize all of it. That should clean it of any remaining impurity, if there are any, and it tends to be white that way (but as you can see in the pictures above, it could remain yellow).
Personally, I wouldn't worry about color.