How do you know you have a mix of 5-MeO-DMT and DMT in the first place?
TLC separation is possible, though generally you'd need to make preparative TLC because normal TLC plates might be too thin and you wont be separating much material in the first place.
Also another question is, how will you visualize the spots? If you use comercial TLC plates, they generally will have a UV fluorescence indicator, so with a simple short-wave UV light you can see the spots and know where to scrape, but if you're preparing TLC yourself, the normal silica wont have the UV fluorescent indicator. You'll need some reagent to show the spots and know where to scrape the silica, but a reagent generally will destroy the sample, so you need something else. While they definitely exist, I'm unfamiliar with non-destructive TLC visualization methods apart from UV light with the plates I mentioned, maybe a chemist here can help.
Another option is to use column chromatography for separation and use TLC just to check your separation, so in that case you'll be able to use a destructive reagent for visualization if you dont have the UV fluorescent plates. If you want the UV fluorescent plates, click my signature for more info on TLC kits, might make things simpler for you.
As for the eluent, I've been using methanol:25% ammonia (100:2.5) for general tests and it does separate 5-meo-dmt and dmt but the separation isn't the best. According to literature you could also try the following eluents:
2N acetic acid
Rf
DMT 0.50
5-MeO-DMT 0.60
(Dutta & Ghosal 1967)
or
t-Butanol-acetic acid-water (70:2:29)
Rf
DMT 0.50
5-MeO-DMT 0.40
(Duttha & Ghosal 1967)
Another way of separating them is through solubilities. One possibility is by the solubility of the acetate salts in chloroform, as you can read
here. It might work with other solvents appart from chloroform, might be worth a try. If you use chlorinated solvents please dont leave the alkaloids for too long otherwise
they might react.
Yet another option that was once discussed but unconfirmed if it really works is by the
solubility of the citrate salts in acetone. Might be worth a try.
That being said, I still wonder how you think you have that mixture in the first place.
Lastly, please do share whatever are the results of your experiment if you go ahead with this, we could all learn from it for sure