MEK evaporates completely within a few minutes, so it's of little concern. Xylene takes ages to evaporate and there’s often enough left in the final product that you can taste it unless you do a lot of steps to remove it. So you can't really compare the two like that.

Also, MEK might have a lower LD50, but in the amounts used its definitely healthier than xylene. At most you might have less than 1% remaining in your product. We don't eat xylene, but people do eat MEK nearly everyday from fruit.

I'd say while xylene has a higher LD50, if you compare small amounts of each then MEK is still healthier. MEK also smells better.

If you were really worried about MEK, you could always do the same cleanup using d-limonene mixed with ethanol at a ratio that simulates the polarity of MEK. So it can be cleaned up using 100% food grade solvents. MEK is just easier because no mixing is required.