I think the deciding factor here is the individual. It doesn't really matter what the psychedelic is. Intention, set/setting, self projection, and a dash of randomness are the variables that separates recreational (hedonistic, which theres nothing wrong with so long you are the one in control, and not the drug) from spiritual, therapeutic/medicinal, introspective, etc.
People use psychedelics recreationally because they create the circumstances cohesive for that type of usage. The same goes for any other uses of psychedelics. Its the same for all drugs. You can be a moderate occasional drinker and by doing so enhance aspects of your life, creativity, etc. Or an alcoholic who needs to drink to feel normal, physically and psychologically. You can trip recreationally occasionally, and get great utility from doing so through whatever it provides to you.
I would hazard a guess that even though the harm is relative, relying on psychedelics to sort your life/mind/emotions/etc out is not healthy usage of them if you don't learn and make positive change that makes you need/want these lessons/experiences less and less until your where you want to be in life. IE integration of what you've learned.
But those who realize the potential after fixing/learning about personal issues by using the drugs to come to terms and understand their problems, imho find the most utility from psychedelics. That's to say you finally settle on what they really do, expand possibility/perspective cognitively. That opens up all kinds of usage of psychedelics, where occasional recreational use is not prohibitive. You find more and more uses for the tools by thinking of them as Swiss army knives that always have a tool to be found.
Hedonistic, recreational, party, tripping balls man, usage of psychedelics where the individual does not wish to use them as tools, is another way. I'd bet theres people out there that can do this to extents most others can't because they are focused/determined on why they use them, their intentions so to say. But having the wrong intentions (no matter how strong) and tripping all the time will cause most people to be caught off guard, and either scare them away, or be forced to realize the mistake of doing so by a rough trip.
Quite possibly even cause lasting positive/negative changes in their life since they were forced over into another use these tools have though breakdown of the cognitive dissonance that allowed them to think of them as "just another euphoric drug".
But having strong, responsible, intentions to use a psychedelic recreationally responsibly imho will elicit that kind of experience if done right. Often time with positive affects for the individual. Just as the same holds true for people using them for any other use. The full potential is realizing the variety of experiences they can induce using these tools and never stop experimenting. But at the same time, always remembering to have strong intentions for doing so every time, and abusing them will quickly degrade that in people who are there just for the high, just for fun, far too often, and without regard for the psychological risks involved.
I've been here there and everywhere with various substances, and the way i see it, the only thing you can do wrong with psychedelics is to try and control the experience and use it for your own needs/purposes compulsively/repeatedly. The way i believe these drugs work is the anti-thesis of that, and perhaps why people don't abuse them as much as other drugs. At its core its telling you don't get caught up here, theres always more uses for these tools to be found.
That would seem a hard thing to do trying to chase the metaphorical psychedelic dragon and abuse them. Your either going to cause psychological damage, or realize that what your doing is no different than any drug abuse, and how its counterproductive to what you could be getting using psychedelics more generally and balanced. Perhaps that holds true for people who claim profound spiritual realizations from psychedelics but they don't stick when the trip is over. Their spirituality is reliant on tripping so to say. IMHO thats abuse since your not really getting anything from it that sticks. That doesn't however mean you should "hang up the phone once the message is heard". There is no true enlightenment offered by psychedelics, just experiences that open you up to more of the bigger picture.
The message is never the same, and thats the magic of it. If you sit there and listen to message after message, you don't leave time to let your mind make sense of it all in relation to the bigger picture. That's the problem with psychedelic abuse, which can manifest through recreational, spiritual, introspective, self medicative, etc usage.
To me it sounds like the OP has legitimate issues with people using them just for recreation, when in fact those people have issues with abusing drugs in general. Occasional recreational use is fine, no matter what the drug, just not too often, and always kept in perspective.