That is a good question. I think the food safeness is merely that - in the context of food. d-limonene comes from citrus peels. You've probably inhaled it before any time you cut a lemon, lime, orange, etc. It's also used as an industrial cleaner, I wash my hands with it at work every day.
But concentration matters. Pure d-limonene is going to be more dangerous than an orange or hand soap. I do recall when I extracted mescaline with d-limo, and was squeezing the solvent out of the cactus dough, it made my hands very dry and kinda itchy. While the soap uses a 1% to 20% concentration that is non-irritating.
I assume inhalation hazards are similarly dependent on concentration / quantity. If it were lethal to inhale any amount, we probably couldn't eat oranges. I don't recall any issues breathing the fumes. I'm not sure what it would take to cause damage via inhalation, but I think you probably have nothing to worry about.
That said, probably every extraction includes solvent residue. I haven't seen any studies on it, but I sent a very pure sample for analysis recently and was told there was weight lost from their drying method, indicating something volatile, (water or solvent) that was included in the sample. And that's after I purged the crystals under vacuum for an hour or so.