Is occasionally using dried lavender herb in the same room as my cats an issue?

I occasionally like to use dried lavender for spells, I typically don't use it often. Maybe once or twice every four months. It's not essential oils or anything. Just the plant itself dried from my garden. I watch them, I'm not worried about them eating it as I would NOT let them eat it or be close to eating it, right now theyre asleep. I'm more worried about the smell since they would certainly not have a chance of getting close.