Sure, but if a drug was truly harmful, we shouldn't encourage it. And if we discover activity X leads to doing something else harmful, it's cause for considering restraint. After all, cigarette ads don't harm anyone directly, yet we make the tradeoff to limit them in order to limit smoking.

(Not that it applies here. Everything I've read and seen indicates the world would be far better off if access to opiates and psychedelics was legal, easy, and well understood/educated.)