Then charge by weight. Just because someone is taller doesn't necessarily mean they weigh more or that I occupy more space. How about making seats narrower instead making me put my knees in the back of seat in front of me?
Size is an issue not just weight. You're carrying the airframe around no matter what, and the fewer seats per flight the more shared weight per passenger.
