Free software assumes that all end users might want to modify it, regardless of those users' current skill set. I may not be capable of modifying kernel source code today, but who knows what the future might bring.
Even if end users can't modify it themselves, they can hire developers to do it. This effectively creates a market for software customization/personalization, whereas closed-source software is more akin to a monopoly (whoever holds the source code, holds the monopoly to fixing bugs/adding features).