I agree that Emacs seriously suffers from "anti-Python syndrome" (or maybe a better term is just "Lisp syndrome") in that there's many ways to do a given thing and no immediately clear "best" option. However, while it may not be immediately clear, I think almost everything you mention here has a "pseudo-consensus" best option. Not something everyone does, but that most people do, and that you're most likely to see recommended as best practice.

- You should be defining groups and variables to provide better documentation and allow use of the interface. But you shouldn't actually use the interface to do customization (put it in your personal configuration files instead). The interface is both pretty annoying and disallows documentation of why exactly you're setting certain options.

- You use the standard package manager (package.el) and ELPA, MELPA, and maybe the Org repo. There are a few of these which will require external tools of some sort, in which case fair enough, that's an extra step. But that's still pretty standard in my view (although definitely doing that automatically would be great). I don't think I have any packages that require alternative package managers anymore.

- Use use-package. No real question at this point.

I do agree that themes are in general a nightmare.

From my experience as a developer and maintainer of several packages now available via MELPA and MELPA Stable, I can say with confidence that the standards for new package submission are quite high - indeed, my trips through that review process, and the excellent feedback I have on all occasions received, have taught me a great deal about how to build Emacs libraries and packages in a reliable, maintainable, usable fashion.

Unfortunately, there is a great deal of older Emacs Lisp code out there, much of which was not written to any perceptible standard whatsoever beyond "works on my machine lol ¯\_(ツ)_/¯". I say this not to disparage the people who wrote and released that code - indeed, a lot of what's in my own ~/.emacs.d, and which I use daily, comes of the same sort of origin. But either way, the Emacs ecosystem has a serious legacy code problem - something which, of course, you already know full well.

Unfortunately, there's no good single suggestion I can make here.

I can say that learning Emacs Lisp will make it easier to cope with nonsense like people unaccountably refusing to implement package configuration in terms of Customize, and with the occasional package where Customize legitimately isn't flexible enough to support everything that package needs to do.

But learning a whole new programming language, especially one that originates in a paradigm not familiar to most working programmers today, is a hell of a big ask as a prerequisite for effectively using an editor. My own experience is that it's well worth doing so - but if you're already frustrated with the nonsense you've observed, I can hardly blame you for deciding it's not worth your own while.

I can say that, well, it's just a matter of needing to invest the time to learn how to leverage this uniquely powerful tool, and not everything worth doing is easy - but, quite aside from being rude and insulting, such a claim has the same problem as the one I just talked about. I mean, I started using Emacs eight years ago, and it took me three or four of those to achieve something that can reasonably be called proficiency. That's a hell of a learning curve.

I can't reasonably say that the problems you're seeing don't exist, or result from error or misperception on your part rather than failure on the part of the ecosystem. (No one else can reasonably say that, either, and I misdoubt some have tried.) Among other traits, Emacs is older than the entire user interface paradigm that's become ubiquitous since 1985, and that makes many of its core metaphors seem either strange or strangely expressed. "Theme", for example, which everywhere else means a specific visual style packaged as a unit with a name, in Emacs means an entire set of Customize configurations packaged likewise. And then Emacs also, separately, has lately gained themes of the newer sort, with the same name referring to both! Of course it's a huge gotcha for the new user. And there are many more.

All I can say is: the Emacs core maintainers, and those of us building new packages for public release, are working the problem - doing our best to relieve Emacs of the manifold architectural and ecosystemic burdens which make it so easy for people to reasonably regard the platform and its partisans as relics of the past, and replace those legacies with a structure and culture sound enough to support Emacs' next 30 years. There's an almighty lot to do, and as with any free or open source project, never enough capable hands for all the doing. But as we can, we are.