Classical music theory is a rigid, structural, almost mathematical system that emerged over several centuries. In those centuries, composers learned some very powerful techniques to express ideas in their music. Although the specifics of classical music — the rhythms, the chords, the tonalities — may be antiquated, I really do think that the forces that guided their development are just as relevant to modern music. For example: why is a V7 chord so powerful? It's because you have the interval of a 5th — a tense interval with almost universal meaning vis a vis the overtone series — combined with two minor 2nd intervals that are brimming with tension to resolve to the tonic and the third. The study of intervals and of tension in voice leading is, I think, absolutely essential in jazz as well, and can help explain some extremely confusing jazz chords in analysis as almost incidental results of voice leading. This is what I mean by classical music theory offering insight into any genre of music: classical music emerged out of the study of things like intervalic tension, and even though our own century's music has diverged from this origin, I think it's very important to study the lessons learned over those hundreds of years.
(EDIT: I just noticed that you said outside of Western music. Yes, that may be correct. Many systems of music in the world are very different from our own. But I think most people writing music today are writing with Western ears in mind. After all, we've been conditioned to understand this kind of music for hundreds of years!)
As for your second point, I think what I latch on to in classical music more than anything is the use of motif, and Beethoven's 5th of course epitomizes this. Don't get me wrong — I love popular music, and I listen to it more than anything — but most of the "top 100" songs in any popular music ranking feature wandering, improvised music more than any systemic use of motif to make a rhetorical point. And that's not a bad thing! I love Stairway to Heaven. But it just "sounds good" — it doesn't really develop its musical material.
Compare to something like this[1], where every single musical detail is repeated and developed across all the voices. It's so intricate. I've never heard this level of motivic development in popular music, and I've been looking for a few years now.
I am sorry if this is not making any sense, I am having beers.
I would say that the V7 gets it's power from the flatted fifth interval, aka the tritone, aka the most dissonant interval this side of a minor second. And like you say, it can be understood in terms of voice leading, e.g.the 7th (the "leading tone"!) "wants" to resolve up to the tonic.
It's interesting that you mentioned motif because I think that fits well with what I was saying about the length of the pieces. Seems to me that one isn't going to have much room to develop motifs unless writing in an extended form. But for me personally, this doesn't seem like a good requirement for a piece to be satisfying in the way that I think we are talking about. If you are seeking that kind of intricacy from modern music, have you checked out Bela Fleck & the Flecktones? Also, have you explored Zappa at all?
I mean, there are many ways to be satisfying in music. I am very satisfied by the exploration of texture and improvisation in modern music, for instance, and I am very grateful that we live in such an exciting time for musical experimentation. But at the same time, in a way that's hard to describe, the moment where the theme returns in the third movement in Beethoven's 5th[1] just absolutely trumps any musical moment I've heard in popular music. I'm talking Schindler's List level goosebumps. There are plenty of long pieces in popular music, too — prog is big on that. But the techniques of Beethoven and ilk just aren't getting used, which I think is a great shame. I don't think it's "magic" or "genius". I really do think it's a matter of harnessing the medium to its full potential, which might require more education in music theory than (in my experience) many musicians feel comfortable with.
I don't think you need a long piece to develop motif. Look at something like "Vocalise"[2], or, heck, any of Bach's Well-Tempered Klavier.
One of my teachers used to stress the original meaning of the word "sublime". I think that's what's missing in modern music. A feeling of awe that leaves you speechless.
Zappa's been on my to-do list for a while, I'll have to give him a listen soon!
(EDIT: I just noticed that you said outside of Western music. Yes, that may be correct. Many systems of music in the world are very different from our own. But I think most people writing music today are writing with Western ears in mind. After all, we've been conditioned to understand this kind of music for hundreds of years!)
As for your second point, I think what I latch on to in classical music more than anything is the use of motif, and Beethoven's 5th of course epitomizes this. Don't get me wrong — I love popular music, and I listen to it more than anything — but most of the "top 100" songs in any popular music ranking feature wandering, improvised music more than any systemic use of motif to make a rhetorical point. And that's not a bad thing! I love Stairway to Heaven. But it just "sounds good" — it doesn't really develop its musical material.
Compare to something like this[1], where every single musical detail is repeated and developed across all the voices. It's so intricate. I've never heard this level of motivic development in popular music, and I've been looking for a few years now.
I am sorry if this is not making any sense, I am having beers.
[1]: https://www.youtube.com/watch?v=hWLotW5AKjg