As a counterpoint, I knew many hobbyists who were not in school for electronics use it to great effect.
Eg, in the Boston artist scene. One girl designed and built a whole bunch of home-brew analog synthesisers. Another guy made a performance art kinetic sculpture that would move and bang on drums and other percussives.
Like I mentioned in my comment, the book by Forrest Mims is a more gentle introduction that can get you started. But if you are hungry for more, AoE is the place to go.
Alright, you've piqued my interest! I've had a couple of electronics projects in mind, mostly audio-related, like building MIDI controllers, experimenting with modular synthesizers, setting up a MIDI-controlled Gameboy synthesizer.
AoE looks pretty good. Some of then intro material I've looked at before doesn't go into much technical detail, and I feel like I'm missing the fundamentals.
But if I want to actually start experimenting with my own circuits, what should I buy? Should I order components one-by-one off of Digi-key when I think I need them? Or is there a nice starter kit of components that would get me started?
- besides components you'll want at least a breadboard, DC power supply, and multimeter. For tools you'll want needle-nose pliers, wire cutters and a wire stripper. Down the road you will want to consider an oscilloscope and soldering iron but can hold off on those for now. Though if you have access to a university lab or hacker space maybe you can work there instead.
- what's your goal, to learn electronics while making synths, or just to make synths yourself? If you really want to go through the "rabbit hole" as Kyzyl said, besides the AoE text you may really want the students lab manual as well. Even if you don't do the labs, it will show you the material in a logical order, without overwhelming you. Maybe this is the frustration Kyzyl was referring to (ie not using that manual)?
- breadboarding is great for hacking around quickly. But it's also like giving your circuit lots of tiny antennas. And you can pick up radio and other signals through accidental rectification + low-pass filtering almost anywhere. So don't despair if your initial attempts have some interference problems, as eventually you may need to consider good noise-reduction methods, and/or use shielded boxes, shielded coaxial cables, etc.
- for components, you could start with a standard assortments of resistors and capacitors. Get one of those sets of pre-stripped wires for breadboards. You'll probably want a few transistors and op-amps. You but can start off with cheaper components as you're learning and hacking, but will likely want to eventually get some quality audio-grade op-amps.
They make DIY audio kits. I built their Fatman synthesiser years ago after graduating university. It sounds great and may give you ideas both on audio generation and midi decoding. It uses an embedded CPU to decode the Midi, and output a voltage linear to the key's pitch. Then has the oscillators (two), filters, low-frequency oscillators, and envelope control. It's a kit you put together, so they give you all the parts. But also they give you schematics. And it could give you some ideas and also a starting place to mod it up yourself. (I had ideas to expound on but never got around to implementing).
If what you're interested in doing is analog/digital synth stuff, the AoE has the information you need. It leaves some gaps and presents it in a way that I am not fond of, but that's just me and I'm far from your typical learner. That said, be prepared for a long, dark rabbit hole ;-)
Depending on the level of "off the shelf" you're looking to deal with, a MIDI controller can be as simple as a USB controlled MIDI chip hooked up to some switches, or as complicated as a manually implemented digital synthesizer that talks to a USB bridge (don't try to implement a USB controller as your first, or tenth project).
Regarding kit/components, the key is trying to strike a balance between knowing what you want before you buy anything, and not waiting too long to start experimenting with real components. You can save yourself a lot of time by simulating the basic circuits in your project using a software like LTSpice[0] or TINATI[1]. They're both free and easy to use. However, simulations are never good enough, and often you don't have a model to simulate some important piece of your circuit, so the picture isn't complete. A good approach is to sketch out the circuits you want, do a quick sim or two to see if the basics like your power supply or oscillator or digital logic are going to basically do what you think, then order all the components you need from digikey. Shipping under $200 costs $8 from digikey, so it pays not to make too many orders. The bonus is that they are extremely timely with their orders (in US and Canada anyhow). Sometimes unbelievably so.
A note when ordering things from Digikey: First, make sure all the parts a in stock (check "in stock" in the search params). Back ordered parts take a long time, and for almost any part you can imagine there is a comparable replacement in stock. Second, when you want low volume stuff you'll be wanting the "cut tape", "bulk", "tube", or "tray" packaging options. "Tape and reel" or "Digi-reel" are for big rolls of one component and this the minimum order is 1000+ units. Almost everything is available in one of the low-volume options.
Regarding kits, take a look at sparkfun[2] for a wide range of boards, kits, etc. They have some good stuff and are targetted at hobbyists. A lot of their products come with not only a datasheet, but also a simplified "hookup diagram" that tells you how to wire everything up for a typical application. You can get some kits of basic components like resistors and caps, but they tend to get pricey and I always found that I would use one or two of the components up and never touch the rest, i.e. use all of the 10k resistors and the 1uF caps and nothing else. If there's a kit specifically for the project you want go for it though.
As a counterpoint, I knew many hobbyists who were not in school for electronics use it to great effect.
Eg, in the Boston artist scene. One girl designed and built a whole bunch of home-brew analog synthesisers. Another guy made a performance art kinetic sculpture that would move and bang on drums and other percussives.
Like I mentioned in my comment, the book by Forrest Mims is a more gentle introduction that can get you started. But if you are hungry for more, AoE is the place to go.