Yes, I have spent a lot of time out in nature and parks, and I generally am aware of not leaving a trace or litter. However, with something like a banana peel, I'd rather it end up as tree food the in a landfill, so I usually toss them in bushes.
The issue is that leaving food for the animals alters their behavior and can cause issues. You're probably not the worst offender wherever you are, as there are many people who just give food to animals, but the argument would be that it contributes to the problem.
Please don't do that with banana or orange peels, they remain present for a long time. I have no problem personally with people tossing apple cores in more remote, little-visited areas (though it still makes sense to crack down on such things a bit more in heavily visited areas like National Parks) but various peels don't rot well and remain in place.
I'm not sure about Yosemite or even nature in the US so maybe this wouldn't be an issue, but even biodegradable things like apple cores or egg shells can spread diseases to local populations of trees or birds. That's a real thing to take into account in natural parks with endemic species at least.
I don't have a link though, it's just something that was been repeated at length by the Natural Reserve agents and the ecobiologists to the personnel and visitors while I was on Kerguelen island (precisely about apple cores and egg shells - most of the vegetation is actually in the same Rosaceae family as apple trees, and there are many local bird species).
It makes sense to me, seeing how a single disease like chestnut blight or phylloxera can be very devastating to a population on a whole continent, it could be both as bad but also less noticeable in a natural park.
