Slew rate does affect how quickly the circuit responds, but this is a matter of "is it fast enough for the given application?" The upper range of human hearing is 20 KHz. The slew rate of an op amp is typically measured in volts per microsecond. Do the conversion to frequency, that means that you won't bump into this limit until you're in the Megahertz range, much higher than any human can hear. That isn't a "subtle" impact on the sound, it's a negligible one.
And this is exactly why people like op amps for voltage amplifiers. If you operate within the basic parameters, you can essentially eliminate the distorting effects that occur when using discrete designs. This is, of course, very boring compared to discrete designs, which is why we still have tube amplifiers on the market when we can develop transistors that are measured in nanometers.
And it should also be mentioned that op amps aren't necessarily good current sources, so delivering enough power to drive speakers and/or headphones can be a problem (so this is another advantage of discrete amplifier designs).
Well, in my above post, I suppose I didn't mean "voltage demand" but overall "power demand". Maybe it's the characteristic of op amps you describe of not being as flexible in supplying demanded current as they are with voltage that introduces this distortion. This is certainly speculation on my part, but I still think that the reason why some people prefer (or at least claim to) like warmer vs more clinical sound has to do with the introduction of distortion (purposeful) - which is likely to be more prevalent at different frequency ranges due to current demands,etc.
After all, isn't that what is going on when people "roll" (ie, switch) vacuum tubes in pre-amplification/amplification stages with tube amps? Different tubes impart different distortion characteristics...
Interestingly, going back the guy that designed those op amps I linked to previously, in his mid and higher end DACs, he claims that a non-feedback designs sound favorable to most of his customers and so has switched to these designs over the last few years. He fully acknowledged that these designs, as measured, show more distortion. But says that feedback from internal auditioning, and from customers suggests that people prefer the result. Basically, over the years, he's played around with a bunch of different circuit designs, and while measuring their performance analytically, claims that, in the end, he just sticks with what sounds best, regardless of whether they don't measure as well. Quite a different perspective from what you'd find in most electrical engineering applications (and I could see how many engineers would write this off as BS at face value). The main method by which performance is upheld while using a non-feedback design is through the use of something called "current conveyor technology" which is apparently a discrete means of increasing signal gain by converting a current signal directly to voltage without a traditional op amp config in the pre-amplification stage...you can read a bit about it here if interested
(And yes, we are talking about pre-amplification analog output stages. Obviously the main amplification stages are discrete.)