Since these are the opening lines to the song, there is no antecedent to the pronoun "you". While later verses seem to maybe make the song addressed to a love interest with whom the singer has co-habitated, the lack of an antecedent in these opening lines means that "you" seems to be addressed to any random listener. It is rude and/or presumptuous to confront any possible random listener with the leading question "But you don't really care for music, do ya?". If anything, the fact that one is listening to the song suggests that that one DOES care for music.