Bach and Beethoven are my favorite composers because of their counterpoint. They are, imo, the greatest composers of polyphony. (Also some Renaissance guys, particularly Monteverdi's early stuff, but that was before the tonality that we know. And Bartok, but he's after the tonality that we know. heh) The way they weave the voices together is, for me, the highest expression of music. The quartet wasn't around in Bach's time, but I guess we know he'd have written lots of fugues if it had been.
The string quartet is maybe the best vehicle for polyphony. (Well, it could be four of any type of instrument. It doesn't necessarily have to be the string quartet, though it's my favorite. Madrigals from the Renaissance are primarily for four or five voices without instrumental accompaniment. To mention Monteverdi again, his madrigals are extraordinary. I particularly love Books IV and V, although he adds some instruments in some of V, and the later books don't truly fit the definition of madrigal any longer. Yes, I digress, but it's to Monteverdi, so all is forgiven.

A symphony can highlight the different tone colors (timbre: the quality of the sound that tells you which instrument you are hearing. Two notes of identical pitch, loudness, and duration, one on a violin and one on a trumpet, sound different.), combining them in a huge number of ways. What would be the point of a symphony that didn't utilize those possibilities?
The quartet, obviously, doesn't have that option. It must do other things. I just happen to prefer those other things, on the whole, to what symphonies do. The texture and intimacy are what do it for me. You said it: either it moves you, or it doesn't. Other than piano lessons, I had no exposure to any kind of classical music before college. Mainly Mozart, but also a child's version of Beethoven's Moonlight Sonata. (Which was still gorgeous!) I don't have any idea how to connect my preference for chamber music to that. But when I got to college, and learned what else had been written, my life had new meaning.
It seems odd even to me, but I feel great power in certain quartets that, for me, rivals the power of a full orchestra. Heck, for that matter, Bach's famous unaccompanied violin chaccone can sometimes make me shake! That kind of power in a single violin? Just amazing.