When people comment on downvotes, I always use it as an opportunity to learn up on the subject as my knowledge was obviously lacking in some way. I'll never retaliate by downvoting them, even if their comment was something along the lines of "That's wrong jackass".

If you have a problem with people retaliating against you, either there is something wrong in the way you are interacting with people or you are interacting with the wrong people.

After downvotes with no comments, my next biggest issue with voting is sympathy voting. Voting should be done on the worth of the question/answer alone. Not to correct some perceived injustice.