Assuming that the papers are really strong and the best you can give (i.e. your high productivity does not trade off against the potential even higher quality research): having had the luck to meet quite some fantastically brilliant people in my career, it is clear to me that hard work may be a necessary, but is far from a sufficient condition for high-quality productivity in a difficult field.
If you belong to this class of people that pick just the right approach, just the right question, just the right set of methods and have a knack for actually finding the low-hanging fruits (if indeed they are), good on you!
However, this means that "hard work" is simply not the appropriate response, other people may be working very hard, too. So this statement sends the message that you refuse to tell them your secret of success. This is perfectly in your right, and they are actually quite shamelessly "invading your territory" by asking you how you do it. Nevertheless, it may be advisable to apply some diplomacy to alleviate this; lower your profile, statements such as "I got lucky" or "I had a good run", or, as mentioned earlier, humour, for instance, to create a screen to seemingly decouple the success from you. Obviously, you shouldn't say that when you interview for a position or report to your boss, but such a strategy may take the sting of the interaction with your equal-status colleagues.
Frankly, your colleagues can find themselves lucky to have such a brilliant co-scientist. This reflects on the whole department. However, unfortunately, it may also endanger the career of some of them in very competitive environments and so you may be a realistic existential danger to them - but don't take it personally. This is not your fault and you should not be tempted to reduce your productivity/quality just to be likeable! Just don't rub your success (and certainly not "hard work") under their noses.
And make sure to demonstrate in your CV that the quality of your output is not trading off against output quantity.
Finally, concerning your worry about hiring committees: I have heard about cases of first-rate people being rejected in interviews because of the reason that they were effectively too good (this is a real case, I am intentionally not using the original phrasing here). You probably don't want to be hired by these departments, anyway. Always remember: first-rate department heads hire first-rate people, second-rate department heads hire third-rate people.