I suggest you offer a self-deprecatory answer. Your answer of "I work hard" suggests that you feel you work harder than your peers. That might even be the case. If so, you probably spend far less of your day on non-work related activities. If you just want a way to answer without inciting envy or sounding like you're too full of yourself, try:
- How do you publish so many papers a year?
- Easy, I have no life.
This has the benefit of i) being self-deprecatory instead of tooting your own horn; ii) being a light and jokular answer; iii) suggesting that there's a trade-off, you loose something for what you gain in output.
I don't care if you're Feynman himself, stating that you work harder/better than your peers will never be well received, even if it's true. In fact, the truer it is, the likelier that people will take offense at it. So, instead of answering along the lines of "I'm better than you", give an answer that highlights that you're losing out on something.