Disclaimer: in this answer, I'm assuming comments' content is not offensive, insulting, or otherwise not in keeping with the spirit of the "Be nice" guideline.
I think there are two things to tackle here:
- automatic deletion of comments;
- manual deletion of comments and removal of some discussions.
Should we automatically delete comments after some time? Absolutely not. While it is true that extended discussion in comments is discouraged universally across the Stack Exchange network, there is still value in comments that add supporting information. Ideally that information should be edited into the answer, but that doesn't always happen.
Comments on questions serve the purpose of someone being able to add a helpful hint that might just be a guess or otherwise not fit to stand on its own as an answer.
There is absolutely no way to automatically or manually ensure that all information given in a comment is preserved in a question or an answer. This leaves us with some comments that are more noise than signal. Those we have to remove as we find them. When I say "we" here I mean both the community and the moderators. As moderators, we typically react to flags on comments, but given enough flags on a comment, it will be removed automatically without our intervention.
Comments have always been and probably will continue to be second-class citizens on the network. That is entirely by design and that's shown in the limitations that comments have:
- comments have a length limit;
- there is no downvoting on comments, so they are not subject to the usual community vetting of their quality;
- comments do not affect reputation;
- comments are collapsed when their number exceeds 5;
- moderators see an automated "more than 20 comments posted" flag on posts that generate a lot of comments in a short amount of time.
The design of each site places primary focus on questions and answers. We want all useful information required to answer a question to ideally be in an answer. How that answer was developed is largely unimportant so long as the final product of that development is valuable. How valuable it is can be determined through up and down votes on the answer.
Stack Exchange set out to make the internet better by creating an environment where the best information rises to the top. Asking the user to dig through a lengthy comment thread on the off chance that the information you're looking for is there goes completely counter to that mission. If we're going to allow that, we might as well just abandon this whole Q&A idea and go back to forums where readers can already sift through pages of unrelated or possibly contradictory information.
For an example, compare the comments between Rook and me on his answer. Is it really beneficial to have our discussion there or is it better for him to express his point in his answer and for me to express mine here? In my opinion, reading one coherent answer is infinitely better than reading a string of broken up comments. Granted, meta posts are a bit of a special case since here we encourage discussion, but I think this still illustrates my point.
I do think we can be better about defining extended discussions and clearing up comment threads. We can discuss that here.
I know it can be difficult to watch your comments be removed and I apologize for that. I know the chat isn't perfect in many ways, and I apologize for that too. But at the end of the day, comments are designed to be transient.
For what it's worth, as moderators we typically only respond to comment flags from community members and to the automatic "more than 20 comments posted" flag as those are usually a good indication that something might be going off-track. It's not typical for posts to generate large amounts of comments in a short amount of time. We do not (far as I know) go out looking for comments to delete.
