Do not delete file at high priority thread pool

[mm304321141]

Running both Flush and Purge operations in the HIGH Priority thread pool poses a risk. Compared to Purge, if Flush becomes slow, it can lead to Write Stop, which is an extremely severe issue. Therefore, Purge operations must use the LOW Priority thread pool.

[mm304321141]

@ajkr @riversand963 @pdillinger @cbi42 Can you maintainers give me some advice on whether this is reasonable? If not, I will not fix the tests.

[andrew-kryczka]

You could mostly bypass the problem by using SstFileManager to delete files in a separate BG thread. It would only fall back to the regular RocksDB threads after deletion falls far behind. At that point, it feels justified for deletion to run in the high-pri pool.

I agree at least that the API is difficult. Ingestion-heavy use cases (especially ones that set avoid_unnecessary_blocking_io) must be hard to tune.

[mm304321141]

You could mostly bypass the problem by using SstFileManager to delete files in a separate BG thread. It would only fall back to the regular RocksDB threads after deletion falls far behind. At that point, it feels justified for deletion to run in the high-pri pool.

I agree at least that the API is difficult. Ingestion-heavy use cases (especially ones that set avoid_unnecessary_blocking_io) must be hard to tune.

This is not a discussion of how to implement it, but a discussion of whether the idea is correct. Whether it is the default behavior or adjusted through parameters is another question.

[andrew-kryczka]

This is not a discussion of how to implement it

There seems to be a misunderstanding. SstFileManager setting is a user API. I was talking about how many users tune RocksDB to not run into the problem you are describing in the first place.