I was going through Database storage engines and found out about LSM trees. I had also read about Kafka architecture and know that Kafka internally is a commit log. I want to know if Kafka internally uses LSM data structure for append-only store or uses some other data structure for storing data?
Add a comment
|
2 Answers
No, Kafka does not use LSM trees or any tree based data structure for data representation. It relies upon Sequential IO so that it is not relying upon JVM to maintain cache which would then be presenting as an overhead.
Kafka generally implements a queue based structure rather than a tree based structure (memory + merge sort on disk , something like sstables as disk store format).
Details can be found in the Kafka documentation.
-
Thanks @Jim for the answer. It would be beneficial if you also linked to some documentation for comprehensive answer.Ayman Patel– Ayman Patel2021-04-15 05:19:16 +00:00Commented Apr 15, 2021 at 5:19
Adding on to @Jim, LSM trees are used to sort based on key, but in kafka we do not try to sort, we use the same order in which we recieve events. So LSM or something that modifies order won't be needed
