This repository was archived by the owner on Mar 1, 2026. It is now read-only.
Adding script to transfer myrocks snapshot using WDT#168
Closed
uddipta wants to merge 3 commits into
Closed
Conversation
Summary: Adding script to transfer myrocks snapshot using WDT. Myrocks snapshot transfer is currently done using tar/xbstream. WDT is much faster than both of this and has many other features like rate limiting, download resumption, progress reporting etc. I measured transfer speed between a prn host and a lla host. Tar over ssh gives about 55 MBytes/sec. xbstream is even slower. WDT can reach upto 700 MBytes/sec. So, speed wise there is no comparison. WDT has rate limiting support. And WDT calls sync_file_range after each disk write, thereby not affecting disk performance. Regarding rate limiting at rack and cluster level, we have used BTM successfully for other deployments inside facebook for this purpose. Also, WDT transfers are encrypted with AES GCM. Implementation details - We use the "create-checkpoint" api of rocksdb to create a snapshot. For files in other directories like mtr, performance_... etc we create a hard link to the snapshot directory. I am assuming that this files are immutable. We then start a WDT transfer with timeout of 5 minutes. If the transfer does not finish within the timeout, it is aborted and a new snapshot is generated. Download resumption is built into WDT. So, we don't have to remember how much we have sent already. WDT's download resumption works for append-only files. CURRENT is the only file in rocksdb which is not append-only. So, we remove it before every transfer to ensure that it is resend every time. WDT also deletes extra files left over from previous transfers. So, if a file is no longer needed in the latest snapshot, it is removed. Test Plan: Added a test for myrocks snapshot transfer using WDT. I am generating about 250Mb snapshot. Throttle speed - 10Mbps. So the transfer takes about 25sec. Checkpoint interval is 5 sec. So, we end up getting 5/6 rounds of backup before success. Reviewers: yoshinorim Subscribers: ldemailly
Contributor
|
How does mtr behave when WDT is not installed? On Fri, Feb 12, 2016 at 12:28 AM, Uddipta Maity notifications@github.com
Mark Callaghan |
Author
|
How do I disable a test under some condition? Can you provide an example? |
Contributor
|
Committed via https://reviews.facebook.net/D54153 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Detailed description is in the commit message