klient/mount: abstract mount types

[ppknap]

This PR adds mount abstract types.

Depends on: -

Description

This is an index-based machine mount architecture. It is intended to solve all(known) performance and stability problems with current implementation. It will allow for:

• multiple mounts per remote machine.
• two-way synchronization using FUSE for fast local->remote synchronization and Poller for remote->local->remote sync.
• FUSE logic is loosely coupled - it is possible to easily replace/extend it if we want to support different VFS(may be useful for Windows support).
• "lazy" mounts - in order to create complete FS on local machine, we only need to fetch remote index. Other files can be downloaded when requested by Kernel.
• Context based synchronization - all sync processes will be stopped if remote machine is unreachable.
• Star network mounts - local machines A & B can mount directory from machine C and changes made by A will be seen by machine B(not in real time).
• etc.

Architecture

• Mount - abstract representation of single mount.

• Local/remote Index - index of files stored in mounted directory.

• Mounts - storage of mounts per machine.

• Sync - type responsible for syncing remote and local indexes.

• Syncer - interface responsible for transferring files between local and remote machine.

• RSync - RSync process used for syncing files.

• Notifier - interface responsible for creating index changes.

• FUSE - real time FS notification system based on FUSE.

• Poller - periodically checks remote and local index in order to keep them synced.

How Has This Been Tested?

Only data structures - nothing to test.

Screenshots (if appropriate):

none

Types of changes

• New feature (non-breaking change which adds functionality)

[rjeczalik]

When I see rpath I think more about this than remote path. What about just RemotePath string?

[cihangir]

• What is Machine Group
• Are we planning to get the index first from remote machine than the files' contents itself?
• could you elaborate the interaction between remote->local & local->remote changes?

[ppknap]

What is Machine Group

All components not described here are already implemented.

Are we planning to get the index first from remote machine than the files' contents itself?

The workflow is:

• user on local machine creates a mount.
• local klient calls remote one.
• remote klient generates index cache and sends the index back to local.
• local kd:
  • informs the user about the size and number of files to be fetched.
  • based on index data creates FUSE mount.
  • starts RSync process which is intended to fetch the actual content.

I want to fetch index first because it's basically the only thing I need to create FUSE mount(actually the directory view of not existing files that I can "lazy" fetch when requested(future performance improvements))

could you elaborate the interaction between remote->local & local->remote changes?

This will depend on Notifiers strategy. For now I'm working on two:

• FUSE - it's obviously local->remote change notifier. When you interact with FUSE mounted dir, it will generate index.Change "event" and send it to Sync object(which is responsible for throttling duplicated changes and sending them to syncers). Sync will also maintain priority queue(used as producer for Syncer workers) and ensure that large files won't block synchronization. FUSE changes will have higher priority than these produced by other Notifiers -> the local changes is going to override these made on remote machine.

• Pooler - it's responsible for remote->local and partially local->local notifications. It will be asynchronous worker that(by comparing remote and local indexes) tries as much as it can to make remote and local indexes identical(or at least make their entropy as small as possible). It will occasionally compare local index with local directory (partial local->local notifications) to catch events not correctly handled by FUSE. However, this most likely will be moved to separate type that satisfies Notifier interface(just to KISS).

My initial goal is to create very extensible POC which will left us a window for numbers of improvements. There are too many use cases and I can't cover them all. That's why I'm using incremental approach.