0

I'm looking for a way to work with a huge amount (~10^9) of files (all sizes) in .NET using unspecified file system (NTFS, BTRFS...).

What I have done up to now is to store them in evenly in a folder-tree so that no folder has any more than a couple of thousand files in it. The fs can handle it but there are other problems with this approach. Whenever I need to scan through the files in .NET or do other file operations on them, for instance move to another disc or delete them etc, it takes forever...

I have made some tests combining the files into a few really large files and then index them from a separate hashtable-file, and this shows much promise. Scanning through the "virtual files" is now extremely fast compared to before. But my new problem now is concurrency. I want to be able to read and write (including delete) transactionally to these virtual files as if they were normal files on disc (preferably lock-free). I could continue work on my own "VFS" for this, but something tells me there's already existing options that fits this usecase?

Improve this question
10
  • 3
    You'r'e describing a database. Commented Nov 16, 2022 at 9:42
  • As Mat said, looks like a NoSQL database? To manage requests on files you could have a secondary database for VFS or at least a message queue mechanism. Commented Nov 16, 2022 at 9:53
  • Why not a folder per file? Like a DB row would do. Then your File ID is mainly the file path (aka URL). If you have thousands of files in a single folder and you have many of these folders, you have to loop to find those you need. Commented Nov 16, 2022 at 9:54
  • A filesystem is a hierarchical key-value DB. Sometimes, using an actual DB could make sense instead – and could improve performance significantly. I recommend SQLite if there's only one process. Commented Nov 16, 2022 at 9:55
  • Thanks, yes you are right. Know of any existing wrapper/api over a regular DB (for instance SQLite) that mimics a basic file system such as this? Commented Nov 16, 2022 at 9:57

1 Answer 1

Reset to default
1

multiple threads that simultaneously read, write, delete and also abort writes to these files without blocking or interfering with each other in any way? One file should be able to be read by multiple threads while another thread writes to it and also commits it atomically.

This is quite an onerous set of requirements, and raises the difficulty of getting it right considerably. At this point you're replicating a large amount of the functionality of a filesystem. So your concurrency solution is likely to hit the same issues that made your on-disk filesystem slow in the first place.

I can offer some suggestions:

  1. Weaken the requirements by redesigning the rest of the system. Especially the requirement for simultaneous access within the same file.

  2. Use an actual filesystem, within a file. That is, take the existing C implementation of ext2 (or another choice) use it directly from .NET

  3. Give up on making the software faster and buy faster hardware.

  4. Cluster the system as a whole: instead of trying to store 10^9 files in one place, store 10^8 files in 10 computers.

  5. Weaken the concurrency and durability requirements. If you have to be able to write to the file from lots of writers AND have it consistently read AND have it survive a crash in a consistent state, you're going to be waiting for flushes to durable storage a lot.

Someone has actually done #2: SharpExt4 claims to offer access to ext2/3/4 from within .NET. I've no idea what its performance or concurrency characteristics are. But that would avoid the issues which make NTFS particularly slow for some use cases.

Improve this answer

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.