FastFileSystem 0.1.0 [ALPHA] — Unified File Search & USN Journal Watch Engine for Java

High-performance unified filesystem indexing, autocomplete searching, and NTFS USN Journal live-synchronization engine for the JVM.

FastFileSystem is the storage-indexing substrate of the FastJava ecosystem. It unifies FastFileIndex (zero-copy mmap scanning), FastFileSearch (Prefix Trie / N-Gram fuzzy search), and FastFileWatch (NTFS USN Journal live change detection) into a single, cohesive, sub-microsecond Java API—delivering Everything-style file search capabilities without Java heap overhead.

import fastfilesystem.FastFileSystem;
import fastfilesearch.SearchResult;

public class Demo {
    public static void main(String[] args) {
        // 1. Mount directory / volume with real-time USN Journal watching
        try (FastFileSystem fs = FastFileSystem.mount("C:\\Projects")) {
            System.out.printf("Indexed %,d filesystem entries in milliseconds.\n", fs.entryCount());

            // 2. Sub-microsecond prefix autocomplete search
            SearchResult[] results = fs.searchPrefix("pom", 10);
            for (SearchResult r : results) {
                System.out.println("Match: " + r.path() + " (Score: " + r.score() + ")");
            }

            // 3. High-speed fuzzy N-Gram search
            SearchResult[] fuzzyMatches = fs.searchFuzzy("system", 5);
        }
    }
}

• Why FastFileSystem?
• Key Features
• Architecture
• Performance
• Real-World Examples
• API Quick Reference
• Installation
• Technical Examples & Hero Demos
• Documentation
• Platform Support
• Related Projects
• License

Standard Java filesystem operations (java.nio.file.Files.walk, WatchService, or custom trie wrappers) suffer from fundamental bottlenecks:

• Massive JVM Heap Bloat: Traversing 1,000,000 files generates millions of Path, File, and String objects, triggering severe GC pauses.
• Stale In-Memory Search: Static search tries require periodic, CPU-intensive rescans of entire storage trees to reflect file creations, renames, or deletions.
• High-Latency Watchers: java.nio.file.WatchService on Windows relies on polling or slow directory handle reads that drop events under heavy burst I/O.

FastFileSystem solves all three issues simultaneously by unifying FastFileIndex, FastFileSearch, and FastFileWatch:

1. Zero-Copy Memory Mapping: Maps the Master File Table and binary directory index directly via mmap (CreateFileMapping / MapViewOfFile) in native memory.
2. Sub-Microsecond Trie Traversal: Search structures operate directly on contiguous native pointers with zero object allocations during autocomplete queries.
3. Hardware-Level USN Journal Synchronization: Listens directly to the Windows NTFS USN Journal (FSCTL_READ_USN_JOURNAL), applying file events to the in-memory Trie in microseconds without touching the disk.

• ⚡ Zero-Copy mmap Index: Leverages memory-mapped file access (CreateFileMapping / MapViewOfFile) for instant <1-3 ms index loading and traversal.
• 🔍 Sub-Microsecond Trie & N-Gram Search: Autocompletes path prefixes and executes fuzzy substring matches in microseconds with zero allocations during queries.
• 🔄 Real-Time USN Journal Sync: Continuously synchronizes with the Windows NTFS USN Journal, applying additions, modifications, and renames directly to in-memory Trie nodes with zero disk rescans.
• 🛡️ Single Source of Truth: Eliminates multi-module heap duplication by keeping a unified pointer representation across indexer, search trie, and watcher.

Measured on Windows 11 x64 (NVMe SSD) with ~150,000 workspace files.

Autonomous coding agents (FastAIAgent) require instant knowledge of file locations across huge monorepos without IO delays:

// Mount workspace with active USN sync
try (FastFileSystem fs = FastFileSystem.mount("C:\\Workspaces\\FastJava")) {
    // Instant lookup of matching source files
    SearchResult[] matches = fs.searchPrefix("FastWebSpider", 5);
    for (SearchResult match : matches) {
        System.out.println("Located context file: " + match.path());
    }
}

Instant fuzzy filename resolution across millions of files with sub-microsecond latency:

FastFileSystem fs = FastFileSystem.mount("C:\\Projects");
// Substring and fuzzy tolerance matching
SearchResult[] results = fs.searchFuzzy("spiderdemo", 10);

Always-fresh in-memory file index that auto-updates when files are created, renamed, or deleted:

FastFileSystem fs = FastFileSystem.mount(new String[]{ "D:\\GameEngine\\Assets" }, true);
System.out.printf("Tracking %,d assets live via NTFS USN Journal.\n", fs.entryCount());

Add the JitPack repository and the dependencies to your pom.xml:

<repositories>
    <repository>
        <id>jitpack.io</id>
        <url>https://jitpack.io</url>
    </repository>
</repositories>

<dependencies>
    <!-- FastFileSystem Core Engine -->
    <dependency>
        <groupId>com.github.andrestubbe</groupId>
        <artifactId>FastFileSystem</artifactId>
        <version>0.1.0</version>
    </dependency>

    <!-- FastFile Ecosystem Modules -->
    <dependency>
        <groupId>com.github.andrestubbe</groupId>
        <artifactId>FastFileIndex</artifactId>
        <version>0.1.2</version>
    </dependency>
    <dependency>
        <groupId>com.github.andrestubbe</groupId>
        <artifactId>FastFileSearch</artifactId>
        <version>0.1.1</version>
    </dependency>
    <dependency>
        <groupId>com.github.andrestubbe</groupId>
        <artifactId>FastFileWatch</artifactId>
        <version>0.1.0</version>
    </dependency>

    <!-- FastCore (Required Native Loader) -->
    <dependency>
        <groupId>com.github.andrestubbe</groupId>
        <artifactId>FastCore</artifactId>
        <version>0.1.0</version>
    </dependency>
</dependencies>

repositories {
    maven { url 'https://jitpack.io' }
}

dependencies {
    implementation 'com.github.andrestubbe:FastFileSystem:0.1.0'
    implementation 'com.github.andrestubbe:FastFileIndex:0.1.2'
    implementation 'com.github.andrestubbe:FastFileSearch:0.1.1'
    implementation 'com.github.andrestubbe:FastFileWatch:0.1.0'
    implementation 'com.github.andrestubbe:FastCore:0.1.0'
}

Download the latest JARs directly to add them to your classpath:

1. 📦 FastFileSystem-0.1.0.jar (The Core Engine)
2. 🗂️ FastFileIndex-0.1.2.jar (mmap Indexing Substrate)
3. 🔍 FastFileSearch-0.1.1.jar (Trie & N-Gram Search Engine)
4. ⏱️ FastFileWatch-0.1.0.jar (NTFS USN Journal Live Sync)
5. ⚙️ fastcore-0.1.0.jar (The Mandatory Native Loader)

Explore the complete source configurations and benchmarks:

• ⚡ Interactive Live Stream Demo: Demo.java (.\run-demo.bat) — Real-time mounting, prefix autocompletion, and live USN Journal monitoring demo.
• 📈 Multi-Tier Comparison: Benchmark.java (.\run-compare.bat) — Races FastFileSystem against standard Java across 3 tiers (Scan, Prefix Trie, Fuzzy).
• 🚀 OpenJDK JMH Benchmark: FastFileSystemJmhBenchmark.java (.\run-benchmark.bat) — Formal JMH microbenchmarks measuring throughput (ops/ms).
• 🧪 Test Suite: FastFileSystemTest.java — Comprehensive JUnit 5 validation.

Run the hero demo locally from the command line:

.\run-demo.bat

• REFERENCE.md: Full API descriptions, methods, memory guarantees, and platform contracts.
• PHILOSOPHY.md: The architectural rationale for unified zero-copy indexing and live-sync.
• ROADMAP.md: Future milestones, Linux fanotify/inotify, and macOS FSEvents integration.
• CHANGELOG.md: Release history and version migration details.

Combine FastFileSystem with other FastJava accelerators for maximum efficiency:

• FastFileIndex — Binary file indexing with zero-copy mmap support.
• FastFileSearch — Prefix Trie, N-Gram index, and Ranking engine.
• FastFileWatch — NTFS USN Journal-based live file change monitor.
• FastFileContentIndex — High-speed 3-gram bloom filter in-file search.
• FastCore — Native library loader and platform abstraction.

MIT License — See LICENSE for details.

Part of the FastJava Ecosystem — Making the JVM faster.