Skip to content

Viktor45/dat2json

Repository files navigation

dat2json β€” Decode/restore Xray/Mihomo geoip.dat/geosite.dat Files to JSON/YAML source

Go Version License

dat2json is a high-performance CLI tool for decoding geoip.dat and geosite.dat files used by Xray, Mihomo (Clash Meta), and other V2Ray-based projects. It supports full or filtered export to JSON or YAML, with parallel processing, progress tracking, and multi-file output.

πŸ” Why use this?
Official .dat files are binary and opaque. This tool lets you recover binary geolocation data files β€” without relying on closed-source generators.


✨ Features

  • βœ… Decode both formats:
    • geoip.dat β†’ country β†’ CIDR lists (1.2.3.0/24)
    • geosite.dat β†’ tag β†’ domain rules (domain:, full:, regexp:, keyword:)
  • πŸ“ Flexible output:
    • Single file (-o output.json)
    • One file per tag/country (--output-dir ./export)
  • πŸ“¦ Multiple formats: JSON, YAML (.yaml or .yml)
  • πŸ” Filtering:
    • --tag=google,netflix (for geosite.dat)
    • --country=US,DE,CN (for geoip.dat)
  • πŸ”€ Sorting: --sort for deterministic, readable output
  • πŸ“Š Progress bar: Automatic for large files (>10k entries)
  • ⚑ Parallel export: Up to 32 concurrent writers for --output-dir
  • 🧩 Universal compatibility: Works with:
  • πŸ›‘οΈ Explicit mode selection: No ambiguity β€” you must specify --ip or --site

πŸš€ Installation

Prerequisites

  • Go 1.23 or higher
  • Git

From Source

git clone https://github.com/viktor45/dat2json.git
cd dat2json
go build -o dat2json .

πŸ’‘ Add to PATH:

sudo mv dat2json /usr/local/bin/

πŸ“– Usage

Basic Examples

# Convert geoip.dat to JSON (explicit --ip required)
./dat2json -i geoip.dat --ip -o countries.json

# Export only NL and RU to YAML
./dat2json -i geoip.dat --ip -o eu.yaml --country=NL,RU

# List all tags in geosite.dat
./dat2json -i geosite.dat --site --list-tags

# Export Netflix and Google to separate YAML files
./dat2json -i geosite.dat --site --output-dir ./rules --tag=netflix,google

Full Flag Reference

Flag Description Required
-i FILE Input .dat file βœ… Yes
--ip Treat input as geoip.dat (IP β†’ CIDR) βœ… One of --ip or --site
--site Treat input as geosite.dat (domains β†’ rules) βœ… One of --ip or --site
-o FILE Output file (.json, .yaml, or .yml) ❌
(unless --output-dir or --list-tags)
--output-dir DIR Export each tag/country to DIR/{name}.{ext} ❌
--format FMT Force output format: json or yaml ❌
--tag LIST Comma-separated tags (e.g., google,netflix) ❌
(--site only)
--country LIST Comma-separated ISO 3166-1 alpha-2 codes (e.g., US,DE) ❌
(--ip only)
--list-tags Print all tags in geosite.dat/geoip.dat and exit ❌
(--site only)
--sort Sort keys alphabetically (countries/tags + domains/CIDRs) ❌
-h Show help ❌

⚠️ Notes:

  • Use either -o or --output-dir β€” not both.
  • Country codes are case-insensitive (us = US).
  • Tags are case-insensitive (GOOGLE = google).

πŸ§ͺ Examples

1. Inspect a Custom geosite.dat

# See what’s inside
./dat2json -i custom-geosite.dat --site --list-tags | grep -i "ad"

# Export ad-related tags
./dat2json -i custom-geosite.dat \
  --site \
  --output-dir ./ads \
  --tag=category-ads,ads-all \
  --sort

2. Build a Minimal GeoIP for EU

./dat2json -i geoip.dat \
  --ip \
  -o eu-countries.yaml \
  --country=DE,FR,IT,ES,NL,BE,AT,CH,SE,NO,DK,FI \
  --sort

3. Bulk Export All Countries (Parallel)

./dat2json -i geoip.dat --ip --output-dir ./countries --format json
# β†’ Creates ./countries/US.json, ./countries/CN.json, etc.

4. Work with Protobuf Files (Mihomo Runtime)

# Mihomo's internal protobuf files have no header β€” use explicit mode
./dat2json -i mihomo-geoip.pb --ip -o countries.json
./dat2json -i mihomo-geosite.pb --site --output-dir ./rules

πŸ›  Technical Details

File Format Support

File Signature Content
geoip.dat GEOI (optional) { "US": ["1.2.3.0/24", ...], ... }
geosite.dat GEOS (optional) { "google": ["domain:.google.com", ...], ... }

πŸ’‘ The tool does not rely on signatures β€” it uses the explicit --ip/--site flag to determine the parser.

Output Format

  • JSON: Standard indented JSON.
  • YAML: Clean, human-readable YAML (uses .yaml extension by default; .yml accepted on input).

Performance

  • Parsing: Optimized for speed and memory efficiency.
  • Export: Parallelized (up to 32 goroutines) when using --output-dir.
  • Progress: Shown automatically for files with >10,000 entries.

πŸ§ͺ Testing

Run unit tests:

go test ./...

πŸ”’ Tests cover:

  • Binary and protobuf parsing
  • Format serialization
  • Filtering and sorting
  • Error handling

🀝 Contributing

Contributions are welcome! Please:

  1. Fork the repo
  2. Create your feature branch (git checkout -b feature/AmazingFeature)
  3. Commit your changes (git commit -m 'Add some AmazingFeature')
  4. Push to the branch (git push origin feature/AmazingFeature)
  5. Open a Pull Request

πŸ“„ License

Distributed under the MIT License. See LICENSE for details.


πŸ™ Acknowledgements

  • V2Fly β€” for the original .dat format and community datasets
  • Mihomo (Clash Meta) β€” for the protobuf schema and reference implementation
  • YAML Spec β€” for standardizing .yaml over .yml

πŸ’‘ Pro Tip: Combine with v2fly/domain-list-community and v2fly/geoip to rebuild .dat files after modification!

About

geoip/geosite .dat files recovery tool

Resources

License

Stars

Watchers

Forks

Packages

 
 
 

Contributors

Languages