Submit samples through headless IDA Pro from the CLI

[r0ny123]

Closes #83.

Problem

mcrit client submit always disassembles with SMDA. The IDA plugin can instead upload IDA's own analysis, including the function names FLIRT recovers, but only for the single database open in the GUI. There was no way to run a folder of samples through IDA and get the same result.

My earlier assessment on the issue was that this needed IDA batch tooling outside this repository's dependencies. That no longer holds: smda ships a headless IDA backend (IdaInterface.fromPath, via ida-domain) behind its optional ida extra, so the work reduces to an optional extra here plus a producer for the report.

Fix

--disassembler ida swaps only the step that produces the SmdaReport. The sample is copied to a temporary directory, opened in a headless IDA database, and exported with Disassembler(backend="IDA"), which is the same call the plugin makes. sha256, filename and binary_size are then taken from the file on disk, because the buffer SMDA sees is IDA's reassembled segment image and its hash is not the sample's. Everything after that is the existing submit path, so skipping known samples, --force_update, --output and the family/version derivation of each mode behave as they do with SMDA.

--ida-sigs DIR points at an unpacked Hex-Rays FLIRT Signature Bundle. Bundles are not applied by IDA on its own, and the current one holds about 4,500 signatures, so candidates are first narrowed by file format, architecture and Go/Rust detection. Each candidate is applied behind an undo point and kept only if it names at least --ida-sig-min-matches functions (default 10); otherwise it is undone. The threshold exists because a few coincidental hits would otherwise reach MCRIT as wrong function labels.

Sample	Candidates tried
PE	windows/**/*_<x64|x86|arm64|arm>.sig, at most 10
ELF	linux/**/*-<debian arch>.sig: about 60 for x86, x64 and AArch64, 116 for 32-bit ARM (armhf, armel and arm together)
Go binary	additionally golang/stdlibs/golang_std_<pc|arm|arm64>_*.sig
Rust binary	additionally rust/rust_bundle_<triple>.sig matching architecture and OS
anything else	none; the report is still produced

One subprocess per file. The headless IDA library holds a single database per process, and one malformed sample must not end a batch, so --worker is switched on automatically for the dir, recursive and malpedia modes. A missing ida-domain package stops the run once, with an install hint, before any file is touched.

Reports carry smda_version MCRIT4IDA cli via SMDA <version>. MongoDbStorage already recognises that prefix and reads the last token as the smda version, so the string has to end with it; a test pins that.

Bugs found on the way

Running a directory through IDA against a live server exposed defects in the existing submit code. Each has its own changelog entry and test.

Defect	Effect	Since
Stray continue in _handle_submit_recursive	--mode recursive printed every file and submitted none	before this branch
--server / --apitoken appended after submit in the worker command	every worker exited with a usage error whenever either option was given	before this branch
Timed-out worker reported but not killed	orphaned processes accumulate; with IDA each holds memory and a licence seat	before this branch
--force_update not forwarded to workers	known samples were always skipped under --worker	before this branch
IDA loads a file of no known format as a raw binary	a file of random bytes was stored as a sample with no functions; such reports are now skipped	new with ida

smda 4.8.0

The floor moves from >=4.2.13 to >=4.8.0, as its own commit. smda 4.5.0 moved ESCAPER_DOWNWARD_COMPATIBILITY from 1.13.16 to 4.4.5, so samples hashed under smda 4.4.4 or older are reported stale after upgrading. That report is accurate, and repair_minhashes brings them current.

.NET samples are the exception. smda 4.4.5 ends CIL blocks at throw, rethrow, endfinally and endfilter, which changes the stored report's structure rather than its escaping. No recalculation from a stored report repairs that; affected samples have to be submitted again. The changelog entry says so.

Limits

FLIRT results reach MCRIT only as function names, which the server stores as function labels attributed to the submitting user. There is no per-function library flag: is_library stays a per-sample property, and changing that would need a schema change in SMDA and in both storage backends. Filename base-address suffixes are ignored under ida, since IDA's loader decides the base address. IDA older than 9.1 is not supported.

Reproduce

pip install -e ".[dev,ida]"
export IDADIR="/path/to/IDA Professional 9.3.app/Contents/MacOS"
mcrit client submit ./samples --mode dir --disassembler ida --ida-sigs /path/to/signatures-bundle -f somefamily -o ./reports -t 1800

Checked against IDA Pro 9.3 on macOS with a server on memory storage: a directory of two PE files and one file of random bytes took about 20 seconds. Both PE files were stored under their on-disk sha256, the random file was skipped, no IDA database files appeared beside the samples, and a second run skipped everything. Re-running with -u and a different family updated the stored samples through the workers. Undo of a rejected signature was confirmed separately inside a headless session: the signature count went from 2 to 3 on apply and back to 2 on undo.

Not measured: a sample where the bundle adds names beyond IDA's stock signatures. The two PE files were small launchers the stock signatures already name completely (225 of 251 functions), so the one signature that passed the threshold, Microsoft.Win10SDK_x64.sig with 18 matches, changed no names. No large binary and no ELF, Go or Rust sample was run.

Tests

ruff format, ruff check and ty check are clean, and pytest -m 'not mongo' passes (235). The new tests need neither IDA nor a database: signature selection runs against a fake bundle tree, signature application against stub IDA modules (including keep, drop, keep in sequence), and the console tests patch the report producer. The suite also passes with ida-domain uninstalled, which is the CI situation.

[r0ny123]

Merged main (up to 1.12.0) in, and added tests/testSmdaFloor.py. It pins down what raising the floor to smda 4.8.0 relies on:

• the smda 1.5.12 / 4.2.16 reports under tests/ still carry every field smda 4.8.0's SmdaFunction.fromDict requires, and load;
• the single ESCAPER_DOWNWARD_COMPATIBILITY threshold still covers every per-architecture pic_hash gate;
• the MCRIT4IDA cli via SMDA <version> string parses as current in recalculateAllPicHashes.

On smda 4.8.0 the full suite passes on the merged head (522 tests).