Update speculator config & converter to support hidden states indexing

[shanjiaz]

Changes:

• Added support for optional arguments eagle_aux_hidden_state_layer_ids and inference_type.
• Added more robust logic for target_vocab_size. We default on using "t2d" length, if not available, load the config file of verifier model, recursively search the dict for vocab_size. (The search is needed for nested dict. e.g. target_config_dict["text_config"]["vocab_size"] )
• Removed tests for adding verifier embeddings as it's handled on the vllm side now.
• Removed forward pass tests since forward function is defined on the vllm side.

Command used:

speculators convert nvidia/Llama-4-Maverick-17B-128E-Eagle3 \
  --algorithm eagle3 \
  --verifier RedHatAI/Llama-4-Maverick-17B-128E-Instruct-quantized.w4a16 \
  --output-path Llama4-Maverick-Eagle3-Speculators \
  --validate-device cuda:0 \
  --algorithm-kwargs '{"eagle_aux_hidden_state_layer_ids": [1,23,44], "inference_type": "text"}'

Converted checkpoint:

shanjiaz/Llama4-Maverick-Eagle3-Speculators-converted

[github-actions]

📦 Build Artifacts Available
The build artifacts (`.whl` and `.tar.gz`) have been successfully generated and are available for download: https://github.com/vllm-project/speculators/actions/runs/18381300407/artifacts/4227990755.
They will be retained for up to 30 days.
Commit: 1f84913

[rahul-tuli]

The PR looks good, but now since we are removing the forward pass through the model, does it still make sense to keep the --validate/ --validate-device arguments?

[fynnsu]

Added a couple comments

[rahul-tuli]

few questions/nits which can be addressed in a follow up, good work on this, LGTM once we raise the NotImplementedError for forward passes

[rahul-tuli]

LGTM!

[dsikka]

Do we have test cases for multiple decoder layers?

[fynnsu]

One question below which might require a fix.

[rahul-tuli]

LGTM pending quality!

[rahul-tuli]

LGTM!

Added tests and review has been addressed.