Skip to content

bug(vision): image analysis should recover gracefully when primary vision path fails #1034

@atuida

Description

@atuida

Summary

Image understanding in Hermes can fail with a generic message like:

"There was a problem with the request and the image could not be analyzed."

In practice this leaves the agent unable to inspect user-sent images, even when the rest of the system is otherwise working.

Why this matters

For chat/gateway use, image handling is not optional. If a user sends an image, Hermes should either:

  • analyze it successfully, or
  • degrade gracefully with a useful fallback path

A generic failure with no retry/fallback makes image support feel unreliable.

Observed behavior

  • User sends image in chat
  • vision_analyze can fail with a generic request error
  • browser_vision may also be unavailable or fail depending on runtime
  • Agent is left without a reliable way to inspect the image

Expected behavior

When primary image analysis fails, Hermes should have a clear recovery path, for example:

  1. retry with auxiliary vision client
  2. fall back to another available vision backend/tool
  3. surface a precise error (missing dependency / backend unavailable / unsupported image path)
  4. avoid vague "could not be analyzed" messaging when the failure reason is knowable

Requested improvement

  • Add robust fallback/retry behavior for image analysis
  • Improve diagnostics so the agent can tell whether the issue is backend, dependency, path, or provider support
  • Prefer graceful degradation over generic vision failure

User impact

This is especially visible on messaging platforms like Telegram, where image understanding is a core expectation. Hermes should be able to reliably inspect user-sent images or fail in a much more actionable way.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions