Discussions

• 1 You must be logged in to vote

  💡

  Load multiple models in a single TGI container

  Rictus started Apr 18, 2025 in Ideas

  

  

  0
• 1 You must be logged in to vote

  🙏

  Is it possible to get all_input_ids in forward or prepare_inputs_for_generation for a custom model?

  Kurokabe asked Apr 14, 2025 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  💡

  Add Sentry / Glitchtip error tracking integration

  andresC98 started Mar 25, 2025 in Ideas

  

  

  0
• 3 You must be logged in to vote

  🙏

  Does HF TGI support Multi Node -Multi GPU server set up ?

  ansSanthoshM asked Feb 14, 2024 in Q&A · Unanswered

  

  

  2
• 1 You must be logged in to vote

  🙏

  Setting up an environment for Langchain development with TGI Containers

  penard-monkey asked Mar 17, 2025 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  🙏

  Keras Support Needed

  chintan-ushur asked Feb 24, 2025 in Q&A · Unanswered

  

  

  0
• 2 You must be logged in to vote

  🙏

  Installation stucked on "build wheel for flash-attn"

  celsowm asked Feb 14, 2025 in Q&A · Unanswered

  

  

  0
• 3 You must be logged in to vote

  🙏

  Huggingface-TGI: Local model not recognized

  n-morgan asked Aug 14, 2024 in Q&A · Unanswered

  

  

  2
• 4 You must be logged in to vote

  🙏

  Is TGI not available for ARM64 platform?

  SMAntony asked Jul 18, 2024 in Q&A · Unanswered

  

  

  1
• 1 You must be logged in to vote

  🙏

  Other generation strategies

  dachenlian asked Dec 11, 2023 in Q&A · Unanswered

  

  

  1
• 1 You must be logged in to vote

  🙏

  How to control request size within a batch?

  cringelord000222 asked Jun 6, 2024 in Q&A · Unanswered

  

  

  

  

  2
• 1 You must be logged in to vote

  🙏

  Question about the temperature parameter in the Hugging Face Inference API

  xufengduan asked Jan 6, 2025 in Q&A · Unanswered

  

  

  1
• 1 You must be logged in to vote

  🙏

  Error for exllamav2_kernel, running TGI on Google Colab

  andychoi98 asked Apr 18, 2024 in Q&A · Unanswered

  

  

  6
• 1 You must be logged in to vote

  🙏

  an beginner for TGI, for to trigger the op which is written by triton, use which model-id, thanks

  alanguo1234 asked Nov 19, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  🙏

  Are there benchmarks comparing Tgi performance on gpus like the rts 2080ti and amd 6800xt ?

  phoenixtomcat asked Nov 16, 2024 in Q&A · Unanswered

  

  

  0
• 3 You must be logged in to vote

  🙏

  Where can I see the chat template that a model is using?

  Simon-Stone asked Oct 22, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  🙏

  Function calling structure

  ankush13r asked Oct 21, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  🙏

  How to truncate using TGI? input tokens exceeded max_input_tokens.

  LanSnowZ asked Oct 16, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  🙏

  Repetition Aware Sampling in VALL-E 2

  zyt1024 asked Oct 9, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  💬

  ModuleNotFoundError: No module named 'ray'

  SMAntony started Sep 26, 2024 in General

  

  

  1
• 2 You must be logged in to vote

  🙏

  /metrics only start returning contents after container's FIRST inference was called?

  cringelord000222 asked Sep 26, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  🙏

  langchain using with_structured_output via OpenAI client results in failure when using HF TGI 2.2.0?

  michael-newsrx asked Sep 10, 2024 in Q&A · Unanswered

  

  

  0
• 1 You must be logged in to vote

  🙏

  Using tools parameter for generation guidance

  Shiniri asked Sep 4, 2024 in Q&A · Unanswered

  

  

  0
• 4 You must be logged in to vote

  🙏

  Llama 3.1 405B with 128k context length

  ChristophRaab asked Aug 9, 2024 in Q&A · Answered

  

  

  2
• 1 You must be logged in to vote

  🙏

  Loading LORA Adapters with specific configuration

  bely66 asked Jul 15, 2024 in Q&A · Unanswered

  

  

  0