Menyajikan LLM dengan GKE Inference Gateway

Tetap teratur dengan koleksi Simpan dan kategorikan konten berdasarkan preferensi Anda.

Mendapatkan akses ke model

Untuk men-deploy model Llama3.1 ke GKE, tanda tangani perjanjian izin lisensi dan buat token akses Hugging Face.

Menandatangani perjanjian izin lisensi

Anda harus menandatangani perjanjian izin untuk menggunakan model Llama3.1. Ikuti petunjuk berikut:

1. Akses halaman izin dan verifikasi izin untuk menggunakan akun Hugging Face Anda.
2. Setujui persyaratan model.

Membuat token akses

Untuk mengakses model melalui Hugging Face, Anda memerlukan token Hugging Face.

Ikuti langkah-langkah berikut untuk membuat token baru jika Anda belum memilikinya:

1. Klik Profil Anda > Setelan > Token Akses.
2. Pilih New Token.
3. Tentukan nama pilihan Anda dan peran minimal Read.
4. Pilih Buat token.
5. Salin token yang dihasilkan ke papan klip Anda.

Menyiapkan lingkungan Anda

Dalam tutorial ini, Anda akan menggunakan Cloud Shell untuk mengelola resource yang dihosting diGoogle Cloud. Cloud Shell telah diinstal dengan software yang Anda perlukan untuk tutorial ini, termasuk kubectl dan gcloud CLI.

Untuk menyiapkan lingkungan Anda dengan Cloud Shell, lakukan langkah-langkah berikut:

1. Di konsol Google Cloud, luncurkan sesi Cloud Shell dengan mengklik Aktifkan Cloud Shell di konsol Google Cloud. Tindakan ini akan meluncurkan sesi di panel bawah Konsol Google Cloud.

2. Tetapkan variabel lingkungan default:

   gcloud config set project PROJECT_ID
   export PROJECT_ID=$(gcloud config get project)
   export REGION=REGION
   export CLUSTER_NAME=CLUSTER_NAME
   export HF_TOKEN=HF_TOKEN
   

   Ganti nilai berikut:

   • PROJECT_ID: Project ID Google CloudAnda.
   • REGION: region yang mendukung jenis akselerator yang ingin Anda gunakan, misalnya, us-central1 untuk GPU H100.
   • CLUSTER_NAME: nama cluster Anda.
   • HF_TOKEN: token Hugging Face yang Anda buat sebelumnya.

Membuat dan mengonfigurasi resource Google Cloud

Untuk membuat resource yang diperlukan, gunakan petunjuk ini.

Membuat cluster dan node pool GKE

Menayangkan LLM di GPU dalam cluster GKE Autopilot atau Standard. Sebaiknya gunakan cluster Autopilot untuk pengalaman Kubernetes yang dikelola sepenuhnya. Untuk memilih mode operasi GKE yang paling sesuai untuk workload Anda, lihat Memilih mode operasi GKE.

gcloud container clusters create-auto CLUSTER_NAME \
    --project=PROJECT_ID \
    --region=REGION \
    --release-channel=rapid \
    --cluster-version=1.32.3-gke.1170000

1. Untuk menyiapkan otorisasi guna meng-scrap metrik, buat secret inference-gateway-sa-metrics-reader-secret:

   kubectl apply -f - <<EOF
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRole
   metadata:
     name: inference-gateway-metrics-reader
   rules:
   - nonResourceURLs:
     - /metrics
     verbs:
     - get
   ---
   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: inference-gateway-sa-metrics-reader
     namespace: default
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRoleBinding
   metadata:
     name: inference-gateway-sa-metrics-reader-role-binding
     namespace: default
   subjects:
   - kind: ServiceAccount
     name: inference-gateway-sa-metrics-reader
     namespace: default
   roleRef:
     kind: ClusterRole
     name: inference-gateway-metrics-reader
     apiGroup: rbac.authorization.k8s.io
   ---
   apiVersion: v1
   kind: Secret
   metadata:
     name: inference-gateway-sa-metrics-reader-secret
     namespace: default
     annotations:
       kubernetes.io/service-account.name: inference-gateway-sa-metrics-reader
   type: kubernetes.io/service-account-token
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRole
   metadata:
     name: inference-gateway-sa-metrics-reader-secret-read
   rules:
   - resources:
     - secrets
     apiGroups: [""]
     verbs: ["get", "list", "watch"]
     resourceNames: ["inference-gateway-sa-metrics-reader-secret"]
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRoleBinding
   metadata:
     name: gmp-system:collector:inference-gateway-sa-metrics-reader-secret-read
     namespace: default
   roleRef:
     name: inference-gateway-sa-metrics-reader-secret-read
     kind: ClusterRole
     apiGroup: rbac.authorization.k8s.io
   subjects:
   - name: collector
     namespace: gmp-system
     kind: ServiceAccount
   EOF
   

Membuat secret Kubernetes untuk kredensial Hugging Face

Di Cloud Shell, lakukan hal berikut:

1. Untuk berkomunikasi dengan cluster Anda, konfigurasikan kubectl:

     gcloud container clusters get-credentials CLUSTER_NAME \
         --location=REGION
   

   Ganti nilai berikut:

   • REGION: region yang mendukung jenis akselerator yang ingin Anda gunakan, misalnya, us-central1 untuk GPU H100.
   • CLUSTER_NAME: nama cluster Anda.

2. Buat Secret Kubernetes yang berisi token Hugging Face:

     kubectl create secret generic HF_SECRET \
         --from-literal=token=HF_TOKEN \
         --dry-run=client -o yaml | kubectl apply -f -
   

   Ganti kode berikut:

   • HF_TOKEN: token Hugging Face yang Anda buat sebelumnya.
   • HF_SECRET: nama untuk secret Kubernetes Anda. Contoh, hf-secret.

Menginstal CRD InferenceModel dan InferencePool

Di bagian ini, Anda akan menginstal Definisi Resource Kustom (CRD) yang diperlukan untuk GKE Inference Gateway.

CRD memperluas Kubernetes API. Hal ini memungkinkan Anda menentukan jenis resource baru. Untuk menggunakan GKE Inference Gateway, instal CRD InferencePool dan InferenceModel di cluster GKE Anda dengan menjalankan perintah berikut:

kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/releases/download/v0.3.0/manifests.yaml

Men-deploy server model

Contoh ini men-deploy model Llama3.1 menggunakan server model vLLM. Deployment diberi label sebagai app:vllm-llama3-8b-instruct. Deployment ini juga menggunakan dua adaptor LoRA bernama food-review dan cad-fabricator dari Hugging Face. Anda dapat mengupdate deployment ini dengan server model dan penampung model, port penayangan, dan nama deployment Anda sendiri. Anda dapat mengonfigurasi adaptor LoRA secara opsional dalam deployment, atau men-deploy model dasar.

1. Untuk men-deploy pada jenis akselerator nvidia-h100-80gb, simpan manifes berikut sebagai vllm-llama3-8b-instruct.yaml. Manifes ini menentukan Deployment Kubernetes dengan model dan server model Anda:

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: vllm-llama3-8b-instruct
   spec:
     replicas: 3
     selector:
       matchLabels:
         app: vllm-llama3-8b-instruct
     template:
       metadata:
         labels:
           app: vllm-llama3-8b-instruct
       spec:
         containers:
           - name: vllm
             image: "vllm/vllm-openai:latest"
             imagePullPolicy: Always
             command: ["python3", "-m", "vllm.entrypoints.openai.api_server"]
             args:
             - "--model"
             - "meta-llama/Llama-3.1-8B-Instruct"
             - "--tensor-parallel-size"
             - "1"
             - "--port"
             - "8000"
             - "--enable-lora"
             - "--max-loras"
             - "2"
             - "--max-cpu-loras"
             - "12"
             env:
               # Enabling LoRA support temporarily disables automatic v1, we want to force it on
               # until 0.8.3 vLLM is released.
               - name: VLLM_USE_V1
                 value: "1"
               - name: PORT
                 value: "8000"
               - name: HUGGING_FACE_HUB_TOKEN
                 valueFrom:
                   secretKeyRef:
                     name: hf-token
                     key: token
               - name: VLLM_ALLOW_RUNTIME_LORA_UPDATING
                 value: "true"
             ports:
               - containerPort: 8000
                 name: http
                 protocol: TCP
             lifecycle:
               preStop:
                 # vLLM stops accepting connections when it receives SIGTERM, so we need to sleep
                 # to give upstream gateways a chance to take us out of rotation. The time we wait
                 # is dependent on the time it takes for all upstreams to completely remove us from
                 # rotation. Older or simpler load balancers might take upwards of 30s, but we expect
                 # our deployment to run behind a modern gateway like Envoy which is designed to
                 # probe for readiness aggressively.
                 sleep:
                   # Upstream gateway probers for health should be set on a low period, such as 5s,
                   # and the shorter we can tighten that bound the faster that we release
                   # accelerators during controlled shutdowns. However, we should expect variance,
                   # as load balancers may have internal delays, and we don't want to drop requests
                   # normally, so we're often aiming to set this value to a p99 propagation latency
                   # of readiness -> load balancer taking backend out of rotation, not the average.
                   #
                   # This value is generally stable and must often be experimentally determined on
                   # for a given load balancer and health check period. We set the value here to
                   # the highest value we observe on a supported load balancer, and we recommend
                   # tuning this value down and verifying no requests are dropped.
                   #
                   # If this value is updated, be sure to update terminationGracePeriodSeconds.
                   #
                   seconds: 30
                 #
                 # IMPORTANT: preStop.sleep is beta as of Kubernetes 1.30 - for older versions
                 # replace with this exec action.
                 #exec:
                 #  command:
                 #  - /usr/bin/sleep
                 #  - 30
             livenessProbe:
               httpGet:
                 path: /health
                 port: http
                 scheme: HTTP
               # vLLM's health check is simple, so we can more aggressively probe it.  Liveness
               # check endpoints should always be suitable for aggressive probing.
               periodSeconds: 1
               successThreshold: 1
               # vLLM has a very simple health implementation, which means that any failure is
               # likely significant. However, any liveness triggered restart requires the very
               # large core model to be reloaded, and so we should bias towards ensuring the
               # server is definitely unhealthy vs immediately restarting. Use 5 attempts as
               # evidence of a serious problem.
               failureThreshold: 5
               timeoutSeconds: 1
             readinessProbe:
               httpGet:
                 path: /health
                 port: http
                 scheme: HTTP
               # vLLM's health check is simple, so we can more aggressively probe it.  Readiness
               # check endpoints should always be suitable for aggressive probing, but may be
               # slightly more expensive than readiness probes.
               periodSeconds: 1
               successThreshold: 1
               # vLLM has a very simple health implementation, which means that any failure is
               # likely significant,
               failureThreshold: 1
               timeoutSeconds: 1
             # We set a startup probe so that we don't begin directing traffic or checking
             # liveness to this instance until the model is loaded.
             startupProbe:
               # Failure threshold is when we believe startup will not happen at all, and is set
               # to the maximum possible time we believe loading a model will take. In our
               # default configuration we are downloading a model from HuggingFace, which may
               # take a long time, then the model must load into the accelerator. We choose
               # 10 minutes as a reasonable maximum startup time before giving up and attempting
               # to restart the pod.
               #
               # IMPORTANT: If the core model takes more than 10 minutes to load, pods will crash
               # loop forever. Be sure to set this appropriately.
               failureThreshold: 3600
               # Set delay to start low so that if the base model changes to something smaller
               # or an optimization is deployed, we don't wait unnecessarily.
               initialDelaySeconds: 2
               # As a startup probe, this stops running and so we can more aggressively probe
               # even a moderately complex startup - this is a very important workload.
               periodSeconds: 1
               httpGet:
                 # vLLM does not start the OpenAI server (and hence make /health available)
                 # until models are loaded. This may not be true for all model servers.
                 path: /health
                 port: http
                 scheme: HTTP
   
             resources:
               limits:
                 nvidia.com/gpu: 1
               requests:
                 nvidia.com/gpu: 1
             volumeMounts:
               - mountPath: /data
                 name: data
               - mountPath: /dev/shm
                 name: shm
               - name: adapters
                 mountPath: "/adapters"
         initContainers:
           - name: lora-adapter-syncer
             tty: true
             stdin: true
             image: us-central1-docker.pkg.dev/k8s-staging-images/gateway-api-inference-extension/lora-syncer:main
             restartPolicy: Always
             imagePullPolicy: Always
             env:
               - name: DYNAMIC_LORA_ROLLOUT_CONFIG
                 value: "/config/configmap.yaml"
             volumeMounts: # DO NOT USE subPath, dynamic configmap updates don't work on subPaths
             - name: config-volume
               mountPath:  /config
         restartPolicy: Always
   
         # vLLM allows VLLM_PORT to be specified as an environment variable, but a user might
         # create a 'vllm' service in their namespace. That auto-injects VLLM_PORT in docker
         # compatible form as `tcp://<IP>:<PORT>` instead of the numeric value vLLM accepts
         # causing CrashLoopBackoff. Set service environment injection off by default.
         enableServiceLinks: false
   
         # Generally, the termination grace period needs to last longer than the slowest request
         # we expect to serve plus any extra time spent waiting for load balancers to take the
         # model server out of rotation.
         #
         # An easy starting point is the p99 or max request latency measured for your workload,
         # although LLM request latencies vary significantly if clients send longer inputs or
         # trigger longer outputs. Since steady state p99 will be higher than the latency
         # to drain a server, you may wish to slightly this value either experimentally or
         # via the calculation below.
         #
         # For most models you can derive an upper bound for the maximum drain latency as
         # follows:
         #
         #   1. Identify the maximum context length the model was trained on, or the maximum
         #      allowed length of output tokens configured on vLLM (llama2-7b was trained to
         #      4k context length, while llama3-8b was trained to 128k).
         #   2. Output tokens are the more compute intensive to calculate and the accelerator
         #      will have a maximum concurrency (batch size) - the time per output token at
         #      maximum batch with no prompt tokens being processed is the slowest an output
         #      token can be generated (for this model it would be about 100ms TPOT at a max
         #      batch size around 50)
         #   3. Calculate the worst case request duration if a request starts immediately
         #      before the server stops accepting new connections - generally when it receives
         #      SIGTERM (for this model that is about 4096 / 10 ~ 40s)
         #   4. If there are any requests generating prompt tokens that will delay when those
         #      output tokens start, and prompt token generation is roughly 6x faster than
         #      compute-bound output token generation, so add 20% to the time from above (40s +
         #      16s ~ 55s)
         #
         # Thus we think it will take us at worst about 55s to complete the longest possible
         # request the model is likely to receive at maximum concurrency (highest latency)
         # once requests stop being sent.
         #
         # NOTE: This number will be lower than steady state p99 latency since we stop       receiving
         #       new requests which require continuous prompt token computation.
             # NOTE: The max timeout for backend connections from gateway to model servers should
         #       be configured based on steady state p99 latency, not drain p99 latency
         #
         #   5. Add the time the pod takes in its preStop hook to allow the load balancers have
         #      stopped sending us new requests (55s + 30s ~ 85s)
         #
         # Because the termination grace period controls when the Kubelet forcibly terminates a
         # stuck or hung process (a possibility due to a GPU crash), there is operational safety
         # in keeping the value roughly proportional to the time to finish serving. There is also
         # value in adding a bit of extra time to deal with unexpectedly long workloads.
         #
         #   6. Add a 50% safety buffer to this time since the operational impact should be low
         #      (85s * 1.5 ~ 130s)
         #
         # One additional source of drain latency is that some workloads may run close to
         # saturation and have queued requests on each server. Since traffic in excess of the
         # max sustainable QPS will result in timeouts as the queues grow, we assume that failure
         # to drain in time due to excess queues at the time of shutdown is an expected failure
         # mode of server overload. If your workload occasionally experiences high queue depths
         # due to periodic traffic, consider increasing the safety margin above to account for
         # time to drain queued requests.
         terminationGracePeriodSeconds: 130
         nodeSelector:
           cloud.google.com/gke-accelerator: "nvidia-h100-80gb"
         volumes:
           - name: data
             emptyDir: {}
           - name: shm
             emptyDir:
               medium: Memory
           - name: adapters
             emptyDir: {}
           - name: config-volume
             configMap:
               name: vllm-llama3-8b-adapters
   ---
   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: vllm-llama3-8b-adapters
   data:
     configmap.yaml: |
         vLLMLoRAConfig:
           name: vllm-llama3.1-8b-instruct
           port: 8000
           defaultBaseModel: meta-llama/Llama-3.1-8B-Instruct
           ensureExist:
             models:
             - id: food-review
               source: Kawon/llama3.1-food-finetune_v14_r8
             - id: cad-fabricator
               source: redcathode/fabricator
   ---
   kind: HealthCheckPolicy
   apiVersion: networking.gke.io/v1
   metadata:
     name: health-check-policy
     namespace: default
   spec:
     targetRef:
       group: "inference.networking.x-k8s.io"
       kind: InferencePool
       name: vllm-llama3-8b-instruct
     default:
       config:
         type: HTTP
         httpHealthCheck:
             requestPath: /health
             port: 8000
   

2. Terapkan manifes ke cluster Anda:

   kubectl apply -f vllm-llama3-8b-instruct.yaml
   

Membuat resource InferencePool

Resource kustom Kubernetes InferencePool menentukan grup Pod dengan LLM dasar dan konfigurasi komputasi yang sama.

Resource kustom InferencePool menyertakan kolom kunci berikut:

• selector: menentukan Pod mana yang termasuk dalam kumpulan ini. Label dalam pemilih ini harus sama persis dengan label yang diterapkan ke Pod server model Anda.
• targetPort: menentukan port yang digunakan oleh server model dalam Pod.

Resource InferencePool memungkinkan GKE Inference Gateway merutekan traffic ke Pod server model Anda.

Untuk membuat InferencePool menggunakan Helm, lakukan langkah-langkah berikut:

helm install vllm-llama3-8b-instruct \
  --set inferencePool.modelServers.matchLabels.app=vllm-llama3-8b-instruct \
  --set provider.name=gke \
  --version v0.3.0 \
  oci://registry.k8s.io/gateway-api-inference-extension/charts/inferencepool

Ubah kolom berikut agar cocok dengan Deployment Anda:

• inferencePool.modelServers.matchLabels.app: kunci label yang digunakan untuk memilih Pod server model Anda.

Perintah ini membuat objek InferencePool yang secara logis merepresentasikan deployment server model Anda dan mereferensikan layanan endpoint model dalam Pod yang dipilih Selector.

Membuat resource InferenceModel dengan tingkat kepentingan penayangan

Resource kustom Kubernetes InferenceModel menentukan model tertentu, termasuk model yang disesuaikan LoRA, dan tingkat kepentingan penayangannya.

Resource kustom InferenceModel menyertakan kolom kunci berikut:

• modelName: menentukan nama model dasar atau adaptor LoRA.
• Criticality: menentukan tingkat kepentingan penayangan model.
• poolRef: mereferensikan InferencePool tempat model ditayangkan.

InferenceModel memungkinkan GKE Inference Gateway merutekan traffic ke Pod server model Anda berdasarkan nama dan tingkat kepentingan model.

Untuk membuat InferenceModel, lakukan langkah-langkah berikut:

1. Simpan manifes contoh berikut sebagai inferencemodel.yaml:

   apiVersion: inference.networking.x-k8s.io/v1alpha2
   kind: InferenceModel
   metadata:
     name: inferencemodel-sample
   spec:
     modelName: MODEL_NAME
     criticality: CRITICALITY
     poolRef:
       name: INFERENCE_POOL_NAME
   

   Ganti kode berikut:

   • MODEL_NAME: nama model dasar atau adaptor LoRA Anda. Contoh, food-review.
   • CRITICALITY: tingkat kepentingan penayangan yang dipilih. Pilih dari Critical, Standard, atau Sheddable. Contoh, Standard.
   • INFERENCE_POOL_NAME: nama InferencePool yang Anda buat di langkah sebelumnya. Contohnya, vllm-llama3-8b-instruct.

2. Terapkan manifes contoh ke cluster Anda:

   kubectl apply -f inferencemodel.yaml
   

Contoh berikut membuat objek InferenceModel yang mengonfigurasi model LoRA food-review di vllm-llama3-8b-instruct InferencePool dengan kritisitas penayangan Standard. Objek InferenceModel juga mengonfigurasi model dasar untuk ditayangkan dengan tingkat prioritas Critical.

apiVersion: inference.networking.x-k8s.io/v1alpha2
kind: InferenceModel
metadata:
  name: food-review
spec:
  modelName: food-review
  criticality: Standard
  poolRef:
    name: vllm-llama3-8b-instruct
  targetModels:
  - name: food-review
    weight: 100

---
apiVersion: inference.networking.x-k8s.io/v1alpha2
kind: InferenceModel
metadata:
  name: llama3-base-model
spec:
  modelName: meta-llama/Llama-3.1-8B-Instruct
  criticality: Critical
  poolRef:
    name: vllm-llama3-8b-instruct

Membuat Gateway

Resource Gateway berfungsi sebagai titik entri untuk traffic eksternal ke dalam cluster Kubernetes Anda. Class ini menentukan pemroses yang menerima koneksi masuk.

GKE Inference Gateway mendukung Class Gateway gke-l7-rilb dan gke-l7-regional-external-managed. Untuk mengetahui informasi selengkapnya, lihat dokumentasi GKE tentang Class Gateway.

Untuk membuat Gateway, lakukan langkah-langkah berikut:

1. Simpan manifes contoh berikut sebagai gateway.yaml:

   apiVersion: gateway.networking.k8s.io/v1
   kind: Gateway
   metadata:
     name: GATEWAY_NAME
   spec:
     gatewayClassName: gke-l7-regional-external-managed
     listeners:
       - protocol: HTTP # Or HTTPS for production
         port: 80 # Or 443 for HTTPS
         name: http
   

   Ganti GATEWAY_NAME dengan nama unik untuk resource Gateway Anda. Contoh, inference-gateway.

2. Terapkan manifes ke cluster Anda:

   kubectl apply -f gateway.yaml
   

Membuat resource HTTPRoute

Di bagian ini, Anda akan membuat resource HTTPRoute untuk menentukan cara Gateway merutekan permintaan HTTP yang masuk ke InferencePool.

Resource HTTPRoute menentukan cara GKE Gateway merutekan permintaan HTTP masuk ke layanan backend, yaitu InferencePool Anda. Aturan ini menentukan aturan pencocokan (misalnya, header, atau jalur) dan backend tempat traffic harus diteruskan.

Untuk membuat HTTPRoute, lakukan langkah-langkah berikut:

1. Simpan manifes contoh berikut sebagai httproute.yaml:

   apiVersion: gateway.networking.k8s.io/v1
   kind: HTTPRoute
   metadata:
     name: HTTPROUTE_NAME
   spec:
     parentRefs:
     - name: GATEWAY_NAME
     rules:
     - matches:
       - path:
           type: PathPrefix
           value: PATH_PREFIX
       backendRefs:
       - name: INFERENCE_POOL_NAME
         group: inference.networking.x-k8s.io
         kind: InferencePool
   

   Ganti kode berikut:

   • HTTPROUTE_NAME: nama unik untuk resource HTTPRoute Anda. Misalnya, my-route.
   • GATEWAY_NAME: nama resource Gateway yang Anda buat. Misalnya, inference-gateway.
   • PATH_PREFIX: awalan jalur yang Anda gunakan untuk mencocokkan permintaan masuk. Misalnya, / untuk mencocokkan semua.
   • INFERENCE_POOL_NAME: nama resource InferencePool yang ingin Anda jadikan tujuan traffic. Misalnya, vllm-llama3-8b-instruct.

2. Terapkan manifes ke cluster Anda:

   kubectl apply -f httproute.yaml
   

Mengirim permintaan inferensi

Setelah mengonfigurasi GKE Inference Gateway, Anda dapat mengirim permintaan inferensi ke model yang di-deploy.

Untuk mengirim permintaan inferensi, lakukan langkah-langkah berikut:

• Ambil endpoint Gateway.
• Buat permintaan JSON yang diformat dengan benar.
• Gunakan curl untuk mengirim permintaan ke endpoint /v1/completions.

Dengan cara ini, Anda dapat membuat teks berdasarkan perintah input dan parameter yang ditentukan.

1. Untuk mendapatkan endpoint Gateway, jalankan perintah berikut:

   IP=$(kubectl get gateway/GATEWAY_NAME -o jsonpath='{.status.addresses[0].address}')
   PORT=PORT_NUMBER # Use 443 for HTTPS, or 80 for HTTP
   

   Ganti kode berikut:

   • GATEWAY_NAME: nama resource Gateway Anda.
   • PORT_NUMBER: nomor port yang Anda konfigurasikan di Gateway.

2. Untuk mengirim permintaan ke endpoint /v1/completions menggunakan curl, jalankan perintah berikut:

   curl -i -X POST https://${IP}:${PORT}/v1/completions \
   -H 'Content-Type: application/json' \
   -H 'Authorization: Bearer $(gcloud auth print-access-token)' \
   -d '{
       "model": "MODEL_NAME",
       "prompt": "PROMPT_TEXT",
       "max_tokens": MAX_TOKENS,
       "temperature": "TEMPERATURE"
   }'
   

   Ganti kode berikut:

   • MODEL_NAME: nama model atau adaptor LoRA yang akan digunakan.
   • PROMPT_TEXT: perintah input untuk model.
   • MAX_TOKENS: jumlah maksimum token yang akan dibuat dalam respons.
   • TEMPERATURE: mengontrol keacakan output. Gunakan nilai 0 untuk output deterministik, atau angka yang lebih tinggi untuk output yang lebih kreatif.

Perhatikan perilaku berikut:

• Isi permintaan: isi permintaan dapat menyertakan parameter tambahan seperti stop dan top_p. Lihat spesifikasi OpenAI API untuk mengetahui daftar lengkap opsi.
• Penanganan error: terapkan penanganan error yang tepat dalam kode klien Anda untuk menangani potensi error dalam respons. Misalnya, periksa kode status HTTP dalam respons curl. Kode status non-200 umumnya menunjukkan error.
• Autentikasi dan otorisasi: untuk deployment produksi, amankan endpoint API Anda dengan mekanisme autentikasi dan otorisasi. Sertakan header yang sesuai (misalnya, Authorization) dalam permintaan Anda.

Mengonfigurasi kemampuan observasi untuk Inference Gateway

GKE Inference Gateway memberikan kemampuan observasi terkait kondisi, performa, dan perilaku workload inferensi Anda. Hal ini membantu Anda mengidentifikasi dan menyelesaikan masalah, mengoptimalkan penggunaan resource, dan memastikan keandalan aplikasi. Anda dapat melihat metrik kemampuan observasi ini di Cloud Monitoring melalui Metrics Explorer.

Untuk mengonfigurasi kemampuan observasi GKE Inference Gateway, lihat Mengonfigurasi kemampuan observasi.

Menghapus resource yang di-deploy

Agar tidak menimbulkan biaya pada akun Google Cloud Anda untuk resource yang Anda buat dari panduan ini, jalankan perintah berikut:

gcloud container clusters delete CLUSTER_NAME \
    --region=REGION

Ganti nilai berikut:

• REGION: region yang mendukung jenis akselerator yang ingin Anda gunakan, misalnya, us-central1 untuk GPU H100.
• CLUSTER_NAME: nama cluster Anda.

Langkah berikutnya

• Baca tentang GKE Inference Gateway.
• Baca artikel Men-deploy GKE Inference Gateway.