Skip to content

Instantly share code, notes, and snippets.

@nerdalert
Created July 26, 2025 23:20
Show Gist options
  • Save nerdalert/f9dc883bea0b803a26fc6f55e87f2af0 to your computer and use it in GitHub Desktop.
Save nerdalert/f9dc883bea0b803a26fc6f55e87f2af0 to your computer and use it in GitHub Desktop.
  • llm-d-infra/charts/llm-d-infra/templates/gateway-infrastructure/gatewayparameters.yaml
{{ if and .Values.gateway.enabled (eq .Values.gateway.gatewayClassName "kgateway") .Values.gateway.gatewayParameters.enabled }}
apiVersion: gateway.kgateway.dev/v1alpha1
kind: GatewayParameters
metadata:
  name: {{ include "gateway.fullname" . }}
  labels: {{ include "common.labels.standard" . | nindent 4 }}
    app.kubernetes.io/gateway: {{ include "gateway.fullname" . }}
    app.kubernetes.io/component: inference-gateway
    {{- if .Values.commonLabels }}
    {{- include "common.tplvalues.render" ( dict "value" .Values.commonLabels "context" $ ) | nindent 4 }}
    {{- end }}
  annotations:
    {{- if .Values.commonAnnotations }}
    {{- include "common.tplvalues.render" ( dict "value" .Values.commonAnnotations "context" $ ) | nindent 4 }}
    {{- end }}
    {{- if .Values.gateway.annotations }}
    {{- include "common.tplvalues.render" ( dict "value" .Values.gateway.annotations "context" $) | nindent 4 }}
    {{- end }}
spec:
  kube:
    envoyContainer:
      securityContext:
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
        runAsNonRoot: true
        {{- if .Values.gateway.gatewayParameters.proxyUID }}
        runAsUser: {{ .Values.gateway.gatewayParameters.proxyUID }}
        {{- end}}
        seccompProfile:
          type: RuntimeDefault
      bootstrap:
        componentLogLevels:
          upstream: info
          http: info
          connection: info
        logLevel: info
      {{- if .Values.gateway.gatewayParameters.resources }}
      resources: {{ .Values.gateway.gatewayParameters.resources | toYaml | nindent 8 }}
      {{- end}}
    service:
      type: {{ .Values.gateway.serviceType | default "NodePort" | quote }}
      extraLabels:
        gateway: custom
    podTemplate:
      extraLabels:
        gateway: custom
    sdsContainer:
      securityContext:
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
        {{- if .Values.gateway.gatewayParameters.proxyUID }}
        runAsUser: {{ .Values.gateway.gatewayParameters.proxyUID }}
        {{- end}}
        seccompProfile:
          type: RuntimeDefault
{{- end}}

  • llm-d-infra/charts/llm-d-infra/templates/gateway-infrastructure/configmap.yaml
{{ if and .Values.gateway.enabled (eq .Values.gateway.gatewayClassName "istio") .Values.gateway.gatewayParameters.enabled }}
apiVersion: v1
kind: ConfigMap
metadata:
  name: {{ include "gateway.fullname" . }}
  labels: {{ include "common.labels.standard" . | nindent 4 }}
    app.kubernetes.io/gateway: {{ include "gateway.fullname" . }}
    app.kubernetes.io/component: inference-gateway
    {{- if .Values.commonLabels }}
    {{- include "common.tplvalues.render" ( dict "value" .Values.commonLabels "context" $ ) | nindent 4 }}
    {{- end }}
  annotations:
    {{- if .Values.commonAnnotations }}
    {{- include "common.tplvalues.render" ( dict "value" .Values.commonAnnotations "context" $ ) | nindent 4 }}
    {{- end }}
    {{- if .Values.gateway.annotations }}
    {{- include "common.tplvalues.render" ( dict "value" .Values.gateway.annotations "context" $) | nindent 4 }}
    {{- end }}
data:
  deployment: |
    spec:
      template:
        spec:
          containers:
          - name: istio-proxy
            args:
              - proxy
              - router
              - --domain
              - $(POD_NAMESPACE).svc.cluster.local
              - --proxyLogLevel
              - info
              - --proxyComponentLogLevel
              - misc:error
              - --log_output_level
              - default:info
            {{- if .Values.gateway.gatewayParameters.resources }}
            resources: {{ .Values.gateway.gatewayParameters.resources | toYaml | nindent 14 }}
            {{- end}}
  service: |
    spec:
      type: {{ .Values.gateway.serviceType | default "NodePort" | quote }}
{{- end}}

  • Deploy with:
# from llm-d-infra root
helm upgrade infra-sim ./charts/llm-d-infra --namespace llm-d-sim

kubectl rollout restart deployment infra-sim-inference-gateway -n llm-d-sim

Result:

kubectl describe pod infra-sim-inference-gateway-f4d7f4c4f-gdzkb -n llm-d-sim
Name:             infra-sim-inference-gateway-f4d7f4c4f-gdzkb
Namespace:        llm-d-sim
Priority:         0
Service Account:  infra-sim-inference-gateway
Node:             minikube/192.168.49.2
Start Time:       Sat, 26 Jul 2025 23:15:40 +0000
Labels:           app.kubernetes.io/instance=infra-sim-inference-gateway
                  app.kubernetes.io/name=infra-sim-inference-gateway
                  gateway=custom
                  gateway.networking.k8s.io/gateway-name=infra-sim-inference-gateway
                  pod-template-hash=f4d7f4c4f
Annotations:      kubectl.kubernetes.io/restartedAt: 2025-07-26T23:15:40Z
                  prometheus.io/path: /metrics
                  prometheus.io/port: 9091
                  prometheus.io/scrape: true
Status:           Running
IP:               10.244.0.17
IPs:
  IP:           10.244.0.17
Controlled By:  ReplicaSet/infra-sim-inference-gateway-f4d7f4c4f
Containers:
  kgateway-proxy:
    Container ID:    docker://72851e729b776ee3daac092858ee4a32848d2ced40d5ba368dfe1e39a94da441
    Image:           cr.kgateway.dev/kgateway-dev/envoy-wrapper:v2.0.3
    Image ID:        docker-pullable://cr.kgateway.dev/kgateway-dev/envoy-wrapper@sha256:3b6f7ea7d62fd5023a99abedb17826f3b8b9fa084c99fed5eebcb56befcd27ea
    Ports:           8080/TCP, 9091/TCP
    Host Ports:      0/TCP, 0/TCP
    SeccompProfile:  RuntimeDefault
    Args:
      --disable-hot-restart
      --service-node
      $(POD_NAME).$(POD_NAMESPACE)
      --log-level
      info
      --component-log-level
      connection:info,http:info,upstream:info
    State:          Running
      Started:      Sat, 26 Jul 2025 23:15:40 +0000
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     2
      memory:  1Gi
    Requests:
      cpu:     100m
      memory:  128Mi
    Environment:
      POD_NAME:       infra-sim-inference-gateway-f4d7f4c4f-gdzkb (v1:metadata.name)
      POD_NAMESPACE:  llm-d-sim (v1:metadata.namespace)
      ENVOY_UID:      0
    Mounts:
      /etc/envoy from envoy-config (rw)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True
  Initialized                 True
  Ready                       True
  ContainersReady             True
  PodScheduled                True
Volumes:
  envoy-config:
    Type:        ConfigMap (a volume populated by a ConfigMap)
    Name:        infra-sim-inference-gateway
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age    From               Message
  ----    ------     ----   ----               -------
  Normal  Scheduled  4m23s  default-scheduler  Successfully assigned llm-d-sim/infra-sim-inference-gateway-f4d7f4c4f-gdzkb to minikube
  Normal  Pulled     4m23s  kubelet            Container image "cr.kgateway.dev/kgateway-dev/envoy-wrapper:v2.0.3" already present on machine
  Normal  Created    4m23s  kubelet            Created container: kgateway-proxy
  Normal  Started    4m23s  kubelet            Started container kgateway-proxy
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment