Version

Z.ai/GLM-4.5-Air

A 106B MoE served from a GGUF checkpoint via llama.cpp on a single A100.

On this page

A 106B MoE served from an Unsloth GGUF checkpoint via llama.cpp instead of vLLM, on a single A100 40 GB. Modelplane treats the engine as any OpenAI-compatible container, so the only changes from a vLLM deployment are the image and args: the container is still named engine and listens on :8000. vLLM can’t load this Unsloth quantization format. llama.cpp can, and -hf pulls the checkpoint straight from Hugging Face at startup, so a one-time deployment needs no ModelCache.

The model is bigger than one A100’s VRAM, so --n-cpu-moe offloads the MoE expert tensors to host RAM and the GPU runs the active path and KV cache. That’s how a 106B model fits one A100 instead of a multi-GPU node. Apply the platform side first, then the ML side.

Validated deployments

MoE 106B A12B 8,192 ctx llama.cpp

Cloud

AWS Azure Nebius Vultr

GPU

A16 16G L4 24G A100 40/80G H100 80G H200 141G 1× per node

Serving mode

Standalone LeaderWorker PrefillDecode

Precision

Engine

vLLM SGLang llama.cpp

Image

ghcr.io/ggml-org/llama.cpp:server-cuda

Manifests

docs/manifests/recipes/glm-4.5-air

Platform

# A single A100 40GB on GKE. GLM-4.5-Air is a ~106B MoE; a 4-bit GGUF doesn't
# fit one A100's VRAM, but llama.cpp offloads the expert tensors to host RAM
# (see the ModelDeployment's --n-cpu-moe), so the GPU only holds the active path
# + KV cache. One A100, not a multi-GPU node.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceClass
metadata:
  name: gke-a100-40-1x
spec:
  description: "GKE a2-highgpu-1g, 1x NVIDIA A100 40GB"
  provisioning:
    provider: GKE
    gke:
      machineType: a2-highgpu-1g
      diskSizeGb: 200
      accelerator:
        type: nvidia-tesla-a100
        count: 1
  devices:
  - name: gpu
    claim: DRA
    driver: gpu.nvidia.com
    deviceClassName: gpu.nvidia.com
    count: 1
    attributes:
      architecture: { string: Ampere }
    capacity:
      memory: { value: "40960Mi" }

# A GKE cluster with a single A100 node offering the class above.
apiVersion: modelplane.ai/v1alpha1
kind: InferenceCluster
metadata:
  name: gke-a100
  labels:
    modelplane.ai/region: us-west
spec:
  cluster:
    source: GKE
    gke:
      region: us-west1
  nodePools:
  - name: gpu-a100
    className: gke-a100-40-1x
    nodeCount: 1
    minNodeCount: 1
    maxNodeCount: 1
    zones:
    - us-west1-b

Deployment

# GLM-4.5-Air (~106B MoE) served from an Unsloth GGUF via llama.cpp instead of
# vLLM, on a SINGLE A100. Modelplane treats the engine as any OpenAI-compatible
# container, so the only changes from a vLLM deployment are the image and args:
# the container is still named `engine` and listens on :8000. vLLM can't load
# Unsloth's UD- dynamic quants; llama.cpp can, and `-hf` pulls the quant from
# HuggingFace at startup (no ModelCache needed for a one-off).
#
# The model is bigger than one A100's VRAM, so --n-cpu-moe offloads the MoE
# expert tensors to host RAM; the GPU runs the active path. That's how a 106B
# model fits one A100 instead of a multi-GPU node. --port 8000 because llama.cpp
# defaults to 8080 and Modelplane scrapes 8000.
apiVersion: modelplane.ai/v1alpha1
kind: ModelDeployment
metadata:
  name: glm-air
  namespace: ml-team
spec:
  replicas: 1
  template:
    spec:
      engines:
      - name: glm
        members:
        - role: Standalone
          nodeSelector:
            devices:
            - name: gpu
              count: 1
              selectors:
              - cel: |
                  device.capacity["gpu.nvidia.com"].memory.compareTo(quantity("35Gi")) >= 0
          template:
            spec:
              containers:
              - name: engine
                image: ghcr.io/ggml-org/llama.cpp:server-cuda
                args:
                - "-hf"
                - "unsloth/GLM-4.5-Air-GGUF:IQ4_XS"
                - "--host"
                - "0.0.0.0"
                - "--port"
                - "8000"
                - "-ngl"
                - "999"
                - "--n-cpu-moe"
                - "99"
                - "--jinja"
                - "-c"
                - "8192"

# One OpenAI-compatible endpoint for the deployment. Read its public address:
#   kubectl get ms glm-air -n ml-team -o jsonpath='{.status.address}'
apiVersion: modelplane.ai/v1alpha1
kind: ModelService
metadata:
  name: glm-air
  namespace: ml-team
spec:
  endpoints:
  - selector:
      matchLabels:
        modelplane.ai/deployment: glm-air