Recommended model parameters
Recommended serving parameters, environment variables, and hardware for supported Managed Inference models.
CosmicAC recommends these serving parameters, environment variables, and hardware for the following models. The Job configuration reference defines each parameter. For how to apply them, see Create a Managed Inference Job.
Tensor parallel equals GPUs per replica for every vLLM model on this page. The web interface and the CLI take tensor parallel from the model master. If a job carries no tensor parallel value, CosmicAC uses GPUs per replica. Parakeet carries no tensor parallel value.
Qwen3-VL-235B-A22B-Thinking-FP8
Model ID Qwen/Qwen3-VL-235B-A22B-Thinking-FP8.
Serving parameters
| Parameter | Value |
|---|---|
| Runtime image | vllm/vllm-openai:v0.15.1 |
| Data type | Auto |
| Quantisation | None |
| Tensor parallel | 8 |
| GPUs per replica | 8 |
| GPU memory utilization | 0.9 |
| Max model length | 131072 |
| Max concurrent sequences | 64 |
| Reasoning parser | deepseek_r1 |
| Video & image input | Yes |
| Root disk size | 500 GB |
Environment variables
TRUST_REMOTE_CODE=true
ENABLE_EXPERT_PARALLEL=true
ENFORCE_EAGER=trueHardware
| Resource | Value |
|---|---|
| GPUs | 8 H100 80 GB |
| CPU cores per GPU | 16 |
| RAM per GPU | 150 GB |
Qwen3.5-122B-A10B
Model ID Qwen/Qwen3.5-122B-A10B.
Serving parameters
| Parameter | Value |
|---|---|
| Runtime image | vllm/vllm-openai:v0.17.1 |
| Data type | Auto |
| Quantisation | FP8 |
| Tensor parallel | 8 |
| GPUs per replica | 8 |
| GPU memory utilization | 0.9 |
| Max model length | 32768 |
| Max concurrent sequences | 32 |
| Reasoning parser | deepseek_r1 |
| Video & image input | Yes |
| Root disk size | 500 GB |
Environment variables
TRUST_REMOTE_CODE=true
SWAP_SPACE=0
ENABLE_EXPERT_PARALLEL=true
ENFORCE_EAGER=trueMiniMax M2.5
Model ID MiniMaxAI/MiniMax-M2.5.
Serving parameters
| Parameter | Value |
|---|---|
| Runtime image | vllm/vllm-openai:v0.15.1 |
| Data type | Auto |
| Quantisation | None |
| Tensor parallel | 4 |
| GPUs per replica | 4 |
| GPU memory utilization | 0.85 |
| Max model length | 27000 |
| Max concurrent sequences | 256 |
| Reasoning parser | deepseek_r1 |
| Video & image input | Yes |
| Root disk size | 500 GB |
Environment variables
TRUST_REMOTE_CODE=true
SWAP_SPACE=0
ENABLE_EXPERT_PARALLEL=true
ENFORCE_EAGER=trueHardware
| Resource | Value |
|---|---|
| GPUs | 4 H100 80 GB |
| CPU cores per GPU | 16 |
| RAM per GPU | 150 GB |
Qwen2-VL-2B-Instruct
Model ID Qwen/Qwen2-VL-2B-Instruct.
Serving parameters
| Parameter | Value |
|---|---|
| Runtime image | vllm/vllm-openai:v0.15.1 |
| Data type | Auto |
| Quantisation | None |
| Tensor parallel | 1 |
| GPUs per replica | 1 |
| GPU memory utilization | 0.9 |
| Max model length | 32768 |
| Max concurrent sequences | 64 |
| Reasoning parser | default, no parser |
| Video & image input | Yes |
| Root disk size | 150 GB |
Environment variables
TRUST_REMOTE_CODE=true
ENFORCE_EAGER=true
SWAP_SPACE=0Parakeet
Model ID nvidia/parakeet-tdt-0.6b-v3. Parakeet is a speech-to-text model, so it takes no runtime image, data type, quantisation, tensor parallel, or reasoning parser. CosmicAC supplies the speech-to-text runtime image from the job type. The Parakeet Managed Inference parameters define each field, and Create a Parakeet Managed Inference Job shows how to apply them.
Serving parameters
| Parameter | Value |
|---|---|
| Chunk duration | 30 seconds |
| Chunk overlap | 5 seconds |
| Max file size | 1024 MB |
| GPUs per replica | 1 |
| Root disk size | 250 GB |
| External disk size | 200 GB |
Environment variables
MODEL_SOURCE=huggingface
HF_HOME=/mnt/external-disk/huggingface
NEMO_CACHE_DIR=/mnt/external-disk/nemo
HOST=0.0.0.0
INFERENCE_BACKEND=parakeet
PORT=8777
PARAKEET_STARTUP_TIMEOUT_MS=600000Hardware
| Resource | Value |
|---|---|
| GPUs | 1 |
| CPU cores per GPU | 20 |
| RAM per GPU | 125 GB |