Inference service model parameters

Last update: October 13 2026Open in Markdown

You can specify model parameters when creating an inference service. They determine how the inference service will process requests and consume compute resources.

Data type for KV cache (Key-Value Cache)	The storage format for intermediate model computations during token generation. Key-Value Cache is a mechanism for accelerating token generation in LLMs based on transformer architecture. More compact formats reduce memory consumption and allow processing longer contexts
Maximum context length	The maximum number of tokens that the model can process within a single request