Inference service model parameters

Last update: September 07 2026Open in Markdown

You can specify model parameters when creating an inference service. They determine how the inference service will process requests and consume computing resources.

Data type for the Key-Value Cache (KV Cache)	The storage format for intermediate model computations during token generation. Key-Value Cache is a mechanism for accelerating token generation in LLMs based on the transformer architecture. More compact formats reduce memory consumption and allow for processing longer contexts
Maximum context length	The maximum number of tokens that the model can process within a single request