Inference service model parameters

Last update: July 28 2026

You can specify model parameters when creating an inference service. They determine how the inference service will process requests and consume computational resources.

Data type for the Key-Value Cache (KV Cache)	Storage format for intermediate model computations during token generation. The Key-Value Cache is a mechanism for accelerating token generation in LLMs based on the transformer architecture. More compact formats reduce memory consumption and allow for processing longer contexts
Maximum context length	The maximum number of tokens that the model can process within a single request