--- outline: deep --- # OpenAI-Compatible API Server This page assumes you've already installed Aphrodite and know how to launch the OpenAI-Compatible server. :::info This page is quite large and extensive; please use the table of contents ("On this page" to the top left) to navigate. ::: ## API Reference Please see the [OpenAI API Reference](https://platform.openai.com/docs/api-reference) for more information on the API scheme, as we support all parameters, except: - in `/v1/chat/completions`: `tools` and `tool_choice` - in `/v1/completions`: `suffix` Otherwise, we support everything, plus many other parameters. Aphrodite also provides experimental support for the OpenAI Vision API. ## Extra Parameters If using the `openai` python library, you cannot pass extra parameters such as `min_p`, `guided_choice`, etc. Thankfully, the library allows you to extend the body as needed: ```py completion = client.chat.completions.create( model="meta-llama/Meta-Llama-3.1-8B-Instruct", messages=[ {"role": "user", "content": "Classify this sentiment: LLMs are wonderful!"} ], extra_body={ "guided_choice": ["positive", "negative"] } ) ``` ## Extra Parameters for Chat API Aphrodite supports the following extra parameters that are not supported by OpenAI: ```py best_of: Optional[int] = None use_beam_search: Optional[bool] = False top_k: Optional[int] = -1 min_p: Optional[float] = 0.0 top_a: Optional[float] = 0.0 tfs: Optional[float] = 1.0 eta_cutoff: Optional[float] = 0.0 epsilon_cutoff: Optional[float] = 0.0 typical_p: Optional[float] = 1.0 smoothing_factor: Optional[float] = 0.0 smoothing_curve: Optional[float] = 1.0 repetition_penalty: Optional[float] = 1.0 length_penalty: Optional[float] = 1.0 early_stopping: Optional[bool] = False ignore_eos: Optional[bool] = False min_tokens: Optional[int] = 0 stop_token_ids: Optional[List[int]] = Field(default_factory=list) skip_special_tokens: Optional[bool] = True spaces_between_special_tokens: Optional[bool] = True truncate_prompt_tokens: Optional[Annotated[int, Field(ge=1)]] = None ``` And the following parameters: ```py echo: Optional[bool] = Field( default=False, description=( "If true, the new message will be prepended with the last message " "if they belong to the same role."), ) add_generation_prompt: Optional[bool] = Field( default=True, description= ("If true, the generation prompt will be added to the chat template. " "This is a parameter used by chat template in tokenizer config of the " "model."), ) add_special_tokens: Optional[bool] = Field( default=False, description=( "If true, special tokens (e.g. BOS) will be added to the prompt " "on top of what is added by the chat template. " "For most models, the chat template takes care of adding the " "special tokens so this should be set to False (as is the " "default)."), ) documents: Optional[List[Dict[str, str]]] = Field( default=None, description= ("A list of dicts representing documents that will be accessible to " "the model if it is performing RAG (retrieval-augmented generation)." " If the template does not support RAG, this argument will have no " "effect. We recommend that each document should be a dict containing " "\"title\" and \"text\" keys."), ) chat_template: Optional[str] = Field( default=None, description=( "A Jinja template to use for this conversion. " "If this is not passed, the model's default chat template will be " "used instead."), ) chat_template_kwargs: Optional[Dict[str, Any]] = Field( default=None, description=("Additional kwargs to pass to the template renderer. " "Will be accessible by the chat template."), ) include_stop_str_in_output: Optional[bool] = Field( default=False, description=( "Whether to include the stop string in the output. " "This is only applied when the stop or stop_token_ids is set."), ) guided_json: Optional[Union[str, dict, BaseModel]] = Field( default=None, description=("If specified, the output will follow the JSON schema."), ) guided_regex: Optional[str] = Field( default=None, description=( "If specified, the output will follow the regex pattern."), ) guided_choice: Optional[List[str]] = Field( default=None, description=( "If specified, the output will be exactly one of the choices."), ) guided_grammar: Optional[str] = Field( default=None, description=( "If specified, the output will follow the context free grammar."), ) guided_decoding_backend: Optional[str] = Field( default=None, description=( "If specified, will override the default guided decoding backend " "of the server for this specific request. If set, must be either " "'outlines' / 'lm-format-enforcer'")) guided_whitespace_pattern: Optional[str] = Field( default=None, description=( "If specified, will override the default whitespace pattern " "for guided json decoding.")) ``` ## Extra Parameters for Text Completions API Aphrodite supports the following extra parameters that are not supported by OpenAI: ```py use_beam_search: Optional[bool] = False top_k: Optional[int] = -1 min_p: Optional[float] = 0.0 top_a: Optional[float] = 0.0 tfs: Optional[float] = 1.0 eta_cutoff: Optional[float] = 0.0 epsilon_cutoff: Optional[float] = 0.0 typical_p: Optional[float] = 1.0 smoothing_factor: Optional[float] = 0.0 smoothing_curve: Optional[float] = 1.0 repetition_penalty: Optional[float] = 1.0 length_penalty: Optional[float] = 1.0 early_stopping: Optional[bool] = False stop_token_ids: Optional[List[int]] = Field(default_factory=list) ignore_eos: Optional[bool] = False min_tokens: Optional[int] = 0 skip_special_tokens: Optional[bool] = True spaces_between_special_tokens: Optional[bool] = True truncate_prompt_tokens: Optional[Annotated[int, Field(ge=1)]] = None allowed_token_ids: Optional[List[int]] = None include_stop_str_in_output: Optional[bool] = False add_special_tokens: Optional[bool] = False ``` And the following parameters: ```py response_format: Optional[ResponseFormat] = Field( default=None, description= ("Similar to chat completion, this parameter specifies the format of " "output. Only {'type': 'json_object'} or {'type': 'text' } is " "supported."), ) guided_json: Optional[Union[str, dict, BaseModel]] = Field( default=None, description=("If specified, the output will follow the JSON schema."), ) guided_regex: Optional[str] = Field( default=None, description=( "If specified, the output will follow the regex pattern."), ) guided_choice: Optional[List[str]] = Field( default=None, description=( "If specified, the output will be exactly one of the choices."), ) guided_grammar: Optional[str] = Field( default=None, description=( "If specified, the output will follow the context free grammar."), ) guided_decoding_backend: Optional[str] = Field( default=None, description=( "If specified, will override the default guided decoding backend " "of the server for this specific request. If set, must be one of " "'outlines' / 'lm-format-enforcer'")) guided_whitespace_pattern: Optional[str] = Field( default=None, description=( "If specified, will override the default whitespace pattern " "for guided json decoding.")) ``` ## Chat Template In order for the LLM to support chat completions protocol, Aphrodite requires the model to include a chat template in its tokenizer config. The chat template is Jinja2 template file that specifies how roles, messages, and other chat-specific tokens are encoded in the input. Most modern LLMs provide this if they're an Instruct/Chat finetune, but sometimes they may not. For those models, you can manually specify their chat template in the `--chat-template` (or `- chat_template` in the YAML) with the path being the URL or local disk path. You may also provide it as in-line string to the argument. Without a chat template, the server will only launch text completions. Aphrodite provides a set of chat templates, which you can view [here](https://github.com/PygmalionAI/aphrodite-engine/tree/main/examples/chat_templates). ## Command-line arguments for the server ```console usage: aphrodite run [options] positional arguments: model_tag The model tag to serve options: -h, --help show this help message and exit --host HOST host name --port PORT port number --uvicorn-log-level {debug,info,warning,error,critical,trace} log level for uvicorn --allow-credentials allow credentials --allowed-origins ALLOWED_ORIGINS allowed origins --allowed-methods ALLOWED_METHODS allowed methods --allowed-headers ALLOWED_HEADERS allowed headers --api-keys API_KEYS If provided, the server will require this key to be presented in the header. --admin-key ADMIN_KEY If provided, the server will require this key to be presented in the header for admin operations. --lora-modules LORA_MODULES [LORA_MODULES ...] LoRA module configurations in the format name=path. Multiple modules can be specified. --prompt-adapters PROMPT_ADAPTERS [PROMPT_ADAPTERS ...] Prompt adapter configurations in the format name=path. Multiple adapters can be specified. --chat-template CHAT_TEMPLATE The file path to the chat template, or the template in single-line form for the specified model --response-role RESPONSE_ROLE The role name to return if `request.add_generation_prompt=true`. --ssl-keyfile SSL_KEYFILE The file path to the SSL key file --ssl-certfile SSL_CERTFILE The file path to the SSL cert file --ssl-ca-certs SSL_CA_CERTS The CA certificates file --ssl-cert-reqs SSL_CERT_REQS Whether client certificate is required (see stdlib ssl module's) --root-path ROOT_PATH FastAPI root_path when app is behind a path based routing proxy --middleware MIDDLEWARE Additional ASGI middleware to apply to the app. We accept multiple --middleware arguments. The value should be an import path. If a function is provided, Aphrodite will add it to the server using @app.middleware('http'). If a class is provided, Aphrodite will add it to the server using app.add_middleware(). --launch-kobold-api Launch the Kobold API server alongside the OpenAI server --max-log-len MAX_LOG_LEN Max number of prompt characters or prompt ID numbers being printed in log. Default: 0 --return-tokens-as-token-ids When --max-logprobs is specified, represents single tokens asstrings of the form 'token_id:{token_id}' so that tokens thatare not JSON-encodable can be identified. --disable-frontend-multiprocessing If specified, will run the OpenAI frontend server in the same process as the model serving engine. --model MODEL Category: Model Options name or path of the huggingface model to use --seed SEED Category: Model Options random seed --served-model-name SERVED_MODEL_NAME [SERVED_MODEL_NAME ...] Category: API Options The model name(s) used in the API. If multiple names are provided, the server will respond to any of the provided names. The model name in the model field of a response will be the first name in this list. If not specified, the model name will be the same as the `--model` argument. Noted that this name(s)will also be used in `model_name` tag content of prometheus metrics, if multiple names provided, metricstag will take the first one. --tokenizer TOKENIZER Category: Model Options name or path of the huggingface tokenizer to use --revision REVISION Category: Model Options the specific model version to use. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version. --code-revision CODE_REVISION Category: Model Options the specific revision to use for the model code on Hugging Face Hub. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version. --tokenizer-revision TOKENIZER_REVISION Category: Model Options the specific tokenizer version to use. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default version. --tokenizer-mode {auto,slow} Category: Model Options tokenizer mode. "auto" will use the fast tokenizer if available, and "slow" will always use the slow tokenizer. --trust-remote-code Category: Model Options trust remote code from huggingface --download-dir DOWNLOAD_DIR Category: Model Options directory to download and load the weights, default to the default cache dir of huggingface --max-model-len MAX_MODEL_LEN Category: Model Options model context length. If unspecified, will be automatically derived from the model. --max-context-len-to-capture MAX_CONTEXT_LEN_TO_CAPTURE Category: Model Options Maximum context length covered by CUDA graphs. When a sequence has context length larger than this, we fall back to eager mode. (DEPRECATED. Use --max-seq_len-to-capture instead) --max-seq_len-to-capture MAX_SEQ_LEN_TO_CAPTURE Category: Model Options Maximum sequence length covered by CUDA graphs. When a sequence has context length larger than this, we fall back to eager mode. --rope-scaling ROPE_SCALING Category: Model Options RoPE scaling configuration in JSON format. For example, {"type":"dynamic","factor":2.0} --rope-theta ROPE_THETA Category: Model Options RoPE theta. Use with `rope_scaling`. In some cases, changing the RoPE theta improves the performance of the scaled model. --model-loader-extra-config MODEL_LOADER_EXTRA_CONFIG Category: Model Options Extra config for model loader. This will be passed to the model loader corresponding to the chosen load_format. This should be a JSON string that will be parsed into a dictionary. --enforce-eager [ENFORCE_EAGER] Category: Model Options Always use eager-mode PyTorch. If False, will use eager mode and CUDA graph in hybrid for maximal performance and flexibility. --skip-tokenizer-init Category: Model Options Skip initialization of tokenizer and detokenizer --tokenizer-pool-size TOKENIZER_POOL_SIZE Category: Model Options Size of tokenizer pool to use for asynchronous tokenization. If 0, will use synchronous tokenization. --tokenizer-pool-type TOKENIZER_POOL_TYPE Category: Model Options The type of tokenizer pool to use for asynchronous tokenization. Ignored if tokenizer_pool_size is 0. --tokenizer-pool-extra-config TOKENIZER_POOL_EXTRA_CONFIG Category: Model Options Extra config for tokenizer pool. This should be a JSON string that will be parsed into a dictionary. Ignored if tokenizer_pool_size is 0. --max-logprobs MAX_LOGPROBS Category: Model Options maximum number of log probabilities to return. --device {auto,cuda,neuron,cpu,openvino,tpu,xpu} Category: Model Options Device to use for model execution. --load-format {auto,pt,safetensors,npcache,dummy,tensorizer,sharded_state,bitsandbytes} Category: Model Options The format of the model weights to load. * "auto" will try to load the weights in the safetensors format and fall back to the pytorch bin format if safetensors format is not available. * "pt" will load the weights in the pytorch bin format. * "safetensors" will load the weights in the safetensors format. * "npcache" will load the weights in pytorch format and store a numpy cache to speed up the loading. * "dummy" will initialize the weights with random values, which is mainly for profiling. * "tensorizer" will load the weights using tensorizer from CoreWeave. See the Tensorize Aphrodite Model script in the Examples section for more information. * "bitsandbytes" will load the weights using bitsandbytes quantization. --dtype {auto,half,float16,bfloat16,float,float32} Category: Model Options Data type for model weights and activations. * "auto" will use FP16 precision for FP32 and FP16 models, and BF16 precision for BF16 models. * "half" for FP16. Recommended for AWQ quantization. * "float16" is the same as "half". * "bfloat16" for a balance between precision and range. * "float" is shorthand for FP32 precision. * "float32" for FP32 precision. --ignore-patterns IGNORE_PATTERNS Category: Model Options The pattern(s) to ignore when loading the model.Defaults to 'original/**/*' to avoid repeated loading of llama's checkpoints. --worker-use-ray Category: Parallel Options Deprecated, use --distributed-executor-backend=ray. --tensor-parallel-size TENSOR_PARALLEL_SIZE, -tp TENSOR_PARALLEL_SIZE Category: Parallel Options number of tensor parallel replicas, i.e. the number of GPUs to use. --pipeline-parallel-size PIPELINE_PARALLEL_SIZE, -pp PIPELINE_PARALLEL_SIZE Category: Parallel Options number of pipeline stages. Currently not supported. --ray-workers-use-nsight Category: Parallel Options If specified, use nsight to profile ray workers --disable-custom-all-reduce Category: Model Options See ParallelConfig --distributed-executor-backend {ray,mp} Category: Parallel Options Backend to use for distributed serving. When more than 1 GPU is used, will be automatically set to "ray" if installed or "mp" (multiprocessing) otherwise. --max-parallel-loading-workers MAX_PARALLEL_LOADING_WORKERS Category: Parallel Options load model sequentially in multiple batches, to avoid RAM OOM when using tensor parallel and large models --quantization {aqlm,awq,deepspeedfp,eetq,fp8,fbgemm_fp8,gguf,marlin,gptq_marlin_24,gptq_marlin,awq_marlin,gptq,quip,squeezellm,compressed-tensors,bitsandbytes,qqq,None}, -q {aqlm,awq,deepspeedfp,eetq,fp8,fbgemm_fp8,gguf,marlin,gptq_marlin_24,gptq_marlin,awq_marlin,gptq,quip,squeezellm,compressed-tensors,bitsandbytes,qqq,None} Category: Quantization Options Method used to quantize the weights. If None, we first check the `quantization_config` attribute in the model config file. If that is None, we assume the model weights are not quantized and use `dtype` to determine the data type of the weights. --quantization-param-path QUANTIZATION_PARAM_PATH Category: Quantization Options Path to the JSON file containing the KV cache scaling factors. This should generally be supplied, when KV cache dtype is FP8. Otherwise, KV cache scaling factors default to 1.0, which may cause accuracy issues. FP8_E5M2 (without scaling) is only supported on cuda versiongreater than 11.8. On ROCm (AMD GPU), FP8_E4M3 is instead supported for common inference criteria. --preemption-mode PREEMPTION_MODE Category: Scheduler Options If 'recompute', the engine performs preemption by block swapping; If 'swap', the engine performs preemption by block swapping. --deepspeed-fp-bits DEEPSPEED_FP_BITS Category: Quantization Options Number of floating bits to use for the deepseed quantization. Supported bits are: 4, 6, 8, 12. --kv-cache-dtype {auto,fp8,fp8_e5m2,fp8_e4m3} Category: Cache Options Data type for kv cache storage. If "auto", will use model data type. CUDA 11.8+ supports fp8 (=fp8_e4m3) and fp8_e5m2. ROCm (AMD GPU) supports fp8 (=fp8_e4m3) --block-size {8,16,32} Category: Cache Options token block size --enable-prefix-caching, --context-shift Category: Cache Options Enable automatic prefix caching. --num-gpu-blocks-override NUM_GPU_BLOCKS_OVERRIDE Category: Cache Options Options If specified, ignore GPU profiling result and use this number of GPU blocks. Used for testing preemption. --disable-sliding-window Category: KV Cache Options Disables sliding window, capping to sliding window size --gpu-memory-utilization GPU_MEMORY_UTILIZATION, -gmu GPU_MEMORY_UTILIZATION Category: Cache Options The fraction of GPU memory to be used for the model executor, which can range from 0 to 1.If unspecified, will use the default value of 0.9. --swap-space SWAP_SPACE Category: Cache Options CPU swap space size (GiB) per GPU --cpu-offload-gb CPU_OFFLOAD_GB Category: Cache Options The space in GiB to offload to CPU, per GPU. Default is 0, which means no offloading. Intuitively, this argument can be seen as a virtual way to increase the GPU memory size. For example, if you have one 24 GB GPU and set this to 10, virtually you can think of it as a 34 GB GPU. Then you can load a 13B model with BF16 weight,which requires at least 26GB GPU memory. Note that this requires fast CPU-GPU interconnect, as part of the model isloaded from CPU memory to GPU memory on the fly in each model forward pass. --use-v2-block-manager Category: Scheduler Options Use the v2 block manager. --scheduler-delay-factor SCHEDULER_DELAY_FACTOR, -sdf SCHEDULER_DELAY_FACTOR Category: Scheduler Options Apply a delay (of delay factor multiplied by previous prompt latency) before scheduling next prompt. --enable-chunked-prefill [ENABLE_CHUNKED_PREFILL] Category: Scheduler Options If True, the prefill requests can be chunked based on the max_num_batched_tokens. --guided-decoding-backend {outlines,lm-format-enforcer} Category: Scheduler Options Which engine will be used for guided decoding (JSON schema / regex etc) by default. Currently support https://github.com/outlines-dev/outlines and https://github.com/noamgat/lm-format-enforcer. Can be overridden per request via guided_decoding_backend parameter. --max-num-batched-tokens MAX_NUM_BATCHED_TOKENS Category: KV Cache Options maximum number of batched tokens per iteration --max-num-seqs MAX_NUM_SEQS Category: API Options maximum number of sequences per iteration --num-lookahead-slots NUM_LOOKAHEAD_SLOTS Category: Speculative Decoding Options Experimental scheduling config necessary for speculative decoding. This will be replaced by speculative decoding config in the future; it is present for testing purposes until then. --speculative-model SPECULATIVE_MODEL Category: Speculative Decoding Options The name of the draft model to be used in speculative decoding. --num-speculative-tokens NUM_SPECULATIVE_TOKENS Category: Speculative Decoding Options The number of speculative tokens to sample from the draft model in speculative decoding --speculative-max-model-len SPECULATIVE_MAX_MODEL_LEN Category: Speculative Decoding Options The maximum sequence length supported by the draft model. Sequences over this length will skip speculation. --ngram-prompt-lookup-max NGRAM_PROMPT_LOOKUP_MAX Category: Speculative Decoding Options Max size of window for ngram prompt lookup in speculative decoding. --ngram-prompt-lookup-min NGRAM_PROMPT_LOOKUP_MIN Category: Speculative Decoding Options Min size of window for ngram prompt lookup in speculative decoding. --speculative-draft-tensor-parallel-size SPECULATIVE_DRAFT_TENSOR_PARALLEL_SIZE, -spec-draft-tp SPECULATIVE_DRAFT_TENSOR_PARALLEL_SIZE Category: Speculative Decoding Options Number of tensor parallel replicas for the draft model in speculative decoding. --speculative-disable-by-batch-size SPECULATIVE_DISABLE_BY_BATCH_SIZE Category: Speculative Decoding Options Disable speculative decoding for new incoming requests if the number of enqueue requests is larger than this value. --spec-decoding-acceptance-method {rejection_sampler,typical_acceptance_sampler} Category: Speculative Decoding Options Specify the acceptance method to use during draft token verification in speculative decoding. Two types of acceptance routines are supported: 1) RejectionSampler which does not allow changing the acceptance rate of draft tokens, 2) TypicalAcceptanceSampler which is configurable, allowing for a higher acceptance rate at the cost of lower quality, and vice versa. --typical-acceptance-sampler-posterior-threshold TYPICAL_ACCEPTANCE_SAMPLER_POSTERIOR_THRESHOLD Category: Speculative Decoding Options Set the lower bound threshold for the posterior probability of a token to be accepted. This threshold is used by the TypicalAcceptanceSampler to make sampling decisions during speculative decoding. Defaults to 0.09 --typical-acceptance-sampler-posterior-alpha TYPICAL_ACCEPTANCE_SAMPLER_POSTERIOR_ALPHA Category: Speculative Decoding Options A scaling factor for the entropy-based threshold for token acceptance in the TypicalAcceptanceSampler. Typically defaults to sqrt of --typical-acceptance-sampler- posterior-threshold i.e. 0.3 --disable-logprobs-during-spec-decoding DISABLE_LOGPROBS_DURING_SPEC_DECODING Category: Speculative Decoding Options If set to True, token log probabilities are not returned during speculative decoding. If set to False, log probabilities are returned according to the settings in SamplingParams. If not specified, it defaults to True. Disabling log probabilities during speculative decoding reduces latency by skipping logprob calculation in proposal sampling, target sampling, and after accepted tokens are determined. --enable-lora Category: Adapter Options If True, enable handling of LoRA adapters. --max-loras MAX_LORAS Category: Adapter Options Max number of LoRAs in a single batch. --max-lora-rank MAX_LORA_RANK Category: Adapter Options Max LoRA rank. --lora-extra-vocab-size LORA_EXTRA_VOCAB_SIZE Category: Adapter Options Maximum size of extra vocabulary that can be present in a LoRA adapter (added to the base model vocabulary). --lora-dtype {auto,float16,bfloat16,float32} Category: Adapter Options Data type for LoRA. If auto, will default to base model dtype. --max-cpu-loras MAX_CPU_LORAS Category: Adapter Options Maximum number of LoRAs to store in CPU memory. Must be >= than max_num_seqs. Defaults to max_num_seqs. --long-lora-scaling-factors LONG_LORA_SCALING_FACTORS Category: Adapter Options Specify multiple scaling factors (which can be different from base model scaling factor - see eg. Long LoRA) to allow for multiple LoRA adapters trained with those scaling factors to be used at the same time. If not specified, only adapters trained with the base model scaling factor are allowed. --fully-sharded-loras Category: Adapter Options By default, only half of the LoRA computation is sharded with tensor parallelism. Enabling this will use the fully sharded layers. At high sequence length, max rank or tensor parallel size, this is likely faster. --qlora-adapter-name-or-path QLORA_ADAPTER_NAME_OR_PATH Category: Adapter Options Name or path of the LoRA adapter to use. --enable-prompt-adapter Category: Adapter Options If True, enable handling of PromptAdapters. --max-prompt-adapters MAX_PROMPT_ADAPTERS Category: Adapter Options Max number of PromptAdapters in a batch. --max-prompt-adapter-token MAX_PROMPT_ADAPTER_TOKEN Category: Adapter Options Max number of PromptAdapters tokens --disable-log-stats Category: Log Options disable logging statistics --engine-use-ray Use Ray to start the LLM engine in a separate process as the server process. --disable-log-requests Disable logging requests. --uvloop Use the Uvloop asyncio event loop to possibly increase performance ``` ## Tool Calling in the chat completions API Aphrodite supports only named function calling in the chat completions API. The `tool_choice` options `auto` and `required` are **not yet supported** but on the development roadmap. To use a named function, you need to define the function in the `tools` parameter and call it in the `tools_choice` parameter. It's the caller's responsibility to prompt the model with the tool info; Aphrodite will not automatically manipulate the prompt. **This may change in the future!** Aphrodite will use guided decoding to ensure the response matches the tool parameter object defined by the JSON schema in the `tools` parameter. Please refer to the OpenAI API reference for more info.