bfloat16 range: ~1e^{-38} to ~3e^{38} float32 range: ~1e^{-38} to ~3e^{38} float16 range: ~5.9e $ ^{-8} $ to 6.5e jpg) ## 转换起来简单一点的：bfloat16（大指数版） - 另一种简单的方法，就是直接暴力地把 32 位浮点从 16 位切断，只取出高 16 位，当做一种非标准的 half 来存储。称为 bfloat16（前面多个 b）。 - 因为 bfloat16 是从 float 中之间暴力切断出来的，所以只有底数被切断了，指数完全没有变。 • bfloat16 具有 8 位指数，7 位底数。 - float16 具有 5 位指数，10 位底数。 - 可见 bfloat16 的指数部分占得比较多，而底数就很少，这样会有一点不精确，优点是和 float之间转换的位运算实现起来比较简单。double:  ![Image](

[--load-format {auto,pt,safetensors,npcache, →dummy,tensorizer,bitsandbytes}] [--dtype {auto,half,float16,bfloat16,float, →float32}] [--kv-cache-dtype {auto,fp8,fp8_e5m2,fp8_ →e4m3}] [--quantization-param-path MAX_LORA_RANK] [--lora-extra-vocab-size LORA_EXTRA_VOCAB_ →SIZE] [--lora-dtype {auto,float16,bfloat16, →float32}] [--long-lora-scaling-factors LONG_LORA_ →SCALING_FACTORS] [--max-cpu-loras MAX_CPU_LORAS] using bitsandbytes quantization. Default:“auto”| |--dtype|Possible choices: auto, half, float16, bfloat16, float, float32 Data type for model weights and activations. •“auto” will use FP16 precision for

[--load-format {auto,pt,safetensors,npcache, →dummy,tensorizer}] [--dtype {auto,half,float16,bfloat16,float, →float32}] [--kv-cache-dtype {auto,fp8,fp8_e5m2,fp8_ →e4m3}] [--quantization-param-path LORA_EXTRA_VOCAB_ EXTRA_CONFIG] ## [--long-lora-scaling-factors LONG_LORA_ ## [--lora-dtype {auto,float16,bfloat16, [SERVED_MODEL_NAME ...] TOKENS] MAX_MODEL_LEN] [--device {auto,cuda,neuron,cpu}] [--image-input-type Examplessection for more information. Default: "auto" --dtype Possible choices: auto, half, float16, bfloat16, float, float32 Data type for model weights and activations. - "auto" will use FP16 precision

[--load-format {auto,pt,safetensors,npcache, →dummy,tensorizer,bitsandbytes}] [--dtype {auto,half,float16,bfloat16,float, →float32}] [--kv-cache-dtype {auto,fp8,fp8_e5m2,fp8_ →e4m3}] [--quantization-param-path MAX_LORA_RANK] [--lora-extra-vocab-size LORA_EXTRA_VOCAB_ →SIZE] [--lora-dtype {auto,float16,bfloat16, →float32}] [--long-lora-scaling-factors LONG_LORA_ →SCALING_FACTORS] [--max-cpu-loras MAX_CPU_LORAS] bitsandbytes quantization. ## Default: "auto" ## --dtype Possible choices: auto, half, float16, bfloat16, float, float32 Data type for model weights and activations. - "auto" will use FP16 precision

DOWNLOAD_DIR] [--load-format {auto,pt,safetensors,npcache,dummy}] [--dtype {auto,half,float16,bfloat16,float,float32}] [--kv-cache-dtype {auto,fp8_e5m2}] [--max-model-len MAX_MODEL_LEN] [--worker-use-ray] next page) |\[--lora-extra-vocab-size LORA\_EXTRA\_VOCAB\_SIZE]| |---| |[--lora-dtype{auto,float16,bfloat16,float32}| |\[--max-cpu-loras MAX\_CPU\_LORAS]| |[--device{auto,cuda,neuron,cpu}| |[--image-inpu values,which is mainly for profiling。 Default:“auto”| |--dtype|Possible choices:auto,half,float16,bfloat16,float,float32 data type for model weights and activations.The“auto”option will use FP16 precision

→state,gguf,bitsandbytes,mistral}] [--config-format {auto,hf,mistral}] [--dtype {auto,half,float16,bfloat16,float,float32}] [--kv-cache-dtype {auto,fp8,fp8_e5m2,fp8_e4m3}] [--quantization-param-path QU [--max-lora-rank MAX_LORA_RANK] [--lora-extra-vocab-size LORA_EXTRA_VOCAB_SIZE] [--lora-dtype {auto,float16,bfloat16,float32}] [--long-lora-scaling-factors LONG_LORA_SCALING_FACTORS] [--max-cpu-loras MAX_CPU_LORAS] available else it will try to load in mistral format --dtype Possible choices: auto, half, float16, bfloat16, float, float32 Data type for model weights and activations. "auto" will use FP16 precision

→state,gguf,bitsandbytes,mistral}] [--config-format {auto,hf,mistral}] [--dtype {auto,half,float16,bfloat16,float,float32}] [--kv-cache-dtype {auto,fp8,fp8_e5m2,fp8_e4m3}] [--quantization-param-path QU [--max-lora-rank MAX_LORA_RANK] [--lora-extra-vocab-size LORA_EXTRA_VOCAB_SIZE] [--lora-dtype {auto,float16,bfloat16,float32}] [--long-lora-scaling-factors LONG_LORA_SCALING_FACTORS] [--max-cpu-loras MAX_CPU_LORAS] available else it will try to load in mistral format --dtype Possible choices: auto, half, float16, bfloat16, float, float32 Data type for model weights and activations. "auto" will use FP16 precision

[--load-format {auto,pt,safetensors,npcache, →dummy,tensorizer,bitsandbytes}] [--dtype {auto,half,float16,bfloat16,float, →float32}] [--kv-cache-dtype {auto,fp8,fp8_e5m2,fp8_ →e4m3}] [--quantization-param-path MAX_LORA_RANK] [--lora-extra-vocab-size LORA_EXTRA_VOCAB_ →SIZE] [--lora-dtype {auto,float16,bfloat16, →float32}] [--long-lora-scaling-factors LONG_LORA_ →SCALING_FACTORS] [--max-cpu-loras MAX_CPU_LORAS] using bitsandbytes quantization. Default:“auto”| |--dtype|Possible choices: auto, half, float16, bfloat16, float, float32 Data type for model weights and activations. •“auto” will use FP16 precision for

safetensors,npcache,dummy,tensorizer,sharded_ →state,gguf,bitsandbytes}] [--dtype {auto,half,float16,bfloat16,float,float32}] [--kv-cache-dtype {auto,fp8,fp8_e5m2,fp8_e4m3}] [--quantization-param-path QU [--max-lora-rank MAX_LORA_RANK] [--lora-extra-vocab-size LORA_EXTRA_VOCAB_SIZE] [--lora-dtype {auto,float16,bfloat16,float32}] [--long-lora-scaling-factors LONG_LORA_SCALING_FACTORS] [--max-cpu-loras MAX_CPU_LORAS] (added to the base model vocabulary). Default: 256 --lora-dtype Possible choices: auto, float16, bfloat16, float32 Data type for LoRA. If auto, will default to base model dtype. Default: "auto"

the serialized weights. Default: "auto" --dtype Possible choices: auto, half, float16, bfloat16, float, float32 Data type for model weights and activations. "auto" will use FP16 precision (added to the base model vocabulary). Default: 256 --lora-dtype Possible choices: auto, float16, bfloat16, float32 Data type for LoRA. If auto, will default to base model dtype. Default: "auto" [--load-format {auto,pt,safetensors,npcache, →dummy,tensorizer}] [--dtype {auto,half,float16,bfloat16,float, →float32}] [--kv-cache-dtype {auto,fp8}] [--quantization-param-path QUANTIZATION_ →PARAM_PATH]