Skip to content

feat: support hy4 mxfp8 rtn - #390

Merged
RuBing-Yang merged 1 commit into
Tencent:mainfrom
RuBing-Yang:hy4-mxfp8
Oct 14, 2026
Merged

RuBing-Yang merged 1 commit into
Tencent:mainfrom
RuBing-Yang:hy4-mxfp8

Conversation

@RuBing-Yang

Copy link
Copy Markdown
Collaborator

This pull request adds a new tool, mxfp8_quant.py, which enables quantization of model weights to the MXFP8 format for efficient storage and inference. The script processes safetensors model files in parallel, applies quantization to specific layers, and generates updated model files and configuration metadata.

Key additions and features:

MXFP8 Quantization Functionality:

  • Implements the mxfp8_quantize function to quantize 2D or 3D tensors to the MXFP8 format, including per-block scaling and handling of padding for block alignment.
  • Defines a set of regex patterns to identify which model layers should be quantized, supporting both standard and expert layers in transformer architectures.

Parallel Processing and File Management:

  • Uses Python multiprocessing to process multiple safetensors files in parallel, improving speed for large models.
  • For each processed file, quantized weights and their associated scale tensors are saved, and a new safetensors index is generated for the output model.

Configuration and Metadata Updates:

  • Updates or generates a new config.json with quantization metadata, supporting two config formats (mxfp8 and modelopt_mxfp8) as specified by the user.
  • Tracks and logs layers that are ignored

@RuBing-Yang
RuBing-Yang requested a review from yghstill October 14, 2026 10:32
@RuBing-Yang
RuBing-Yang merged commit 7fbd3a3 into Tencent:main Oct 14, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants