An open-source inference library from Ant Group and the vLLM community reported up to 43% higher throughput in a serving test, while separate benchmarks measured kernel performance, according to the project’s report.
Humming targets quantized AI inference, where model weights and activations can use varied numerical formats. The project says its design accommodates those combinations across dense models and mixture-of-experts (MoE) models. Quantization can lower memory requirements and data movement, while memory, bandwidth and compute remain deployment constraints, the project said.
One set of tests compared Humming’s general matrix multiplication (GEMM) kernels with implementations from vLLM Marlin, FlashInfer and DeepGEMM. The project reported higher performance across representative workloads.
A separate end-to-end test served DeepSeek-V4.1-Flash on eight H20 GPUs. The authors reported roughly 14% to 43% higher throughput across comparisons involving different MoE backends and activation formats.
Humming supports NVIDIA GPU architectures from Turing onward, as well as T-Head’s Zhenwu 810E, which is CUDA-compatible. Developers can work with integer and floating-point formats, multiple bit widths and different scaling methods.
The project says using common kernel components can reduce maintenance work for developers. It plans to expand hardware support and add autotuning, which tests configuration options on a target device to select settings for a workload.
The project says Humming will be included as a dependency in CUDA installations starting with vLLM 0.29.0.
