audio.cpp

audio.cpp is a high-performance C++ audio inference framework built on top of ggml, designed to make modern local audio models practical, portable, and fast.

Tired of juggling a dozen Conda environments, hundreds of Python packages, and dependency conflicts just to try a few audio models? audio.cpp gives those paths a shared native runtime instead. Runs on Windows, Linux, and macOS, with support for NVIDIA, AMD, Apple Silicon, and CPU-only machines.

Huggingface main repo: https://huggingface.co/audio-cpp/audio.cpp-gguf

ModelScope repo mirror: https://www.modelscope.cn/models/HereIsMark/audio.cpp-gguf

[!IMPORTANT]

> 2026-09-10 - Dev testing: Yue2 3B: Yue2 native song generation is available on the dev branch for community testing and optimization. Dev binaries are available from Actions -> Release -> latest dev artifacts.

> Arena UI: The new Arena tab makes it easier to compare local models side by side for TTS, voice conversion, and ASR. Use one shared input, queue multiple models or GGUF variants, then review outputs with metrics!

> CUDA performance headline: multiple TTS paths already run 1.8x to up to 8x faster than their Python reference paths while cutting end-to-end latency by 45%-85%.

> GGUF performance: all released model families support GGUF loading, and tested Q8 packages can run up to 1.53x faster while reducing peak VRAM by up to about 37% on routes such as Higgs Audio, Fish Audio, and Voxtral. See the GGUF guide for support status and the Q8 performance report for 16-bit vs Q8 measurements.

> Production deployment example: Try Fun-ASR-Nano with audio.cpp on the FunASR platform https://www.funasr.com/en/deploy/audio-cpp.html!

> VibeVoice 1.5B: generates a 93.9-minute podcast in 18.2 minutes with 10 diffusion steps and without quantization, running about 5.15x faster than real time.

> Supertonic 3: generates about 10 hours of audio in 3 minutes on RTX5090. Up to 200x+ real-time on CUDA, 6x+ real-time on CPU, and 47 ms TTFT in CUDA streaming mode.

Demo: 10 hours of audio generated in 3 minutes.

> Real-world ASR win: In TranscrIA benchmark on messy French meeting audio, audio.cpp’s Nemotron 3.5 ASR matched the same WER as other implementations while using about 1/4 of the wall time.

It is built for real end-to-end execution rather than one-off model demos: the same runtime powers TTS, voice cloning, voice conversion, ASR, diarization, VAD, source separation, alignment, codec-style models, and higher-level workflows through a common framework surface.

Highlights:

• Parity. Strong parity tooling against Python reference paths.
• Performance. Performance-focused execution, reusable sessions, and batch-style offline inference. Optimized for CUDA.
• Portability. A portable native stack centered on ggml, with CUDA, HIP/ROCm, Vulkan, Metal, and CPU backends behind shared CLI and server entry points instead of Python-only deployment paths.
• Pipelines. Experimental JSON pipeline support for higher-level multi-step workflows.
• Audio Utilities. Built-in denoise, enhancement, resampling, and STFT/ISTFT utilities for real production-style task paths.

The goal of the framework is to provide highly optimized, reusable building blocks for audio-related models, so new model integrations can be brought up faster, shared components can be improved once and benefit many families, and real end-to-end inference paths can stay efficient, maintainable, and portable.

audio.cpp would not be moving this quickly without generous contributors bringing in real fixes, new capabilities, and careful polish. See CONTRIBUTING.md for how to contribute and for a shout-out to the people already helping shape the project.

[!TIP]

Contribution focus: the most helpful contributions right now are improvements to the UI, API server, and pipeline/workflow subsystems. These areas make the existing model surface easier to use, serve, compose, and validate. See CONTRIBUTING.md for more details.

> New model PRs: before starting a new model port, please check the supported model table because several families are already implemented or under testing. New ports should start under the community models surface, where review is lighter than core models but still needs reproducible validation. Please follow the measurement style in PR #19 and PR #63: exact build/run commands, model paths or package ids, generated outputs, parity or path-test results, and relevant performance or memory notes.

News

[!IMPORTANT]

2026-09-12 - Release 0.7.4: This release adds VibeVoice ASR Streaming 7B, Irodori-TTS v4.1 Anime, Moonshine Streaming ASR, and Kokoro 82M, plus GGUF/package and UI updates for the latest community models. Thanks @DrewThomasson for the Colab UI!

> 2026-08-26 - Release 0.7: This release adds MiniMax Music 3, MagpieTTS, PersonaPlex, MeanVC2, AudioSR, ControlFoley, FireRedTTS3, FireRedAudio, MiDashengLM-Gen, F5-TTS/Habibi, Granite Speech 5.0 TurboCTC, MMS Forced Aligner, and MOSS-VoiceGenerator, plus DotTTS Edit and ACE-Step 1.5 XL variants, bringing audio.cpp to 62 total model families and 85+ model variants! It also introduces the new Arena UI for side-by-side TTS, voice-conversion, and ASR comparison with shared inputs, queued runs, metrics, and result sorting.

> 2026-08-13 - Release 0.6: This release adds 5 new model families - DotTTS, NeuTTS, MuScriptor, MiniMax-H3, and SenseVoice - bringing audio.cpp to 49 total model families and 70+ model variants, alongside the new native WebUI from @mirek190, expanded GGUF packaging, and more shared framework runtime pieces.

> 2026-07-31 - Release 0.5: audio.cpp reaches 44 model families with 9 new additions, early HIP/ROCm support for AMD GPUs, Nix ROCm/HIP build support, Metal optimizations with tested VoxCPM2 runs up to 2.56x faster on Apple Silicon, and a major GGUF-first WebUI/package-spec usability pass.

2026-06-25 to 2026-07-23 (release 0.1 to 0.4): audio.cpp grew from the first released model wave into broad TTS, ASR, music generation, source separation, VAD, diarization, codec, and voice-conversion coverage, with VibeVoice 1.5B/7B, LoRA adapter loading, initial streaming support, and major CUDA Conv1DTransp speedups.

Supported Models

Task tags: TTS text to speech, Clone voice cloning, VC voice conversion, S2S speech-to-speech, ASR speech recognition, Align forced alignment, VAD voice activity detection, Diar speaker diarization, Codec audio codec, Sep source separation, MIDI audio-to-symbolic MIDI/events, Music music/song generation, SFX sound effects, Video video generation, Edit audio/music editing, Design voice design, Dialogue multi-speaker dialogue TTS, Ctrl TTS/clone voice control such as emotion, style, instruction, caption, or non-verbal tag control.

Runtime tags summarize the supported loading paths. GGUF package precision varies by model and release; check the audio.cpp GGUF repo or docs/gguf.md for the exact package list. Bundled means the tiny runtime asset ships under assets/framework/models and needs no separate model download. Stream means the family exposes a streaming server/session path.

Speech Generation And Conversation

| Family | Task | Lang | Variants | Runtime | |---|---|---|---|---| | breeze_tts | TTS, Clone, Design, Ctrl | zh, en | BreezeTTS 2 instruction-conditioned TTS and prompt-audio voice cloning | GGUF BF16/Q8, Stream | | chatterbox | TTS, Clone, VC| ar, da, de, el, en, es, fi, fr, hi, it, ko, ms, nl, no, pl, pt, sv, sw, tr | Chatterbox with 0.5B backbone | GGUF 16/Q8 | | confucius4_tts | Clone | zh, en, ja, ko, de, fr, es, id, it, th, pt, ru, ms, vi | Confucius4-TTS multilingual voice cloning | GGUF F32, Stream | | cosyvoice3 | TTS, Clone | zh, en, ja, ko, de, es, fr, it, ru, yue | Fun-CosyVoice3 zero-shot, cross-lingual, and instruction-conditioned TTS | GGUF F32/Q8 | | dots_tts | TTS, Clone, Edit, Ctrl | multilingual | DotTTS SOAR, MeanFlow, and Edit | GGUF 16/Q8, Stream | | dramabox | TTS, Clone | en | DramaBox expressive TTS and voice cloning | GGUF Q8 | | fish_audio | TTS, Clone, Ctrl | auto, en, zh | Fish Audio S2 Pro | GGUF 16/Q8 | | firered_audio | ASR, TTS, Clone, Design, Ctrl | zh, en | FireRedAudio multimodal speech/audio model with ASR, understanding, cloning, design, and edit paths | GGUF original/Q8 | | fireredtts3 | TTS, Clone, Design, Ctrl | 24 langs + 21 zh dialects | FireRedTTS3 Base and Instruct packages for voice cloning, design, semantic edit, and acoustic edit | GGUF original/Q8 | | higgs_audio_tts | TTS, Clone, Ctrl | auto | Higgs Audio v3 TTS 4B | GGUF 16/Q8 | | index_tts2 | TTS, Clone, Ctrl | zh, en, ja, es, ar | IndexTTS-2, IndexTTS-2.5 (variant) | GGUF 16/Q8 | | kokoro_tts | TTS | en-us, en-gb, es, fr, hi, it, ja, pt-br, zh | Kokoro 82M, 54 preset voices | Safetensors, local GGUF BF16/Q8 | | irodori_tts | TTS, Clone, Design, Ctrl | ja | Irodori-TTS-v4-Small, Irodori-TTS-500M-v3, Irodori-TTS-600M-v3-VoiceDesign | GGUF 16/Q8 | | magpie_tts | TTS | ar-AE, ar-MSA, ar-SA, de, en, es, fr, hi, it, ko, pt-BR, vi, zh | NVIDIA MagpieTTS Multilingual 357M (v2607) with baked speaker prompts and NanoCodec decode | GGUF original/Q8 | | miotts | TTS, Clone | en, ja | MioTTS-1.7B | GGUF 16/Q8 | | moss_tts_local | TTS, Clone, Ctrl | auto, optional language hint | MOSS-TTS-Local-Transformer-v1.5 | GGUF 16/Q8 | | moss_tts_nano | TTS, Clone | auto | MOSS-TTS-Nano-100M | GGUF 16/Q8 | | neutts | TTS, Ctrl | en | NeuTTS 2E with built-in speaker prompts and emotion control | GGUF original precision, Stream | | omnivoice | TTS, Clone, Design, Ctrl | 646+ langs | OmniVoice, Qwen3-0.6B based | GGUF 16/Q8, Stream | | personaplex | Dialogue, S2S | en | PersonaPlex 7B v1 speech-to-speech conversational model with packaged voice/persona prompts | GGUF Q4/Q8, Stream | | pocket_tts | TTS, Clone | en, de, it, pt, es | PocketTTS-100M | GGUF 16/Q8, Stream | | qwen3_tts | TTS, Clone, Design, Ctrl | zh, en, fr, de, it, ja, ko, pt, ru, es | Qwen3-TTS-12Hz-0.6B-Base, Qwen3-TTS-12Hz-1.7B-Base, Qwen3-TTS-12Hz-1.7B-CustomVoice, Qwen3-TTS-12Hz-1.7B-VoiceDesign | GGUF 16/Q8 | | supertonic | TTS | en, ko, ja, ar, bg, cs, da, de, el, es, et, fi, fr, hi, hr, hu, id, it, lt, lv, nl, pl, pt, ro, ru, sk, sl, sv, tr, uk, vi, na | Supertonic 3 | GGUF F32, Stream | | vibevoice | TTS, Dialogue | en, zh | VibeVoice-1.5B, VibeVoice-7B | GGUF 16/Q8 | | voxcpm2 | TTS, Clone, Design, Ctrl | ar, da, de, el, en, es, fi, fr, he, hi, id, it, ja, km, ko, lo, ms, my, nl, no, pl, pt, ru, sv, sw, th, tl, tr, vi, zh | VoxCPM2-2B, 48 kHz | GGUF 16/Q8, Stream |

Speech Recognition And Analysis

| Family | Task | Lang | Variants | Runtime | |---|---|---|---|---| | citrinet_asr | ASR | en | Citrinet-256 | GGUF Q8 | | fun_asr_nano | ASR | auto, zh, en, ja | Fun-ASR-Nano-2512 | GGUF 16/Q8 | | higgs_audio_stt | ASR | en | Higgs Audio v3 STT | GGUF 16/Q8, Stream | | hviske_asr | ASR | da | Hviske v5.3 | GGUF Q8 | | marblenet_vad | VAD | lang agnostic | MarbleNet VAD | Bundled | | moonshine_asr | ASR | en | Moonshine Streaming Tiny/Small/Medium | GGUF Q8, Stream | | nemotron_asr | ASR | 100+ ASR prompt codes incl. auto | Nemotron 3.5 ASR Streaming 0.6B | GGUF 16/Q8, Stream | | qwen3_asr | ASR | zh, en, yue, ar, de, fr, es, pt, id, it, ko, ru, th, vi, ja, tr, hi, ms, nl, sv, da, fi, pl, cs, fil, fa, el, ro, hu, mk | Qwen3-ASR-0.6B, Qwen3-ASR-1.7B-hf | GGUF 16/Q8, Stream | | qwen3_forced_aligner | Align | zh, yue, en, de, es, fr, it, pt, ru, ko, ja | Qwen3-ForcedAligner-0.6B | GGUF 16/Q8 | | silero_vad | VAD | lang agnostic | Silero VAD | Bundled, Stream | | sortformer_diar | Diar | en | Sortformer-4spk-v1 | - | | vibevoice_asr | ASR | auto | VibeVoice ASR | GGUF 16/Q8 | | vibevoice_asr_streaming | ASR | en, zh, es, pt, de, ja, ko, fr, ru, it | VibeVoice ASR Streaming 7B with persistent decoder state and speaker turns | GGUF BF16/Q8/Q4, Stream | | voxtral_realtime | ASR | auto | Voxtral-Mini-4B-Realtime-2602 | GGUF 16/Q8/Q4, Stream |

Audio Conversion And Processing

| Family | Task | Lang | Variants | Runtime | |---|---|---|---|---| | audiosr | S2S | lang agnostic | AudioSR Basic audio super-resolution package | GGUF F32 | | bs_roformer | Sep | lang agnostic | BS-RoFormer vocal separation checkpoints | GGUF Q8 | | controlfoley | SFX | auto | ControlFoley 44 kHz multimodal Foley generation from text, video, and reference audio conditioning | GGUF F32/Q8 | | htdemucs | Sep | lang agnostic | HTDemucs, HTDemucs_ft | GGUF 16/Q8 | | meanvc2 | VC | lang agnostic | MeanVC2 120 ms/40 ms zero-shot voice conversion | GGUF F32/Q4, Stream | | mel_band_roformer | Sep | lang agnostic | Mel-Band RoFormer MLX vocal separation variants | GGUF 16/Q8 | | miocodec | Codec, VC | lang agnostic | MioCodec v2, 25 Hz, 44.1 kHz | GGUF 16/Q8 | | muscriptor | MIDI | music | MuScriptor Small audio-to-symbolic transcription | GGUF F32, Stream | | rvc | VC | lang agnostic | RVC F16 GGUF with packaged v1/v2 voices and optional retrieval blending | GGUF 16 | | seed_vc | VC | lang agnostic | SeedVC XLS-R + HiFT, SeedVC Whisper-small + BigVGAN | GGUF 16/Q8 |

Music, Media, And Editing

| Family | Task | Lang | Variants | Runtime | |---|---|---|---|---| | ace_step | Music, Edit | 50+ langs | ACE-Step 1.5 Turbo/Base and XL Turbo/SFT with acestep-5Hz-lm-1.7B | GGUF 16 | | heartmula | Music | zh, en, ja, ko, es | HeartMuLa-oss-3B with HeartCodec-oss | GGUF 16/Q8 | | midashenglm_gen | Music, SFX | auto | MiDashengLM-Gen structured-prompt generation for speech, music, sound effects, and ambience | GGUF F32/Q8 | | minimax_h3 | Video, Music, TTS/Dialogue | auto | MiniMax-H3 Q4_K with optional INT8 ConvRot DiT | GGUF Q4/INT8 | | minimax_music3 | Music | auto | MiniMax Music 3 text-to-music generation with lyrics conditioning | GGUF Q4/Q8 | | stable_audio | Music, SFX, Edit | en | Stable Audio 3 Small Music, Stable Audio 3 Small SFX, Stable Audio 3 Medium | GGUF 16/Q8 | | vevo2 | TTS, Music, VC, Edit | en, zh | Vevo2 with Qwen2.5-0.5B AR model | GGUF 16 |

Some model families in the supported table started as outside contributions before being promoted into the core release surface. Thanks to Mirek @mirek190 for BS-RoFormer, @justinjohn0306 for MOSS-TTS-Local, and @LauraGPT from the official FunASR team for Fun-ASR-Nano.

Community Models

Community model ports live under community_models to make the ownership boundary clear while keeping them available through the normal audio.cpp CLI and server paths. Some community-contributed models graduate into the core model tree when they become part of the main release surface. Huge thanks to the contributors who bring these models in, test them, and keep pushing the framework into new territory. See docs/community_models/models.md for community-model expectations and current entries.

| Family | Task | Lang | Runtime | Contributor | What They Added | |---|---|---|---|---|---| | audio8_asr | ASR | en, zh, yue, ja, ko, fr, de | GGUF Q8, Safetensors | @gqf2008 | Audio8-ASR-0.1B compact multilingual autoregressive ASR reusing the Qwen3-ASR encoder with an MLP-tower adapter and an 8-layer Qwen2-style decoder (CC-BY-NC, local conversion only) | | audio8_tts | TTS, Clone | auto, yue, zh, nl, en, fr, de, it, ja, ko, pl, es | GGUF Q8, Stream | @jasonchen31 | Audio8 TTS Preview 0.6B DualAR multilingual TTS and zero-shot voice cloning with a Qwen backbone and neural codec | | chatterbox_turbo | TTS (testing) | en | GGUF 16/Q8 | @pannagaps | Chatterbox Turbo distilled 350M GPT2 T3 backbone + 2-step meanflow S3Gen decoder; built-in voice | | echo_tts | Clone | en | GGUF 16/Q8 | @5uck1ess | Echo-TTS 44.1 kHz zero-shot voice cloning with EchoDiT latents and Fish S1-DAC decoding | | f5_tts | TTS, Clone | en, ar (Habibi) | GGUF | @tareko | F5-TTS flow-matching DiT synthesis and voice cloning, with Habibi Arabic aliases habibi/habibi_tts | | glm_tts | TTS, Clone | zh, en | GGUF | Mirek @mirek190 | GLM-TTS zero-shot synthesis and voice cloning support | | granite5asr | ASR | en | GGUF Q8 | @ampersandru | IBM Granite Speech 5.0 470M TurboCTC ultra-fast Conformer-CTC ASR with Shaw relative positional embeddings and ByteLevel BPE | | inflect_v2 | TTS | en | GGUF FP32 | Jan @JanWerder | Inflect Micro v2 and Nano v2 native offline synthesis | | kroko_asr | ASR | de, en, es, fr, it, he, nl, pt, sv, tr | Safetensors, GGUF Q8 | Mirek @mirek190 | Kroko Community ASR native offline/streaming Zipformer2/RNN-T transcription with word timestamps | | minimax_h3 | Video, Music, TTS/Dialogue | auto | GGUF Q4/INT8 | @0xShug0 | MiniMax-H3 text-to-audio/video generation with Q4_K and optional INT8 ConvRot DiT | | minimax_music3 | Music | auto | GGUF Q4/Q8 | @0xShug0, @JoeMattie | MiniMax Music 3 text-to-music generation with lyrics conditioning | | mira_tts | TTS, Clone | en | Local conversion | Mirek @mirek190 | MiraTTS experimental native Qwen2 + ECAPA/Perceiver zero-shot voice cloning with progressive segment streaming (CC-BY-NC-SA-4.0 weights) | | mms_forced_aligner | Align | nl (nld), en (eng); pre-romanized Latin | Safetensors, GGUF 16/Q8 | Community | MMS-300M-1130 Forced Aligner word-timestamp alignment from a wav2vec2 CTC checkpoint (safetensors or local GGUF) | | moss_tts_local | TTS, Clone, Ctrl | auto, optional language hint | GGUF | @justinjohn0306 | MOSS-TTS-Local Transformer v1.5 support | | moss_voicegen | Voice Design | en, zh | GGUF | Joost @jrohde | MOSS-VoiceGenerator speech in a voice designed from a written instruction | | outetts | TTS, Clone | en, ar, zh, nl, fr, de, it, ja, ko, lt, ru, es, pt, be, bn, ka, hu, lv, fa, pl, sw, ta, uk | GGUF | Mirek @mirek190 | Llama-OuteTTS-1.0-1B TTS and voice cloning support | | parakeet_tdt | ASR | auto, bg, cs, da, de, el, en, es, et, fi, fr, hr, hu, it, lt, lv, mt, nl, pl, pt, ro, ru, sk, sl, sv, uk | GGUF F32/16/Q8, Stream | @dleiferives | Parakeet-TDT 0.6B v3 offline, long-form, and buffered-streaming ASR support, plus the Orukeet r3 weight variant | | sanotts | TTS | en, vi, id, cs, de, es, fr, it, pt, ro, ru, tr, ne, hi | GGUF FP32 | Ashish @voidash | sanoTTS voice family eighteen voices from 294k to 2.27M parameters, native offline synthesis | | sense_asr | ASR | auto, zh, en, yue, ja, ko, pt, ru, es, it, fr, de, nl, pl, tr, ar, hi, vi, th, id, ms, fa, nospeech | GGUF Q8, Stream | Jason Chen @jasonchen31, @LauraGPT / FunASR | SenseVoice-Small offline/streaming SAN-M + CTC transcription with event/emotion/language tags and ITN | | sopro_tts | TTS, Clone | en, pt, fr, de | Safetensors, GGUF, Stream | Community | Sopro V2 Turbo 120M zero-shot voice cloning: style-prefix semantic LM over FSQ tokens, rectified-flow acoustic DiT, Vocos ISTFT vocoder at 24 kHz | | soprano_tts | TTS | en | GGUF Q8, Stream | @drzsdrtfg | Soprano-1.1-80M ultra-lightweight TTS with Qwen3 LM + Vocos decoder | | sortformer_diar_v2 | Diar | multilingual | GGUF F32/mixed F16, Stream | Community | NVIDIA Sortformer v2.1 four-speaker streaming diarization; local conversion only pending redistribution approval | | vietneu_tts | TTS, Clone | vi, en | GGUF | Phuoc @phuocnguyen90 | VieNeu-TTS-v3-Turbo TTS and voice cloning support | | vibeasr | ASR | en | GGUF I8_S + I2_S | @XsquirrelC | VibeASR fully quantized port of VibeASR.cpp: VibeVoice acoustic/semantic tokenizers on INT8 weights and INT8 activations, feeding a ternary BitNet Qwen2 decoder. Offline, CPU only | | voxcpm1 | TTS, Clone | zh, en, ja, ko | GGUF Q8, Stream | @jasonchen31 | VoxCPM1 tokenizer-free 0.5B TTS with 16 kHz output, streaming, and continuation-mode voice cloning |

Docker

Docker CUDA and CPU images are available for both CLI and server use. See docker.md for available images, build commands and working Docker examples.

Model Manager and GGUF Downloads

Use tools/model_manager_v2.py for normal model downloads. It reads model_specs/*.json and installs the default package for each family, preferring ready-to-use GGUF packages when they are available.

Native builds configured with -DAUDIOCPP_BUILD_NATIVE_MODEL_MANAGER=ON also provide audiocpp_model_manager, a standalone headless frontend over the same reusable C++ package-management library used by the server and embedded WebUI. It supports list, info, install, clean, and remove without starting audiocpp_server; this is the preferred native path for CLI, Docker, remote provisioning, and other scripted environments. The Python v2 manager remains available as an alternative during migration. The native path uses bundled TLS by default and does not require libcurl.

```bash audiocpp_model_manager list audiocpp_model_manager install q