espnet/espnet

• ESPnet 202610.post1 — the command line grows espnet demo (the OWSM browser demo) and --live (the microphone, transcribed as you speak); one Speech2Text now loads either kind of OWSM checkpoint, with best_path() for CTC decoding without a search; espnet/espnet:inference-cpu-latest and -gpu-latest run a published model with nothing installed; three more demo Spaces (TTS, enhancement, speaker verification).
• ESPnet 202610 — one-line inference from the command line (pip install espnet && espnet asr audio.wav), two OWSM v4 demos as Hugging Face Spaces, a core install without the training stack (training is espnet[train]), batched beam search, PyTorch 2.11-2.14.
• ESPnet 202609 — ESPnet3 complete on egs3/librispeech_100 at ESPnet2 parity, CI rebuilt on a prebuilt image (compute per run halved), OpenBEATs pretraining, ten new recipes (ASR, TTS, SER, ST, audio SSL), Python 3.12-3.13.
• ESPnet 202604 — Docker-based CI, PyTorch 2.9.1 support, FastSpeech2 inference ~1.9x faster at batch 8, new recipes (Kinyarwanda, Emilia, kosp2e).
• ESPnet 202511 — parallel-processing primitives, refactored inference and evaluation pipeline, expanded SpeechLM support.
• ESPnet 202509 — Python 3.9-3.13, Debian 12 CI, the LID subsystem completed, multi-optimizer training (HybridOptim / HybridLRS).
• ESPnet 202506 — ESPnet3 groundwork (data organizer, trainer, model), LID training and task setup, codec1 recipes, USES2 speech enhancement, IPAPack++ S2T recipes.
• ESPnet 202503 — PyTorch Lightning trainer support, Hugging Face front-end, scaled dot-product attention, ML-SUPERB 2024 recipe.

pip install "espnet[all]"                       # training plus every task extra (except sds)
pip install git+https://github.com/espnet/espnet  # latest master

• Full setup (recipes, DNN training, Kaldi-style tooling): see the installation guide.
• Docker: see docker/ and the Docker docs.
• Task-specific tools live in tools/installers.
• ESPnet1 is no longer supported — use ESPnet2 (egs2/) or ESPnet3 (egs3/). See the ESPnet1 notice.

|system/pytorch ver.|2.11.0|2.13.0|2.14.0| | :---- | :---: | :---: | :---: | |ubuntu/python3.12/pip|||| |ubuntu/python3.13/pip|||| |debian12/python3.12/conda|||| |windows/python3.12/pip|||| |macos/python3.12/pip|||| |macos/python3.12/conda||||

Each badge is its workflow's aggregate status on master, where the full grid runs. Coverage is not uniform — some suites run on one pytorch only, and a pull request runs less than master does. What each column covers.

docker run --rm -v "$PWD:/data" -v "$HOME/.cache/huggingface:/cache/huggingface" \
    espnet/espnet:inference-cpu-latest asr /data/audio.wav

The second mount keeps the downloaded model between runs. With a GPU, use espnet/espnet:inference-gpu-latest, --gpus all and --device cuda; on a Linux host add --user "$(id -u):$(id -g)". The other images are in docker/.

| Demo | | Last run | | :-- | :-- | :-- | | Spoken dialogue — listen, think, speak | | | | Speech recognition, in any of 151 languages | | | | The words appearing as the audio arrives | | | | Speech translation — the same model, a different task symbol | | | | Text-to-speech, one voice and then 128 | | | | Speech enhancement, and what it did to the signal-to-noise | | | | Speaker verification — two recordings, one score | | | | Neural codecs — a waveform as a few integers a frame | | |

Each runs top to bottom on a CPU and pins the release it was checked against. The second badge is that notebook being executed cell by cell every Sunday, so a red one names the demo that broke rather than leaving you to find out by opening it. More, including the CMU course material: espnet/notebook.

cd egs3/librispeech_100/esp2_asr
train=conf/tuning/training_e_branchformer.yaml   # the config the model was trained with
python run.py --stages pack_model  --training_config $train --publication_config conf/publication.yaml  # -> exp/.../model_pack
python run.py --stages pack_demo   --training_config $train --demo_config conf/demo.yaml                # -> demo/
python run.py --stages upload_demo --training_config $train --demo_config conf/demo.yaml                # needs hf auth login

Run the packed app locally with python demo/app.py.

Kaldi-style complete recipe

• Support numbers of ASR recipes (WSJ, Switchboard, CHiME-4/5, Librispeech, TED, CSJ, AMI, HKUST, Voxforge, REVERB, Gigaspeech, etc.)
• Support numbers of TTS recipes in a similar manner to the ASR recipe (LJSpeech, LibriTTS, M-AILABS, etc.)
• Support numbers of ST recipes (Fisher-CallHome Spanish, Libri-trans, IWSLT'18, How2, Must-C, Mboshi-French, etc.)
• Support numbers of MT recipes (IWSLT'14, IWSLT'16, the above ST recipes etc.)
• Support numbers of SLU recipes (CATSLU-MAPS, FSC, Grabo, IEMOCAP, JDCINAL, SNIPS, SLURP, SWBD-DA, etc.)
• Support numbers of SE/SS recipes (DNS-IS2020, LibriMix, SMS-WSJ, VCTK-noisyreverb, WHAM!, WHAMR!, WSJ-2mix, etc.)
• Support voice conversion recipe (VCC2020 baseline)
• Support speaker diarization recipe (mini_librispeech, librimix)
• Support singing voice synthesis recipe (ofuton_p_utagoe_db, opencpop, m4singer, etc.)

ASR: Automatic Speech Recognition

• State-of-the-art performance in several ASR benchmarks (comparable/superior to hybrid DNN/HMM and CTC)
• Hybrid CTC/attention based end-to-end ASR
• Fast/accurate training with CTC/attention multitask training
• CTC/attention joint decoding to boost monotonic alignment decoding
• Encoder: VGG-like CNN + BiRNN (LSTM/GRU), sub-sampling BiRNN (LSTM/GRU), Transformer, Conformer, Branchformer, or E-Branchformer
• Decoder: RNN (LSTM/GRU), Transformer, or S4
• Attention: Flash Attention, Dot product, location-aware attention, variants of multi-head
• Incorporate RNNLM/LSTMLM/TransformerLM/N-gram trained only with text data
• Batch GPU decoding
• Data augmentation
• Transducer based end-to-end ASR
• Architecture:
• Custom encoder supporting RNNs, Conformer, Branchformer (w/ variants), 1D Conv / TDNN.
• Decoder w/ parameters shared across blocks supporting RNN, stateless w/ 1D Conv, MEGA, and RWKV.
• Pre-encoder: VGG2L or Conv2D available.
• Search algorithms:
• Greedy search constrained to one emission by timestep.
• Default beam search algorithm [[Graves, 2012]](https://arxiv.org/abs/1211.3711) without prefix search.
• Alignment-Length Synchronous decoding [[Saon et al., 2020]](https://ieeexplore.ieee.org/abstract/document/9053040).
• Time Synchronous Decoding [[Saon et al., 2020]](https://ieeexplore.ieee.org/abstract/document/9053040).
• N-step Constrained beam search modified from [[Kim et al., 2020]](https://arxiv.org/abs/2002.03577).
• modified Adaptive Expansion Search based on [[Kim et al., 202