In this paper, we propose a Unified Speech Enhancement and Editing (uSee) model with conditional diffusion models to handle various tasks at the same time in a generative manner. Specifically, by providing multiple types of conditions including self-supervised learning embeddings and proper text prompts to the score-based diffusion model, we can enable controllable generation of the unified speech enhancement and editing model to perform corresponding actions on the source speech. Our experiments show that our proposed uSee model can achieve superior performance in both speech denoising and dereverberation compared to other related generative speech enhancement models, and can perform speech editing given desired environmental sound text description, signal-to-noise ratios (SNR), and room impulse responses (RIR).
Overall Demo:
Source
Source
Source
Ablation Experiments:
Source
Without Acoustic Prompts
With Acoustic Prompts
Overall Demo:
Example sound
Example sound
Example sound
Contollable Generation Effect:
Raw audio
Add small room RIR
Add medium room RIR
Add large room RIR
@article{yang2023usee,
title={uSee: Unified Speech Enhancement and Editing with Conditional Diffusion Models},
author={Yang, Muqiao and Zhang, Chunlei and Xu, Yong and Xu, Zhongweiyang and Wang, Heming and Raj, Bhiksha and Yu, Dong},
journal={arXiv preprint arXiv:2310.00900},
year={2023}
}