TR2024-006

TS-SEP: Joint Diarization and Separation Conditioned on Estimated Speaker Embeddings


Abstract:

Since diarization and source separation of meeting data are closely related tasks, we here propose an approach to perform the two objectives jointly. It builds upon the target- speaker voice activity detection (TS-VAD) diarization approach, which assumes that initial speaker embeddings are available. We replace the final combined speaker activity estimation network of TS-VAD with a network that produces speaker activity estimates at a time-frequency resolution. Those act as masks for source extraction, either via masking or via beamforming. The technique can be applied both for single-channel and multi-channel input and, in both cases, achieves a new state-of-the-art word error rate (WER) on the LibriCSS meeting data recognition task. We further compute speaker-aware and speaker-agnostic WERs to isolate the contribution of diarization errors to the overall WER performance.

 

  • Software & Data Downloads

  • Related Publication

  •  Boeddeker, C., Subramanian, A.S., Wichern, G., Haeb-Umbach, R., Le Roux, J., "TS-SEP: Joint Diarization and Separation Conditioned on Estimated Speaker Embeddings", arXiv, March 2023.
    BibTeX arXiv
    • @article{Boeddeker2023mar,
    • author = {Boeddeker, Christoph and Subramanian, Aswin Shanmugam and Wichern, Gordon and Haeb-Umbach, Reinhold and Le Roux, Jonathan},
    • title = {TS-SEP: Joint Diarization and Separation Conditioned on Estimated Speaker Embeddings},
    • journal = {arXiv},
    • year = 2023,
    • month = mar,
    • url = {https://arxiv.org/abs/2303.03849}
    • }