Abstract:

Detection and Classification of Acoustic Scenes and Events 2023 Challenge BEATS-BASED AUDIO CAPTIONING MODEL WITH INSTRUCTOR EMBEDDING SUPERVISION AND CHATGPT MIX-UP Technical Report Shih-Lun Wu 1, Xuankai Chang 1, Gordon Wichern 2, Jee-weon Jung 1, Franc ̧ois Germain 2, Jonathan Le Roux 2, Shinji Watanabe 1 1 Language Technologies Institute, Carnegie Mellon University, Pittsburgh, PA, USA {shihlunw, xuankaic, jeeweonj, swatanab}@andrew.cmu.edu 2 Speech & Audio Team, Mitsubishi Electric Research Labs, Cambridge, MA, USA {wichern, germain, leroux}@merl.com ABSTRACT DCASE 2023 Task 6A, automated audio captioning (AAC), aims at generating informative descriptions for various sounds from nature and/or human activities. Our AAC system follows the sequence-to- sequence (seq2seq) architecture. The audio encoder stack is com- prised of a frozen BEATS Transformer followed by a 2-layer Con- former. The BEATS module, which has been pretrained on both masked audio token prediction and audio event classification, ex- tracts fine-grained (i.e., ≈ 50 Hz) audio features, while the Con- former downsamples and summarizes the audio features before they are cross-attended by the BART text decoder. Besides the autore- gressive negative log-likelihood (NLL) loss computed on decoder outputs, we simultaneously apply an audio-text contrastive loss on our encoder output to infuse language modality knowledge into it. Specifically, we feed ground-truth captions into INSTRUCTOR Transformer, a state-of-the-art text embedding model, and teach our audio encoder to predict the INSTRUCTOR text embeddings through InfoNCE loss. In addition, we leverage ChatGPT to produce cap- tion mix-ups (i.e., grammatical and compact combinations of two captions) which, together with the corresponding audio mixtures, increases not only the amount but also the complexity and diversity of our training data. During inference, we employ nucleus sampling and a hybrid reranking algorithm that considers both likelihood and audio-caption representation similarity. Combining our efforts, our best single model and ensemble system achieve 0.326 and 0.336 SPIDEr-FL scores, respectively, on the Clotho (V2) evaluation split.