This project presents a contrastive learning framework for cross-modal discrimination of video from audio and vice-versa. We show that optimizing for cross-modal discrimination, rather than within-modal discrimination, is important to learn good representations. With this simple but powerful insight, our method achieves highly competitive performance when finetuned on action recognition tasks. We also generalize the definition of positive associations beyond instances by exploring cross-modal agreements. Similar instances in both the video and audio feature spaces are grouped together to form the instance's positive set. We show that cross-modal agreement creates better positive sets for contrastive learning, allowing us to calibrate visual similarities by seeking within-modal discrimination of positive instances, and achieve significant gains on downstream tasks.

paper code

Cross-modal instance discrimination approaches introduce two sources of training noise: uncorrelated audio and video signals within the same instance produce faulty positives; faulty random sampling of negative instances yield faulty negatives (i.e. negatives that are semantically similar to the base instance). In this project, we study the detrimental impact of these two sources of training noise, and propose solutions for them. We validate our contributions through extensive experiments on action recognition tasks and show that they address the problems of audio-visual instance discrimination and improve transfer learning performance.

paper code

Methods such as audio-visual instance discrimination predict whether audio and video clips originate from the same or different video instances. While these approaches learn high-quality representations for downstream tasks such as action recognition, their training objectives disregard spatial cues naturally occurring in audio and visual signals. To learn from these spatial cues, we proposed the audio-visual spatial alignment task, which requires the model to reason about local correspondences. We also introduce a cross-modal feature translation architecture inspired by transformers that is capable of reasoning over the full spatial content of the 360 video.

paper code & data video

Beyond representation learning, we also studied how well local associations between audio and visual signals can be predicted, not just as a means to learn visual models but as an end in itself. To showcase this ability, we introduce an approach that converts mono audio recorded by a 360° video camera into spatial audio, a representation of the distribution of sound over the full viewing sphere. Inspired by the spatial audio generation process, we developed an effective architecture for this task that performs sound source separation and localization independently. Using our approach, we show that it is possible to infer the spatial location of sound sources based only on 360° video and a mono audio track.

paper code & data video project