Abstract:

Standard contrastive learning approaches usually require a large number of negatives for effective unsupervised learning and often have slow convergence. We suspect this is due to the sub-optimal selection of negatives that offer the most contrastiveness to the positives. To this end, we present max-margin contrastive learning (MMCL), inspired by support vector machines (SVM). We select useful negatives as the sparse support vectors via solving a quadratic optimization problem, and contrastiveness is enforced by maximizing the decision margin. As SVM optimization can be computationally demanding, especially in an end-to-end learning setting, we present simplifications to the problem formulation. We provide experiments using our approach on standard vision benchmark datasets, demonstrating better performances in unsupervised representation learning over state of the art, while having better convergence properties.