Visual data association aims to find task-specific mappings involving visual data. Two significant examples are the mapping of physics models to complex scenes for planning overtaking manoeuvrers in autonomous driving, or matching collections of 3D shapes for medical analysis. Despite the high relevance of visual data association, its progress has not kept pace with the revolutionary developments fuelled by recent deep learning advances: existing data association machinery lacks theoretical guarantees (e.g. global optimality, or structure such as geometric consistency in 3D shape matching) that are critical for high-stakes settings, or suffers from poor scalability. Moreover, current procedures fall short of understanding complex interconnections across different observable entities (collections of e.g. objects or scenes). The vision of Harmony is to tackle these shortcomings by harmonising the complex interconnections between observable entities and underlying fundamental principles (e.g. geometry, or physics). This research direction is challenging, largely unexplored and will require to break substantially new ground at conceptual, algorithmic and practical levels simultaneously. Harmony is organised into four complementary challenges: Challenge A addresses global optimality and scalability for 3D shape matching; Challenge B addresses structure and dynamics inference from static images; Challenge C addresses non-linear synchronisation in data collections defined over graphs; Challenge D will exploit synergies and cross-fertilise insights across Harmony. Overall, Harmony will benefit both researchers and practitioners by providing solutions to more complex tasks in practically relevant settings (e.g. geometrically consistent medical shape analysis, or physics-based scene understanding).