Non-thermal emission represents a crucial tool in the comprehension of the high-energy sky. Yet, mysterious excesses exist. Among them, the Fermi GeV excess and the 511 keV line emission still lack a definitive explanation and might thus point towards new sources and/or emission mechanisms. I plan to conclusively test the hypothesis that these two emissions arise from faint point-source populations, linked to binary systems in the Galactic bulge, with a new multi-wavelength analysis. I will use gravitational wave signatures as further leverage in the modelling of Galactic binaries, and thus probe the synergy of the high-energy sky with this new observational window for the first time. My research will also sharpen our diagnostic capabilities on dark matter and boost the discovery potential of future searches building on the new techniques developed. The characterisation of point-source populations in the bulge will be a leap forward in high-energy astrophysics and Galactic astronomy.