The determination of the elements of the CKM matrix provides a stringent test of the Standard Model of Particle Physics, which predicts the CKM matrix to be unitary. Finding possible (small) deviations from unitarity requires high-precision estimates of the individual matrix elements. The CKM matrix elements Vus and Vud can be obtained from leptonic kaon and pion decays, respectively. In recent years, lattice calculations of the required hadronic decay constants have reached sub-percent precision, where isospin breaking corrections become relevant. In this talk I will discuss about calculating such isospin breaking corrections on the lattice and present the status of our calculation directly at the physical point.