Have you tried measuring different part of the application and see if there is a particular part that is slower when it doesn't have the focus ?
Is your application covering the entire screen or is it a window ? When a window covers the entire screen, the window compositor behaves differently (
DirectX 12: Presentation Modes In Windows 10, I don't think OpenGL is mentioned in the video but I suppose something similar could happen).