My point is, you would keep the switch on no matter what and flick it off and on again when switching the lights on/off

Assuming technically possible, this solution requires no other hardware:

Is it possible to make the smart plug default to the opposite of the last state? You could use the wall switch like a pushbutton (cut and restore power) and it would toggle, then maintain control from HASS. You might be able to save the last state of the smart plug, detect when it comes online after a power failure (theoretically), and set it to the opposite of the previous state in an automation.