Ableton 11.1 Huge CPU spikes and audio glitches when dropping plugins. Macbook Pro M1x 100% Native

[Krubbadoo]

Hi.
Got this new rig. Amazing laptop for everything except Live...
Whenever I am on Internal speakers, buffer @ 128 (tried others too) I get huge CPU spikes.
Even with only 1-2 tracks playing, let's say that I drop a device (audio plugin) while audio is playing on a track, I see the current CPU go in the hundreds/thousands and can hear pops in the speakers as if the audio is glitching. Like LOUD pops even if I monitor at the lowest volume.
System is 100% native. No plugin duplicates between VST2 and VST3. Also tried running only AU plugins with same results.
This is with no audio interface @ 44.1 khz. Same result with other sample rates. 100% native plugins, no Rosetta.
Please help

Thanks

[thecontrolcentre]

Have you tried pressing space bar before loading a new device? Why do you need to keep audio running at all times?