The scaled mode uses a toggle keyboard shortcut Host Key + З. By default the Host Key in VirtualBox is the right Ctrl key on Linux і Windows. On Mac it is the left Command Key. If you have changed your Host key you can check it by going to File -> Preferences -> Input -> Virtual Machine -> Host Key Combination .
If you are in scaled mode pressing the Host Key + З will exit scaled mode.