The scaled mode uses a toggle keyboard shortcut Host Key + Ц. By default the Host Key in ВиртуалБок is the right Ctrl key on Линукс и Виндовс. On Mac it is the left Command Key. If you have changed your Host key you can check it by going to File -> Preferences -> Input -> Virtual Machine -> Host Key Combination .
If you are in scaled mode pressing the Host Key + Ц will exit scaled mode.