Allocate plenty of video memory for the virtual machine. At least allocate 128mb or 256mb if you have enough system resources.
To set the video memory:
Machine Settings->Display->Video Memory, set as 128 MB
In the Video Capture tab, you can check/change the captured video file path.
To change the settings check the Enable Video Capture checkbox.
Video files are stored in the same directory as the VM by default. To view the VM directory, right click the VM in the VirtualBox VM list and choose “Show in Explorer”.
Keeping the defaults is fine.
Uncheck the Enable Video Capture checkbox once you have finished changing the setttings otherwise the video capture will start as soon as the virtual machine is started. Click OK to save any changes and exit.
Now the virtual machine is ready for video capture.
Click View->Video Capture to start the video capture.
You will see the video capture animated icon in the bottom right hand side.
To stop video capture right click the video capture animated icon and click and choose “Video Capture”.