Hello growalonegame,
Regarding the Game Recording Mode, please refer to the following page and try recording again.
https://www.bandicam.com/how-to/game-recorder/
Please note that the 'Game Recording' mode can record the streaming videos that run on DirectX (Or OpenGL), such as various 2D/3D games (World of Warcraft, Minecraft, RuneScape, Aion, StarCraft 2, Maplestory, Guild Wars 2...), internet TV streaming videos, iTunes, Windows Media Player, HDTV capture card.
If you still have the problem, please let us know your recording target.
We look forward to your reply.
Thank you.