We have no idea why you can't record BF3 and dota2.
1. Can you record other games?
2. Do you use the latest version of Bandicam (18.104.22.168)?
3. What is your graphic card model?
4. Can you see the green FPS in battlefield 3?
5. Did you check the shortcut key of Bandicam (F12 key)?