Play in the resolution you want, but decide if you want to record on 720p or 1080p.
Make sure, like rubyeye said, to run a resolution that is
equal or greater than the resolution you are recording at. IMHO, for doing gameplay footage, Let's Plays and such, 720p is fine. There's still plenty of folk who completely ignore 1080p due to long loading times, having low bandwidth available or just not feeling it is cost-effective. (You know, the amount of time it has to load before you see the footage vs. the increase in perceived quality between 720p and 1080p.)
Use x264 and set it to quantizer based. 21-25q is your golden ticket for 'good enough' quality and efficient filesize.
I'd love to use two-pass, which is by far the best option when trans-/encoding with x264, but for whatever reason I can't get it to work. That's another story, though!
If you want to record a game or software that doesn't want to be hooked, have it up fullscreen (or windowed, doesn't matter) and use 'record rectangle on screen'. You can set it up to your desktop resolution, effectively making 1080p footage if you're playing a game fullscreen at 1920 × 1080, but it doesn't get hooked by Bandicam.
If you're playing a very old game, you might want to set Bandicam to record at a certain resolution as well, so that it 'enlarges' the footage without making much of a mess. (I hope, I haven't tried this myself. But I imagine this method works better than upscaling it with video editing software.)
That's all I can think of right now, my mind is going all fuzzy!