Hey,
Not to be rude or anything, but it might be worthwhile looking into 64-bit codecs.
These allow for more system resources to be used more efficiently and will give Bandicam more power whilst recording.
I know it basically means creating an entire new program from scratch to support 64 bits codecs, but with other software and the codecs themselves slowly switching over to 64bits, it is an option you might want to consider looking in to for the future.
I'm not sure if the new h265 HEVC codec has a fully fledged 64bit counterpart to it's original, but it has been released and it will become the mainstream video codec soon enough.
Like with h264, I'm pretty sure there will be an x265 free version.
That codec needs much less bandwidth with comparable quality, compared to h264, but 'requires more computational power'. It's not mentioned very clear, but I figure that means it will demand more system resources than h264. If it is 64bits, it can use a lot more of your system to keep it smooth and stable than with 32bits.
Just my two cents.