If I may pop in, this is how most Video Stream Data is handled, actually. The closest you are going to get is something like Lagarith, unless you go with Full RGB or RGBA - but then the sizes will be huge, to keep of the details. The usual MPEG-4 Codecs, at high Bitrates, should be enough for compressing video and still be quite watchable - but if you demand perfect quality, you will have to use the larger, lossless Codecs.
Suggestion:
Try GPU-Accelerated codecs (H.264/AVC compressed using your videocard, or even h.265 if you can utilize it). They will be faster and have pretty good quality these days (but not 'perfect quality', sorry).
If you have onboard video (on the mainboard or on the CPU), then this is not an option.
You can still get good quality using high bitrate h.264/AVC codecs, but again, it cannot be 'perfect quality'.
You can also try 'Super-Sizing', where you Record at a higher resolution than the game runs at (playing at say 800x600, but telling it to record at 1600x1200). Then, when you compress the video for YouTube/Streaming/Sharing/etc in whatever Editing Program you use, the 'resizing back down' may not lose as much detail - however everything will still be blurred slightly, so the result may be close to the same anyway. It is just an idea that may help, depending on the game material.
Unfortunately, with compressed video as we handle it in modern times, there has to be a tradeoff between size and quality.
If you want perfect quality, it will be huge in size. There is no way around that.
If you don't mind lesser quality, it will be smaller - and the more Quality you are willing to give up, the smaller you can create your video file size.
In the end, you will have to make a decision on how much quality is 'good enough' for what you are doing, then try find the smallest size you can obtain from that. Good Luck with it!
I hate to mention
a competitor here, but how is Camtasia Studio able to record this while keeping a very low filesize:
(No, that's not a screenshot)
But I can't use Camtasia Studio because it only officially supports up to 30fps, anything above that is not good.
And overall, Bandicam is much cooler & better in my opinion.
And I have a GTX 970 for a GPU and a i7-4790K for a CPU.
So I should have at least some power there.
Hello again,
Yes, you have good system components; but unfortunately those do not dictate how good ("Quality") video/image compression is, only the speed at which it is achieved (how long the file will take to Export, for instance).
I believe Camtasia also uses their own proprietary Codec there, as many other applications do, to create a 'temporary recording' that looks like that; but if you were to create/export a file to upload to YouTube (for example, any file that you can 'actually use/edit'), I assure you it will not stay looking that way, sadly.
Yes, Bandicam is cool
but your system specifications ("power/speed") do not affect how good the video data will look, only how fast/easy it is produced.
Bandicam is capable of creating perfect quality video - but it will be huge in size - just like any other game recording program.
It can also produce smaller size video - but the quality will be less, the more you reduce the file size - just like any other game recording program.
You cannot have "perfect quality" and small file sizes at the same time, unfortunately.
If you want, try to do a Search for Codecs that others use to record similar games or television/movie streams and what Codec they use and at what settings - this will give you a 'starting point' as to what to configure Bandicam to do, and you can try to improve it from there.
Good luck with it, try to have fun figuring a good balance between Quality and Filesize!