NVENC H.264 has a resolution limit of 4096x4096, which made it unfit to handle 8K. Leave alone its missing support for 10-bit.
And going to uncompressed solution is apparently not an option (do the math, the bitrate for 8K... SSD arrays? may work), while x264vfw (RGB) has performance issues (x264, or H.264 is designed for YUV colorspace, couldn't be effectively utilized when using RGB), and I can't seem to get it work with YUV 4:4:4 (bug?..). Anyway 10-bit for software encoding (realtime) seems unreal (performance limitation, unless you've got some ridiculous PC...). Retreat to 4:2:0? it defies the whole point (why not use lossy encoding then...), so...
NVENC H.265, but Bamdicam has limited its usage to lossy encoding only, while NVENC H.264 for the same container (.mp4) can use lossless. (what's the point?.. indeed H.264 is known to be more efficient handling lossless but...)
And maybe Matroska (.mkv) container support? just to save a few seconds remuxing...