I think I had this same interesting problem when I purchased a set of Creative Labs Sound Blaster Tactic 3D headphones. They came with their own USB dongle that acted as a sound adapter ('sound card'), routing all sounds through it in Windows as a sound device. It sounded great - the setup was THX-certified when the headphones were used together with the special USB connector, as it added enhanced audio features and effects.
The problem I had with it was the same as yours it seems... Once I plugged in the headphones, Windows made the headphones my Default sound device for output and I could no longer record the sound output in Bandicam (or Fraps or Dxtory or MSI Afterburner, nothing). I could change the output device, but that would just make the sound come out of other things and not the headphones, and in the end I could not record from the headphone output no matter what combination I tried. I assumed there was some incompatibility between how sound adapters are handled, whether it was a problem with Windows, or game recording applications, or the software that came with the headphones - and I simply ignored it for the time being, content to plug the headphones in directly to the sound card and record the Stereo Mix output for all Bandicam recordings. I am missing out on all the special bells-n-whistles of the headphones, but it works fine and I have used it that way since.
This is an odd problem and may not be Bandicam's issue; but if it is figured out, it would be great to be able to record from these headphone-soundadapter-combination devices, I suspect there are more of them out there than previously thought. I wish you luck with it and will be watching this thread - it would be great to get it working...
Short Version / TLDR:
Headphones that came with a USB sound adapter cannot be recorded through in Bandicam for me as well
("Hope That Helps!") ~ Troy