When I'm just standing there, looks great. But when all that stuff is going on (lightning, spells, etc), quality goes down. Is there a setting doing that or is that just how h264 encodes?
edit - You man need to look in 1080p to see what I'm talking about
Is there a setting doing that or is that just how h264 encodes?
Both yes and no.
You're seeing the artifacts just because youtube's bitrate doesn't go above 6Mbps. There's really nothing you can do about improving the quality in 1080p, but... What you can do is to either capture at higher resolution (1920x1200 (16:10), 2560x1440p (16:9) and above) or upscale the video to higher resolution with a video editor. If the video is bigger than 1080p YT will apply different profile which will have higher bitrate / better encoding.