It appears to be a container problem, simply change the video's file extension from .mp4 to .m4v. m4v is Apple's version of mp4 and for some reason Premiere likes it better.
In the tests I've done performing the above change will prevent your audio from going out of sync while editing in Premiere Pro. I'd be excited to hear the results from others who've tried this.