Ugh. That's why I mute everyone except friends.
That's why you have to make sure your mic icon doesn't show up in the lobby.
Then, you blast your music during the loading screen, since they probably didn't mute you.
This should be fixed in the PS4 version at least.