I think the Listen Mode, as most people agree, is a bit too much. Though I think it shouldn't be removed or turned off either.
Instead I've thought of ways to change it:
1. Only people who make actual noise should show up, like clickers on patrol, or others walking or talking/growling. People who are noiseless shouldn't magically show up, unless Joel has Xray vision.
2. Also it shouldnt give exact location, but just a DIRECTON and INTENSITY from which the sound is coming from (like a blurry region, whose size corresponds to the loudness of the noise made.)
3. Listen mode could only be used when Joel stands completely still, and background sounds such as running water should dampen his hearing.
This should espacially be used in multiplayer, so players who are doing a good job trying to be noiseless can't just be seen through walls. And so that players can't pinpoint the location and body stance of others, but rather get a general region.