The overworld and chibi Link (when it released) were my biggest hangups. Every Zelda since OOT is superior in some way or another -lets face it BOTW has by far the best overworld, its flaws are in other areas.
OOT just hasn't really been topped as a complete, flawless manifestation of 3D Zelda. Sure, it could be improved because it was their very first try and on N64 -the same way Resident Evil was a masterpiece that still benefited from some tweaking in the REmake. In that way OOT is the one that stands to benefit the most from a legit remaster/remake.
I think the decision to only release their most iconic 3D titles for portable was very intentional, it's the Disney Vault mentality. We got Prime because it isn't a big seller and therefore not a high stakes deal to them. Even that took 6 years. They will save OOT to bail them out of a fuck up -if/when a console doesn't do as well as they hoped, next holiday bam OOT remake. Besides, if they did a full on HD remake of OOT, it would make other Zelda titles look shitty and would suddenly have to try harder with their new games. Their worst nightmare if you look at TotK.