Fly is just broken.
I'm in the library of the magic tower. I cast Fly since I can't get to the room in the very back with Jump or Misty Step. I had hoped it might also let me get up to the chest in that room with the ladder that does not work. Not only can I not fly into the room in the back, there are 1 square jumps that I cannot even fly across. I can't fly to most of the locations in the library that I could walk to.
I can play with the "horizontal plane" toggle, but that just shows me lots of red unreachable markers and even if they highlight cyan, I just get "No path to destination".
FYI: You can at least get to the chest on the little tower with the ladder that isn't working by using fly. You just have to cast it when you are close enough that you never have to cross any gaps.