Advantage and Disadvantage don't cancel out

Context: Ambushed by bandits on the road, party member was asleep, woken up by an attack and was still prone before their turn. Enemy bandit melee attacked my prone party member. Paladin party member next to them, with a shield, was given the choice to impose disadvantage and took it. Instead of the advantage and disadvantage cancelling out to a normal roll, the enemy instead still attacked with advantage. Only had this occurrence once but if it happens any more times I'll report my findings here.