When I came upon the Dark Tower quest I stood looking at the Tower with the small cutscene. After I was moved inside the Dungeon and was speaking with an invisible Boss man (Guess his name didn't stick with me). I intimidated him into giving me the gem. The teleporter lit up and I received a message to head back to town.
For my good or ill, I wandered across the hall before leaving and the battle there ensued. I was third level at this point, which I made at the end of the Library dungeon. I made it through the battle and made level 4 off this one battle. I seen the campfire and went about figuring out how to get to it, discovered the puzzle journal and figured that out. Made it to Akasha's area but she wasn't home. I was stopped at the top of the stairs with the door not opening.
I backtracked and left via the teleporter, exited the courtyard and was put right back at the courtyard. I explored the area and found the various entrances. No attacks during this time at all. I used one of the drain shafts, and discovered the proper way into the castle. I proceeded to clear it, and was able to fight Akasha this time around. Continued forward to the bossman again, but no teleporter was highlighted.
I backtracked all the way out of the castle, was walking out the courtyard and explored to the left and right, got Agniz's bones quest, found those, and still couldn't leave.
Back in the sewer drain, fast travel to the bossman, walk to the teleport area and when the characters stepped into the area, it light up, and teleported out. Followed a short path and was able to zone back to town.
The quest would nice and engaging and would have been even more fun if I did it the proper way the first time around.
Crap happens. Still enjoying the game and looking forward to the next area.