It consisted of three large and two small town gates and several round towers. Today, only the remains of the town wall in Weierhohl with the witches' and angels' towers still exist. The Hexenturm from the 14th century is the oldest surviving secular building in the town. The name could refer to the persecution of witches, during which it may have served as a prison. However, there is no evidence of this. Olpe local historian Dr. Manfred Schöne assumed that the name derives from "Hessenturm".
In addition, the memorial designed by Ewald Mataré has been located below the town wall since 1963 in memory of the victims of the war and as a memorial for the living, as well as the Bracken Monument since 1996.