In "By the Waters of Babylon", why do you think it is forbidden for anyone but a priest to visit the Dead Places?

Your excellent question points towards the setting of the story, and in particular how the story shows that in this dystopian world man has slipped back into the dark ages - a time of ignorance, taboos and unquestioning authority. Consider how we as a race learnt about danger and how gradually this awareness permeated our consciousness. This too is what has happened in this world - man has gradually learnt by trial and error of the dangers of entering what we recognise as sites with a high level of radioactivity, and this learning has been enshrined as a taboo. In this world, taboos are controlled and maintained by the Priests. Consider the first paragraph of the story:

The north and the west and the south are good hunting ground, but it is forbidden to go east. It is forbidden to go to any of the Dead Places except to search for metal, and then he who touches the metal must be a priest or the son of a priest. Afterward, both the man and the metal must be purified. These are the rules and the laws; they are well made.

Note the number of times the phrase "It is forbidden" is repeated, and consider to the unthinking obedience that the narrator displays. The priests have developed their own "charms" and "spells" to protect them and they thus have the power and control to enter the dead places and to purify themselves afterwards. Thus the setting establishes the potential danger to our civilisation of how far we can slide back in the event of a nuclear cataclysm.