These buildings were used to store gunpowder, which was both a necessity in colonial America, and extremely dangerous to have around. This is not a place you’d want to move next door to. Spilled powder, damaged barrels, improper airing of building... any one of these was hazardous, and a random spark could set off powder in the air or on the ground.
People who worked in the magazines were warned to check their pockets for stray matches, or made to wear clothes with no pockets, to eliminate the possibility entirely. Their shoes had to be made without nails, or covered with 'magazine slippers' to reduce the risk of sparks. All the tools used in the magazines were made of wood and copper.
Generally, powder magazines were built in isolated areas (like the one on the Boston Common). Of course, if you're inside a fort you need your gunpowder to be close at hand, so the buildings were designed to collapse in on the fire and smother it, rather than spreading it outward.
I probably don't need to tell you this, but you're not really allowed to smoke around here.