Sir William Johnson built Johnson Hall in 1763. Well, I say "he" built it. He had more than a little help. He had 60 slaves to do the manual labour, making him the biggest slaveholder in the north. The property is more homestead than house - there was a sawmill and a grist mill on the property, and tenant farmers worked the lands.
The exterior is wood painted to look like stone, which is always a classy look, but the block houses you see on either side are actual stone, meant for defense. When the house was built, the French and Indian War had just ended, so it made sense to have the house ready for an attack. That, or William Johnson made some well-armed enemies in his business dealings.