The mansion's grounds were extensive, featuring not only a grand villa with a sizable inner courtyard, but also other, less luxurious, amenities for the servants working the adjoining plantation fields. Like many contemporary residencies of colonial governments, the mansion possessed its own prison facilities.
In July 1715, Torres inducted Julien du Casse, Woodes Rogers and the pirate Edward Kenway, who had assumed the identity of Duncan Walpole, into the Templar Order at the mansion. After the ceremony, the Templars discussed their plans for the future, which focused on eliminating various leaders of Assassin bureaus and finding the Observatory. The Sage Bartholomew Roberts was temporarily detained in the prison shortly thereafter, but managed to escape fairly quickly.
The mansion continued to house new governors after Torres' death until 1791, when a new mansion was constructed across the square.