This was the first covered market in New York. It started as a one-story building in 1675, but it was replaced with the building you see here in 1752. The lowest floor is for the market, and the second floor contains rooms that could be used for concerts, public meetings and balls.
Balls as in dances. Not just a big room full of balls.
It's no accident that this building bears a resemblance to Faneuil Hall in Boston. It was common architecture for markets at the time. Private houses didn't have the space for large numbers of people, so these buildings combined open space for a market, with closed-in space for the upper crust to have fancy parties.
Of course, public rooms could be used by the rabble as well. The Sons of Liberty brought crowds here, notably to choose delegates for the Second Continental Congress. (Technically, that was the New York Assembly's job, but they'd refused to send anyone.)
After the Revolutionary War ended, this building was the first home of the Supreme Court. Its first session began on February 2, 1790. It didn't stay here for long however, and moved to Philadelphia in 1791.