Right, so one thing I discovered putting together this research: there's more than one Montagu House. I know. Try and calm yourself. The first one, built in 1675 and belonging to Ralph Montagu, would eventually become the home of the British Museum. This is not that Montagu House.
In 1731, Ralph's son John sold his father's house and bought up three adjoining plots of land in Whitehall. On these plots, he sought to build a larger, grander house than his father's - Sigmund Freud would've had some things to say about this. Mind you, that bloke said a lot of stuff, and most of it was about your mum.
Hey! My first "your mum" joke!
Anyway, in reality, the second Montagu House was a relatively modest mansion by the standards of the day - keep in mind the property was 4,300 square feet, so to hell with eighteenth-century standards. John Montagu died in 1749 and the house became the property of his daughter Mary and his son-in-law George Brudenell. That's right - James Brudenell's great-grandfather owned this house for about 40 years.
Or at least, he owned the house that used to be here.
In 1859, the house was demolished by its latest owner, Walter Francis Scott, who was James Brudenell's second or third cousin, if I'm tracking this family tree correctly. I hope you're taking notes. In three years' time, Scott had built in its place this French Renaissance-inspired mansion, which contemporaries called a "palatial residence." It was designed by the architect William Burn, known for the hospitals and castles he designed in his native Scotland.
The house was acquired by the British government in 1917 and converted into office space. That is the dream of every piece of classic architecture, isn't it? "Oh please! Allow me one day to be converted into a workspace housing petty squabbles and bad coffee and Maureen who can't work Excel and lit by only the most jaundicing of stip-lighting!" Anyway, it was demolished in 1950, so that dream was over pretty fast.