The house, originally known as Forest Court, was built by Thomas Sandby for Countess Waldegrave in the 1760s. She named it Gloucester Lodge following her marriage to the Duke of Gloucester in 1766.
The house was bought by William Harcourt, 3rd Earl Harcourt in 1781 and, after remaining in the Harcourt family for over a century, it was acquired by Sir Francis Barry, 1st Baronet in 1872. Barry made substantial alterations in the Châteauesque style and renamed it St Leonard's Hill. Following the death of Lady Barry in 1924 the house was largely demolished.