Richmond House was first built as his London townhouse by Charles Stewart, 3rd Duke of Richmond, 6th Duke of Lennox (1639-1672) of Cobham Hall in Kent, an English nobleman of Franco-Scottish ancestry and a 4th cousin of King Charles II of England. It was built shortly after the Restoration of the Monarchy of 1660 when King Charles II returned to Great Britain from his exile in France during the Civil War and Commonwealth. It was built on the former bowling green of the royal Palace of Whitehall, at the southern end of the Privy Gardens. Its west side looked onto Whitehall, but the main front looked northward towards the Banqueting House and Charing Cross. The 3rd Duke of Richmond died without issue in 1672 but his widow remained in occupation until her death in 1702.
In 1710, during the reign of Queen Anne, the house was granted to Charles Lennox, 1st Duke of Richmond, 1st Duke of Lennox (1672-1723), of Goodwood House in Sussex, the youngest of the seven illegitimate sons of King Charles II (by his mistress Louise de Kérouaille, Duchess of Portsmouth). Charles Lennox had in many ways been placed by the king into the persona of the deceased 3rd Duke of Richmond, whose family was much beloved by the Stuart monarchs. In 1733-4 His son Charles Lennox, 2nd Duke of Richmond, 2nd Duke of Lennox (1701-1750) built a new house adjacent to the first house, to the design of Lord Burlington (1694-1753), the pioneer of Palladian architecture in England. After 1738 he demolished the first house to improve his view. In 1747 he commissioned Canaletto to paint a pair of views from Richmond House, today in the collection of his descendant the 11th Duke of Richmond at Goodwood House. The 2nd Duke also enlarged his country seat at Goodwood House, to the designs of Matthew Brettingham. In 1758 his son Charles Lennox, 3rd Duke of Richmond, 3rd Duke of Lennox (1735-1806) converted part of the house to a school for the study of painting and sculpture and in 1782 remodelled a part of it, to the designs of James Wyatt.
To the immediate west of Richmond House and adjoining Whitehall stood the "House of the Earls of Loudoun and Mar" (joint Secretaries of State of the Kingdom of Scotland 1705-1708/9), built 1687-93 by John Drummond, 1st Earl of Melfort (1650-1715), Secretary of State of the Kingdom of Scotland 1684-9, for use by holders of that office (abolished 1707) and demolished after 1820 to make way for Richmond Terrace built in 1822 and surviving today. It was divided into two separate dwellings, the leases of which were held personally after 1707 by the descendants of Loudon and Mar. Richmond House built by the 2nd Duke and burned down in 1791, stood to its east, closer to the river. The Loudoun and Mar house was acquired between 1766-90 by the 3rd Duke of Richmond, after which it also appears to have become known as "Richmond House".
Richmond House was destroyed by a fire on 21 December 1791, and was not rebuilt by the 3rd Duke or his descendants. The library and art collection were rescued from the fire and were removed to Goodwood House, where the surviving two wings were built (to the designs of James Wyatt) to re-house them.
The fire started in the bedroom of Henrietta Anne le Clerc,[a] called "a protegée of the Duchess" and "a long acknowledged daughter of His Grace", believed to have been the Duke's illegitimate daughter, to whom he referred in his will as "Miss Henrietta Anne le Clerc, who resides with me and though Christened by the name of Anne only is called Henrietta and whom I have [educated?] from her childhood" and to whom he bequeathed an annual income of £2,000 (also leaving £10,000 to each of his three illegitimate daughters by his housekeeper Mrs Bennet). The events are recorded as follows in The Annual Register of 1791:
At half-past eight o'clock, a fire broke out in Miss Le Clerc's apartments on the second floor in Richmond-house, Privy-gardens, which was occasioned by a spark having shot from the fire to the bed furniture, where the young lady lay asleep. The duke was then writing a letter in the library, where the breakfast cloth was laid. In a few minutes afterwards, his grace, the duchess, and Miss Le Clerc, the duchess carrying a favourite dog under her arm, left the house, and the ladies were escorted to the duke of Buccleugh's (i.e. Montagu House) by a gentleman, who appeared to be a friend of the family, and who met this party upon the steps. The duke returned to the yard of his house, and there being then no engines, and very little readiness either in the astonished servants, or the populace, to afford assistance, he seemed likely to be, in a very short time, a witness to the destruction of his entire property there. A gentleman at this time ran up the great stair-case, and presently afterwards, some of the populace encouraged by his example, and entreaty followed. Eight or nine persons seemed then to be employed by his direction in lowering furniture from the windows, and bearing it down stairs. Three looking-glasses, said to be worth twelve hundred pounds, were thus rescued; two large cabinets, containing his grace's papers, were lowered from the rails of the balcony by this unknown gentleman. Upon the whole it appears, that the endeavours then used for the preservation of the valuable furniture and effects were so far successful, that all the papers in the office fronting towards the garden, and appropriated by the duke to ordnance business, are saved; all the furniture of the first floor, even to the hangings of the duke's bed; all his private papers, with the letter which he had left unfinished, and the valuable paintings, are saved. One looking-glass of great value was broken and left behind, the others were carried down the great staircase. The books in the library were saved by being thrown from the windows upon mattresses, which the stranger, who seemed to conduct the whole, had ordered to be placed under them. The model of the new house intended to be built by the duke at Goodwood, and all the valuable busts from the library, were also saved. About one o'clock, the whole roof fell in; three floating-engines on the river played the water on the east-side, and a number of engines in the yard played very rapidly; so that soon after four o'clock they got it nearly under. His royal highness the duke of York, with about 300 of the Coldstream regiment, assisted the watermen, and kept off the mob. During the rage of the fire, a favourite spaniel dog of the duke's was observed at the window of an apartment, jumping and making endeavours to force his way through the glass. His grace offering a reward to any person that would save him, a waterman by means of ladders fastened together, mounted to the window, threw up the sash, and brought the dog down safely. The duke gave him ten guineas and the duke of York one, for this act of humanity and courage. The pictures and most of the numerous writings and curious books which his grace possessed, we are extremely happy to hear, were saved. At such a fire, the loss of property is not the highest consideration; science often suffers irreparably. No lives were lost, nor have we heard of any material accident sustained by the persons who assisted
The lease of the site passed to other ownership and in 1822 was built the surviving structure of eight large terraced houses known as "Richmond Terrace", occupying approximately the same footprint and orientation. These became fashionable private residences, until the 1920s when the leases expired and they returned to use as government offices until the redevelopment of 1982.
In 1987 Richmond Terrace Mews, behind the building, was built over and joined to the rear of Richmond Terrace to form a modern government office block to house the headquarters of the Department for Health and Social Security, the main entrance to which was number 79 Whitehall (formerly the entrance to the mews). The ministerial team and key National Health Service officials were based there until November 2017.
In January 2018, the House of Commons voted to move to Richmond House in 2025, for an estimated six years, to allow a full renovation of the Palace of Westminster, which accommodates the Houses of Commons and Lords. Plans revealed in October 2018 indicated that most of the 1980s structure would be demolished in preparation for this move, with only the facade retained in front of a new building designed by Allford Hall Monaghan Morris, containing a permanent chamber and offices.