Bertie was born in Gainsborough, Lincolnshire, the second eldest son of Willoughby Bertie, 3rd Earl of Abingdon and Anna Maria Collins. On 29 January 1759, he matriculated at Magdalen College, Oxford and received his MA on 29 May 1761.
Bertie was a music patron and composer, as well as a political writer. His brother-in-law Giovanni Gallini brought him into contact with J.C. Bach and Carl Friedrich Abel, and he was subsequently very involved in their careers. During his time in England (1791-1792, 1794-1795), Abingdon was a patron of Haydn's, who may have encouraged him to compose. Abingdon is credited with the composition of one hundred and twenty musical works.
He and his family lived at Rycote in Oxfordshire and in 1769 he funded the construction of the Swinford Toll Bridge across the River Thames near Eynsham. Bertie bred the famous racehorse Potoooooooo, considered one of the finest racehorses of the 18th century. The unusual name came from instructions to his stable boy to write the horse's intended name, Potato, on its feed bin, and the boy instead wrote out "Pot" with eight o's. The joke greatly amused Bertie and he decided to make the name official.
Abingdon earned himself the reputation of a political maverick. His obituary in the Gentleman's Magazine remarked that "his frequent speeches in the House of Peers were peculiarly eccentric". An outspoken critic of Lord North and his administration, he rigorously defended the liberties of the American colonies, yet denounced the French Revolution as a threat to "the Peace, the Order, the Subordination, the Happiness of the whole habitable Globe." He argued that the movement for the abolition of the slave trade was simply the result of a "new philosophy" inspired by the new French republic.
When his elder brother James died in a fire at Rycote in 1745, Bertie became his father's heir, succeeding him as 4th Earl of Abingdon on 10 June 1760. In 1761, he sold the manor of West Lavington, Wiltshire to Robert Palmer and Thomas Walker, and in 1762, he sold the manor of Frilsham, Berkshire to George Amyand.
He married Charlotte Warren, daughter of Admiral Peter Warren, on 7 July 1768. They had seven children:
Abingdon was plagued by financial problems from the moment he inherited the earldom. With his own extravagant lifestyle doing little to alleviate his problems, he died insolvent in 1799. Much of his estate at Westbury, Wiltshire, was sold off over a period from 1777 until his death. The manor of Weston-on-the-Green in Oxfordshire, which he inherited from his brother Captain Peregrine Bertie in 1790, was entailed by Peregrine as a secundogeniture and passed to his younger sons, eventually becoming the property of the Rev. Frederic Bertie.
Abingdon Square Park in Manhattan is named after him.