Born in Rouen, he moved to London in 1568 with his Huguenot parents Peter and Epiphany Oliver to escape the Wars of Religion in France. He then studied miniature painting under Nicholas Hilliard; and developed a naturalistic style, which was largely influenced by Italian and Flemish art. His first wife, Elizabeth, died in 1599. With her he fathered Peter Oliver, who was also eminent in miniature painting. In 1602 he married Sara, daughter of the well-known portrait painter Marcus Gheeraerts the Elder (c. 1520 – c. 1590) and his wife Susannah de Critz. Susannah was the daughter of Troilus de Critz, a goldsmith from Antwerp, and close relative of John de Critz, the Queen's Serjeant-Painter. She was also the eldest sister or cousin of Magdalen de Critz who married Marcus Gheeraerts the Younger (1562–1635).
Richard Sackville, 3rd Earl of Dorset, cabinet miniature, 1616