The son of a clergyman, Gregory was born in Ireland but moved to Liverpool upon his father's death in 1766. He studied at the University of Edinburgh and was ordained into the Anglican Church in 1776. He served as a curate in Liverpool while also contributing to periodicals and magazines. He moved to London in 1782 where he was curate at St. Giles, Cripplegate. He went on to carry out the curacy and lectureship of St. Botolph's, the lectureship of St. Luke's, one of the weekly lectureships of St Antholin's, and a small prebend in St. Paul's, which he relinquished for the rectory of Stapleford in Hertfordshire. He occasionally preached at the Foundling Hospital.
He died suddenly after a brief illness on 12 March 1808. At the time of his death he held the positions of the Domestic Chaplain to the Bishop of Landaff, Prebendary of St. Paul's, Vicar of West Ham, and Lecturer at St. Giles.
Gregory is best known for his literary compilations and writings. In 1782-1783, he helped compile a book of excerpts from other authors called Beauties which sold well. His first original lengthy work, in 1785, was Essays, Historical and Moral and was published anonymously. The success of this work led to a second edition in 1788 in which he revealed his identity as the author of the first edition. In 1787, he published a volume of sermons. He is often cited for his translation of Bishop Robert Lowth's Lectures on the Sacred Poetry of the Hebrews, also published in 1787. He published numerous other scholarly and scientific works under his name and contributed to other writings and compilations but without credit.