He was born in Bolton. Although he was essentially self-taught as a composer, he studied piano, cello and harmony at the Royal Manchester College of Music, where his teachers were Thomas Keighley, Frank Merrick and Carl Fuchs. Later he won a scholarship to study art and cabinet-making at the Bolton School of Art.
After training as a teacher, he became art master at Tettenhall College, Wolverhampton. Whilst there, as a pacifist, he joined the Peace Pledge Union. In the Second World War, he registered as a conscientious objector, with a condition that he continue teaching. He taught composition at the Royal Manchester College of Music from 1947 to 1973, where his pupils included David Ellis, John Golland, John McCabe, John Ogdon and Ronald Stevenson.
Between 1986 and 1993 he wrote a three volume autobiography. He continued to create art and music until his nineties.
As a composer Pitfield was influenced by Ralph Vaughan Williams, Percy Grainger and Frederick Delius. He was a prolific composer and his compositions include collections of miniatures for students and amateurs, a five-movement Sinfonietta, a Trio for flute, oboe and piano, concertos for piano, violin, recorder and percussion, a Xylophone Sonata, an Oboe Sonata, and solo works for accordion, clarsach, and harmonica. He also invented an instrument called "patterphone" to produce rain-like sounds.
His music was published by more than 50 publishers. Hubert J. Foss of the Oxford University Press published many of his compositions, illustrations, frontispieces and cover-designs, which he made for various publications, including the one for Benjamin Britten's Simple Symphony.