Gillum was born in Indianola, Mississippi. He ran away from home at age seven and for the next few years lived in Charleston, Mississippi, working and playing for tips on street corners. He moved to Chicago in 1923, where he met the guitarist Big Bill Broonzy. The duo started working at nightclubs around the city. By 1934 Gillum was recording for ARC Records and Bluebird Records.
Gillum's recordings, under his own name and as a sideman, were included on many of the highly popular "Bluebird beat" recordings produced by Lester Melrose in the 1930s and 1940s. In 1940, he was the first to record the blues classic "Key to the Highway" (featuring Broonzy on guitar), utilizing the now-standard melody and eight-bar blues arrangement. (The song had first been recorded a few months earlier by Charlie Segar, with a different melody and a 12-bar blues arrangement.) Gillum's version of the song was covered by Broonzy a few months later, and his version has become the standard arrangement of this now-classic blues song.
Gillum's records were some of the earliest featuring blues with electric guitar acompaniment, when the 16-year-old jazz guitarist George Barnes played on several songs on Gillum's 1938 session that produced "Reefer Headed Woman" and others.
After the Bluebird label folded in the late 1940s, he made few recordings. His last, slightly sad recordings were on a couple of 1961 albums with Memphis Slim and the singer and guitarist Arbee Stidham, for Folkways Records.
On March 29, 1966, Gillum was shot in the head during a street argument and was pronounced dead on arrival at Garfield Park Hospital, in Chicago. He is buried at Restvale Cemetery, in Alsip, Illinois.
Hs daughter, Ardella Williams, is a blues singer in Chicago.