Brown was born in Dallas, Texas. He joined McShann's orchestra, which also included the saxophonist Charlie Parker, in 1941. Brown sang on some of the band's most successful recordings, including "Confessin' the Blues" and "Hootie Blues", before leaving to be replaced by Jimmy Witherspoon. in 1947 he recorded some sides with the Tiny Grimes Sextet, which resulted in their version of the hit "Open the Door Richard". The record was considered too risque and was banned from most radio playlists, and the label withdrew it from sale soon after.
Brown's subsequent solo singing career was unsuccessful, although he recorded for the King, Signature and Mercury labels, and he briefly reunited with McShann for recording sessions in 1949. His last two recordings were completed in Houston in 1951 and released on the Peacock label.