Born in a Sephardic Jewish family Vienna; his first language was Ladino. He studied with Adolf Bak and Simon Pullman at the Vienna Conservatory from the age of twelve and graduated in 1928. With his three sisters he founded the Galimir Quartet in 1927 to commemorate the centenary of the death of Ludwig van Beethoven. During the early 1930s Galimir studied with Carl Flesch in Berlin In 1936, the Galimir Quartet recorded the Lyric Suite of Alban Berg and the String Quartet of Maurice Ravel under the supervision of the composers, who were present during the rehearsals and recording sessions. In 1936, he joined the Vienna Philharmonic Orchestra. He was harassed because of his Jewish ethnicity - at one performance, writes Allan Kozinn, "just as the lights went down, the principal clarinetist called out, in a voice audible throughout the theater, 'Galimir - have you eaten your matzos today?'" The next season, the orchestra expelled him because he was Jewish. He then emigrated to Palestine to join the newly founded Palestine Symphony Orchestra.
In 1938, Galimir moved to New York, where he founded another quartet and served as member of the NBC Symphony Orchestra from 1939 to 1956.(later, when the NBC ensemble was disbanded, Galimir was concertmaster of the Symphony of the Air.) In the 1950s he began acquiring a reputation as a music teacher and began teaching at The Juilliard School in New York in 1962 and from 1972 at the Curtis Institute of Music in Philadelphia. In 1976 he began teaching at Mannes College of Music in New York.
In 1952, after the death of Adolf Busch, pianist Rudolf Serkin asked Galimir to join the faculty of the Marlboro Music Festival, where he was in residence every year from 1954 until his death in 1999.