He was a student of Judah bar Ezekiel and was Abaye's teacher, and a scholarly disputant (bar plugata) of Rabbah. When his teacher Judah died, Yosef was expected to take Judah's place as head of the Pumbedita yeshiva, due to his excellent knowledge of rabbinic law (as opposed to Rabbah, who excelled in analysis rather than knowledge). Yosef, however, refused to take the position. Rabbah took it instead, at the age of 18, and held it until his death at the age of 40. At this point Yosef agreed to become head of the yeshiva. He held this position for two years, until his own death.
Despite being blind, Yosef managed to accumulate an exceptional knowledge of both written and oral Torah. When some of the canonical Biblical translations were forgotten, he managed to restore them from his memory. However, at one point he contracted a disease which caused him to forget his studies. With the help of his main student, Abaye, he was then able to reconstruct his knowledge.
He is recorded as speaking, paradoxically, of his own humility: "[A tanna recited:] Since Rabbi died, there is no more humility or fear of sin [in the world]. Rav Yosef said to him: Do not teach this regarding humility, for there is still me."