Kholmyansky became interested in Jewish revival and, above all, in learning Hebrew. Gradually, he became a prominent Hebrew teacher in the late 1970s and 1980s in the USSR, heading a clandestine network of Hebrew teachers in different cities across the USSR. At the peak of its activities, the network was active in 20 cities of USSR, with contacts in a further 30 cities. The KGB began close surveillance of Kholmyansky and arrested him in Tallinn, capital of the Estonian Soviet Socialist Republic on 29 August 1984.
While Kholmyansky was in preliminary detention, the KGB conducted a house search of his Moscow apartment, where they claimed to have recovered a shotgun, but his supporters among Soviet Jews and campaigners claimed that he was framed and that the charges were false. During the subsequent criminal investigation, however, the accusation against him of illegal possession of a weapon was dropped, leaving only the charge of illegal possession of ammunition.
The defendant proclaimed an unlimited hunger strike, which he also continued after he was sentenced. When Kholmyansky was hospitalized in the prison hospital, his weight was 42.5 kg (he is 177 cm tall). There, he was force-fed, spent a lot of time in the special punishment cell. It is widely believed that it was due to Kholmyansky's staunch struggle that he received a relatively short term of imprisonment - 1 and a half years.
Some time after completing his term of imprisonment, Kholmyansky was granted an exit visa to Israel, where he arrived with his family. US Secretary of State George P. Shultz, three US Senators (including Ted Kennedy), ten US Congressmen, Margaret Thatcher, Bob Hawke and other public figures took part in the struggle to free Kholmyansky. His case may be seen as a valuable historical document about the struggle for human rights in the USSR in the period of the Brezhnev stagnation.