He was brought up in the Shepherd's Bush area and was a fervent Fulham F. C. supporter as a child. His father ran a fish stall on North End Road, Fulham, and in 1942 at the age of 14 when his father went into the army he took over the running of the stall. When Gregory senior came back from the War, Jim set up his own buying and selling business. In the early 1950s he entered the second-hand car business, setting up on a Hammersmith bombsite. By the end of the 1950s he had bought 3 acres (12,000 m2) in Hounslow and set up "Gregory's Motordome" (which could possibly be seen as a forerunner of the car supermarkets of today). Further businesses were acquired and sold. Gregory attempted to buy into his boyhood idols Fulham but his offer was rejected by Fulham chairman Tommy Trinder; he would later try again only to meet with the same answer. Gregory was deeply upset by this rejection and in late 1964 he joined the board of Queens Park Rangers (QPR) instead.
He became chairman a few months later in early 1965. When he took over at Loftus Road, the South Africa Road side of the ground consisted of a muddy bank and crowds were low. Gregory's enthusiasm and money - he would invest millions - soon however brought a great run of success. The manager at the time, Alec Stock, was building a young and dynamic team and one of Gregory's early deals in conjunction with Stock helped to add the final piece in the jigsaw: The signing from Fulham of Rodney Marsh for £15,000 in March 1966.
Over the coming seasons QPR would achieve three promotions and also win the League Cup in 1967. Gregory also modernised the stadium, financing the building initially of two new stands on the South Africa Road (1968) and Ellerslie Road (1972) sides of the ground. In 1987 after advice from doctors to slow down he sold QPR to Marler Estates, a property company run by the then Fulham Chairman David Bulstrode.
However, after leaving QPR in 1987, he very soon became Chairman at Portsmouth remaining until 1995 when ill-health forced him to step down and hand over the reins to his son, Martin. He died three years later.