The fort was built between 1808 and 1812 to prevent invaders gaining access from Maidstone Road to the River Medway. The work was composed of a long brick revetted dry ditch running between a fortified guardroom on the Rochester-Maidstone Road to a similar tower alongside the Medway. The principal work (still surviving) is a massive red brick keep, in the style of a medieval castle, which served as gun tower and observation post. In the sides of the tower were embrasures to sweep the ditch with fire. The dry ditch running across St Margaret's Street was crossed by drawbridge through a substantial casemated guardhouse in the form of an arch (which was demolished in the 1930s). From the tower ran a series of tunnels to the outlying guardhouses. Behind the dry ditch running from the tower down to Maidstone Road was a range of domestic building and barracks.
After 1815, the fort served a variety of purposes, including military prison and lunatic asylum. After nearby Fort Pitt became a military hospital, the patients were moved from Clarence to a new asylum, although the prison remained, with accounts of floggings being given in local newspapers. The fort was used by the garrison artillery throughout the First World War as a recruiting centre. After the war, a large Territorial and Volunteer Reserve centre was built alongside the site and the main barrack site run down. During the Second World War, the Home Guard used Fort Clarence as headquarters and, with the invasion scare, the fort was pressed into service again.
After the war, the fort became derelict, then in the mid 1960s the GPO (now British Telecom) moved in, demolishing all the barracks, filling in a substantial part of the moat and demolishing the Maidstone Road guardhouse.
The most substantial remains now are the brick gun tower and section of ditch from St Margaret's Street into the public gardens opposite. Below the gardens -- donated to the city of Rochester by former mayor Charles Willis in memory of a son killed in the First World War -- is a sally port with sealed-up door. This connected with tunnel that led to the gun tower and probably to the Medway tower, which is long demolished. Intruders into the tunnel system would have been greeted by fire from loopholes built into the entrance tunnels walls.
The tower was transferred by English Heritage to private developers in the late 1990s and has now been converted to apartments. From the outside, the tower has been restored to pristine condition although there is an obvious contemporary structure added on the roof. On both sides the lowest pair of embrasures appear to be below ground level which indicates that originally the ditch was much deeper and more formidable than it now appears. On the east site, the brow of the hill has also been flattened to allow houses to be built and therefore the strategic dominance of the location harder to visualise. The site is a Scheduled Ancient Monument.