Popular enthusiasm for the British Empire – whether contemporary or historical – is notoriously difficult to measure or evaluate. Yet the degree to which Empire captured the imagination of the British people in the early twentieth century, the high water mark of imperialism, remains a matter of disagreement. A definitive verdict on British attitudes to Empire is even more elusive when it comes to the following period, when Empire was formally dissolved and its remnants, legacy and memory played, at best, a marginal role in British political life.

It is more challenging still to incorporate British attitudes into an argument about the future survival of today’s Commonwealth, an organisation which, in its more radical incarnation, tried to define itself in terms of virtually everything the British Empire was not. Perhaps the key reason for this complexity was that, for the vast majority of the British population, Empire was not a “lived” experience; at least, not as it was for their compatriots who crossed the seas to live and work in it. As such, for those growing up in the “decolonizing” Britain of the post-war decades, the surviving meaning, if any, of the Empire/Commonwealth continued to be heavily determined by factors such as region, class and even religion.

During my doctoral research into the Conservative Party’s attitudes to decolonisation in Africa, I became familiar with the idea of the Commonwealth as a great, soothing comfort blanket for the party’s dwindling band of post-war imperial enthusiasts.

They could reassure themselves that the sad business of granting independence to British colonies wasn’t really the end of the line. Like the souls of the faithful departed, these countries would simply join the heavenly throng of the Commonwealth and live in eternal peace and harmony. But mine were essentially the findings of an archive rat poring over files from the 1950s and ’60s. My personal experience of the Commonwealth ­– as something actually “out there” in the wider world – didn’t really expand by one iota.

But at a deeper level of understanding and personal identification, there is no substitute for direct contact. I was powerfully reminded of this a few years ago when my colleague at the ICwS, the distinguished Indian politics expert, Professor James Manor, learned to his horror that I had never visited the subcontinent, and thereafter made it his business to “educate” me. And educate me he certainly did. He invited me to a conference he was organising at the Indian Institute of Advanced Studies in Shimla, the old summer capital of the Raj. On the way, I got my first taste of Delhi.

Of course, I knew the basic story. In 1911, the British announced that they were going to relocate the main administrative capital of the Indian Empire there, and commissioned the architects Edward Lutyens and Herbert Baker to draw up suitably grandiose plans. But nothing I’d read had prepared me for the sheer, dizzying folly of the enterprise. My taxi excursion took me past India Gate and down Rajpath, Lutyens’ steroid-fuelled answer to the Mall, and between the two vast Secretariat buildings. We stopped by a set of gigantic gates. Directly ahead, shimmering in the distance, was the 340-room Rashtrapati Bhavan, now the official residence of the Indian president, but originally the Viceroy’s Palace. At every stage of this unfolding spectacle, my only thought was, “Holy shit! What is this?”

In the grey, battle-scarred era between the world wars, the age of Lowry’s hunched, shadow-like Lancashire factory workers, the British had constructed a capital city 4,000 miles from home, on a scale Albert Speer would probably have found intolerably extravagant and bombastic. And only sixteen years after “New Delhi” was formally inaugurated in 1931, the British were gone, sent packing back to Lowry-land. “And what should they know of England, who only England know?” asked Rudyard Kipling rhetorically. Well, quite; but the differences between England and its Empire were thanks in no small part to the crazy, megalomaniac schemes of some of Kipling’s closest friends and allies. Empire, in its wide variety of manifestations, was too remote from the daily experience of most people in the UK to be easily imaginable, let alone an object of nostalgia once it was gone.

So the blanket term “imperial nostalgia” risks missing an important distinction between those people in Britain who had close personal or family associations with the now Commonwealth countries, and those who did not.

The former group has always tended to be drawn disproportionately from the upper levels of British society. It is among those individuals that one senses a genuine, personal nostalgia for Empire, and in many cases that feeling of identification translated itself into an enthusiasm for the Commonwealth. That is not to say that all Commonwealth enthusiasts share this sense. Many of those most closely engaged in the activities of London’s tight network of Commonwealth groups identify instead with the more radical trajectory on which the organisation seemed set in the 1960s and ’70s. Then, the Commonwealth worked closely with Britain’s former colonies to deliver their ambitions for a bright, post-independence future.

Yet what such individuals did share with the “imperial nostalgics” was direct experience of the member states. As a rule, if someone effortlessly drops references to the Commonwealth into their everyday conversation, the chances are they’re either the head of a Commonwealth “civil society” organisation or a member of the British royal family. The correlation between social class and Commonwealth awareness only seems to break down when we come to the UK’s various diaspora communities, many of which retain close links to particular Commonwealth countries. For the rest of the British population, “Empire” and “Commonwealth” were what academics sometimes call “floating signifiers”, untethered to personal experience and so changing over time in line with broader ideological undercurrents.

By the time I was growing up in the ’70s and ’80s, the collective memory of Empire was heavily infused with a sense of irony, not to say absurdity. I was reminded of this when I told my friends that I’d been appointed director of the Institute of Commonwealth Studies. The comic references came thick and fast: “Carry on up the Khyber”; “It Ain’t Half Hot, Mum”; “Roger of the Raj”; and, of course, the mandatory “Will you be having tea with the Queen?” As Karl Marx definitely never said, history on the telly always repeats itself: first as drama and second as parody. “Carry on up the Khyber” seemed to be on constant repeat during the BBC2 early evening slot in the 1970s, to the extent that by the time we reached puberty we could recite the script almost word-for-word.

In the aftermath of the rapid liquidation of colonial rule, the British people were waking up from what seemed like a bizarre, psychedelic dream.

And what stuck in the collective memory was the kitschy, camp surrealism of those imperial dreamscapes. The fantastic uniforms of the viceroy and his staff were recycled by the trendy boutiques of Carnaby Street and sold on to London’s beautiful people with an added sheen of playful irony. The Kinks devoted a whole concept album to the theme of Arthur (or the Decline and Fall of the British Empire). And when Sergeant Pepper first taught the band to play, it was probably in the foothills of the Himalayas, to a restless audience of imperial functionaries retreating from the summer heat of Delhi.

The popular perception of the Commonwealth was also changing, in a far darker and less playfully ironical sense. From the mid-1960s onwards, Commonwealth gatherings became the scenes of pitched battles between Britain and many of its newly independent former colonies over the latter’s determination to end white minority rule in Southern Rhodesia and South Africa. For much of the Conservative Party and its mouthpieces in Fleet Street, the old assumption that the Commonwealth would be some sort of ersatz Empire was neatly and swiftly inverted. Its public enemy number one embodied the true spirit of Empire, and the Commonwealth itself had become the real threat to British interests – both in Southern Africa, where it seemed hellbent on replacing stable, pro-Western regimes with Marxist dictatorships, and at home, where it became synonymous with non-white immigration.

As East African Asians were expelled first from Kenya in 1968 and from Uganda in 1972, the rightwing press in the UK vilified both victims and perpetrators, and governments from both major British parties vied with each other successively to remove the last remaining rights of Commonwealth citizens to enter the UK. Those who had already settled in the country found themselves viciously mocked by popular comedians on primetime national television, a phenomenon as redolent of British culture in the 1970s as David Bowie or the chopper bike.

All this time, the Commonwealth itself was really getting going as a sensible and serious operation.

A Commonwealth Secretariat, fully independent of the British government, was established in 1965. Six years later, the first Commonwealth Heads of Government Meeting in Singapore issued a ringing condemnation of racial discrimination – something as inextricably linked to British imperialism as ice cream and jelly. In general, it’s difficult to arouse passions about something as painfully well-meaning as the modern Commonwealth. Against the far-from-well-meaning British Empire, the contrast is as stark as the architectural one between inter-war Delhi and post-war Coventry. The projection of a stable image of the Commonwealth to the British public has partly been hindered by the fact that Britain’s rulers have so frequently been unsure precisely what to do with it.

Some Commonwealth enthusiasts describe the organisation as the ultimate “soft power” instrument for the UK. I remain unconvinced. It is far from obvious how the Commonwealth, composed of countries which display widely varying attitudes towards things like religious and press freedom, LGBT rights and the death penalty, can serve as an effective instrument of British soft power. And when the organisation, in its wisdom, allows a leader with a seriously tarnished international reputation to become its chair-in-office, as it did in 2013, it’s unlikely that the British government will be able to bask in much reflected glory. In any case, it doesn’t help that the Commonwealth is seen by many as merely a relic of British imperialism. As such, it could hinder just as well as help the UK’s self-projection on the world stage as a modern, liberal, humane society.

Excerpted with permission from The Empire’s New Clothes: The Myth of the Commonwealth, Philip Murphy, Hurst.