Is it reasonable to hope for a better world? Study the cruelty and indifference of governments, the disarray of opposition parties, the apparently inexorable slide towards climate breakdown, the renewed threat of nuclear war, and the answer appears to be no. Our problems look intractable, our leaders dangerous, while voters are cowed and baffled. Despair looks like the only rational response.

But over the past two years, I have been struck by four observations. What they reveal is that political failure is, in essence, a failure of imagination. They suggest to me that it is despair, not hope, that is irrational. I believe they light a path towards a better world.

The first observation is the least original. It is the realisation that it is not strong leaders or parties that dominate politics as much as powerful political narratives. The political history of the second half of the 20th Century could be summarised as the conflict between its two great narratives: the stories told by Keynesian social democracy and neoliberalism.

First one and then the other captured the minds of people across the political spectrum. When the social democracy story dominated, even the Conservatives and Republicans adopted key elements of the programme. When neoliberalism took its place, political parties everywhere, regardless of their colour, fell under its spell. These stories overrode everything: personality, identity and party history.

This should not surprise us. Stories are the means by which we navigate the world. They allow us to interpret its complex and contradictory signals. We all possess a narrative instinct: an innate disposition to listen for an account of who we are and where we stand.

When we encounter a complex issue and try to understand it, what we look for is not consistent and reliable facts but a consistent and comprehensible story. When we ask ourselves whether something “makes sense”, the “sense” we seek is not rationality, as scientists and philosophers perceive it, but narrative fidelity. Does what we are hearing reflect the way we expect humans and the world to behave? Does it hang together? Does it progress as stories should progress?

A string of facts, however well-attested, will not correct or dislodge a powerful story. The only response it is likely to provoke is indignation: people often angrily deny facts that clash with the narrative “truth” established in their minds. The only thing that can displace a story is a story. Those who tell the stories run the world.

I came to the second, more interesting, observation with the help of the writer and organiser George Marshall. It is this. Although the stories told by social democracy and neoliberalism are starkly opposed to each other, they have the same narrative structure. We could call it the Restoration Story. It goes like this:

Disorder afflicts the land, caused by powerful and nefarious forces working against the interests of humanity. The hero – who might be one person or a group of people – revolts against this disorder, fights the nefarious forces, overcomes them despite great odds and restores order.

Stories that follow this pattern can be so powerful that they sweep all before them: even our fundamental values. For example, two of the world’s best-loved and most abiding narratives – Lord of the Rings and the Narnia series – invoke values that were familiar in the Middle Ages but are generally considered repulsive today. Disorder in these stories is characterised by the usurpation of rightful kings or their rightful heirs; justice and order rely on their restoration. We find ourselves cheering the resumption of autocracy, the destruction of industry and even, in the case of Narnia, the triumph of divine right over secular power.

If these stories reflected the values most people profess – democracy, independence, industrial “progress” – the rebels would be the heroes and the hereditary rulers the villains. We overlook the conflict with our own priorities because the stories resonate so powerfully with the narrative structure for which our minds are prepared. Facts, evidence, values, beliefs: stories conquer all.

The social democratic story explains that the world fell into disorder – characterised by the Great Depression – because of the self-seeking behaviour of an unrestrained elite. The elite’s capture of both the world’s wealth and the political system resulted in the impoverishment and insecurity of working people. By uniting to defend their common interests, the world’s people could throw down the power of this elite, strip it of its ill-gotten gains and pool the resulting wealth for the good of all.

Order and security would be restored in the form of a protective, paternalistic state, investing in public projects for the public good, generating the wealth that would guarantee a prosperous future for everyone. The ordinary people of the land – the heroes of the story – would triumph over those who had oppressed them.

The neoliberal story explains that the world fell into disorder as a result of the collectivising tendencies of the over-mighty state, exemplified by the monstrosities of Stalinism and Nazism, but evident in all forms of state planning and all attempts to engineer social outcomes. Collectivism crushes freedom, individualism and opportunity. Heroic entrepreneurs, mobilising the redeeming power of the market, would fight this enforced conformity, freeing society from the enslavement of the state.

Order would be restored in the form of free markets, delivering wealth and opportunity, guaranteeing a prosperous future for everyone. The ordinary people of the land, released by the heroes of the story (the freedom-seeking entrepreneurs) would triumph over those who had oppressed them.

Then – again with Marshall’s help – I stumbled into the third observation: the narrative structure of the Restoration Story is a common element in most successful political transformations, including many religious revolutions. This led inexorably to the fourth insight: the reason why, despite its multiple and manifest failures, we appear to be stuck with neoliberalism is that we have failed to produce a new narrative with which to replace it.

You cannot take away someone’s story without giving them a new one. It is not enough to challenge an old narrative, however outdated and discredited it may be. Change happens only when you replace it with another. When we develop the right story, and learn how to tell it, it will infect the minds of people across the political spectrum.

***

But the best on offer from major political parties is a microwaved version of the remnants of Keynesian social democracy. There are several problems with this approach. The first is that this old story has lost most of its content and narrative force. What we now call Keynesianism has been reduced to two thin chapters: lowering interest rates when economies are sluggish and using counter-cyclical public spending (injecting public money into the economy when unemployment is high or recession threatens).

Other measures, such as raising taxes when an economy grows quickly, to dampen the boom-bust cycle; the fixed exchange rate system; capital controls and a self-balancing global banking system (an International Clearing Union) – all of which John Maynard Keynes saw as essential complements to these policies – have been discarded and forgotten.

This is partly because the troubles that beset the Keynesian model in the 1970s have not disappeared. While the oil embargo in 1973 was the immediate trigger for the lethal combination of high inflation and high unemployment (‘stagflation’) that Keynesian policies were almost powerless to counteract, problems with the system had been mounting for years. Falling productivity and rising cost-push inflation (wages and prices pursuing each other upwards) were already beginning to erode support for Keynesian economics. Most importantly, perhaps, the programme had buckled in response to the political demands of capital.

Strong financial regulations and controls on the movement of money began to weaken in the 1950s, as governments started to liberalise financial markets. Richard Nixon’s decision in 1971 to suspend the convertibility of dollars into gold destroyed the system of fixed exchange rates on which much of the success of Keynes’s policies depended. The capital controls used to prevent financiers and speculators from sucking money out of balanced, Keynesian economies collapsed. We cannot hope that the strategies deployed by global finance in the 20th Century will be unlearnt.

But perhaps the biggest problem residual Keynesianism confronts is that, when it does work, it collides headfirst with the environmental crisis. A programme that seeks to sustain employment through constant economic growth, driven by consumer demand, seems destined to exacerbate our greatest predicament.

Without a new, guiding story of their own, allowing them to look towards a better future rather than a better past, it was inevitable that parties which once sought to resist the power of the wealthy elite would lose their sense of direction. Political renewal depends on a new political story. Without a new story, that is positive and propositional, rather than reactive and oppositional, nothing changes. With such a story, everything changes.

The narrative we build has to be simple and intelligible. If it is to transform our politics, it should appeal to as many people as possible, crossing traditional political lines. It should resonate with deep needs and desires. It should explain the mess we are in and means by which we might escape it. And, because there is nothing to be gained from spreading falsehoods, it must be firmly grounded in reality.

This might sound like a tall order. But there is, I believe, a clear and compelling Restoration Story to be told that fits this description.

***

Over the past few years, there has been a convergence of findings in different sciences: psychology, anthropology, neuroscience and evolutionary biology. Research in all these fields points to the same conclusion: that human beings are, in the words of an article in the journal Frontiers in Psychology, “spectacularly unusual when compared to other animals”. This refers to our astonishing degree of altruism. We possess an unparalleled sensitivity to the needs of others, a unique level of concern about their welfare, and a peerless ability to create moral norms that generalise and enforce these tendencies.

We are also, among mammals, the supreme cooperators. We survived the rigours of the African savannahs, despite being weaker and slower than our predators and most of our prey, through developing a remarkable capacity for mutual aid. This urge to cooperate has been hard-wired into our brains through natural selection. Our tendencies towards altruism and cooperation are the central, crucial facts about humankind. But something has gone horribly wrong.

Our good nature has been thwarted by several forces, but perhaps the most powerful is the dominant political narrative of our times. We have been induced by politicians, economists and journalists to accept a vicious ideology of extreme competition and individualism, that pits us against each other, encourages us to fear and mistrust each other, and weakens the social bonds that make our lives worth living.

The story of our competitive, self-maximising nature has been told so often and with such persuasive power that we have accepted it as an account of who we really are. It has changed our perception of ourselves. Our perceptions, in turn, change the way we behave.

With the help of this ideology, and the neoliberal narrative used to project it, we have lost our common purpose. This leads in turn to a loss of belief in ourselves as a force for change, frustrating our potential to do what humans do best: to find common ground in confronting our predicaments, and to unite to overcome them. Our atomisation has allowed intolerant and violent forces to fill the political vacuum. We are trapped in a vicious circle of alienation and reaction. The hypersocial mammal is falling apart.

But by coming together to revive community life we, the heroes of this story, can break the vicious circle. Through invoking our capacity for togetherness and belonging, we can rediscover the central facts of our humanity: our altruism and mutual aid. By reviving community, built around the places in which we live, and by anchoring ourselves, our politics and parts of our economy in the life of this community, we can restore the best aspects of our nature.

Where there is atomisation, we will create a thriving civic life. Where there is alienation, we will forge a new sense of belonging: to neighbours, neighbourhood and society. Community projects will proliferate into a vibrant participatory culture. New social enterprises will strengthen our sense of attachment and ownership.

Where we find ourselves crushed between market and state, we will develop a new economics, that treats both people and planet with respect. We will build it around a great, neglected economic sphere: the commons. Local resources will be owned and managed by communities, ensuring that wealth is widely shared. Using common riches to fund universal benefits will supplement state provision, granting everyone security and resilience.

Where we are ignored and exploited, we will revive democracy and retrieve politics from those who have captured it. New methods and rules for elections will ensure that every vote counts and financial power can never vanquish political power. Representative democracy will be reinforced by participatory democracy, that allows us to refine our political choices. Decision-making will be returned to the smallest political units that can discharge it.

The strong, embedded cultures we develop will be robust enough to accommodate social diversity of all kinds: a diversity of people, of origins, of life experiences, of ideas and ways of living. We will no longer need to fear people who differ from ourselves; we will have the strength and confidence to reject attempts to channel hatred towards them.

Through restoring community, renewing civic life and claiming our place in the world, we build a society in which our extraordinary nature – our altruism, empathy and deep connection – is released. A kinder world stimulates and normalises our kinder values. I propose a name for this story: the Politics of Belonging.

***

Some of this can begin without waiting for a change of government: one of the virtues of a politics rooted in community is that you do not need a national movement in order to begin. But other aspects of this programme depend on wider political change. This too might sound like an improbable hope – until you begin to explore some of the remarkable things that have been happening in the United States.

The Big Organising model developed by the campaign to elect Bernie Sanders as the Democratic nominee is potentially transformative. Rather than relying on big spending, big data and a big staff, it uses proliferating networks of volunteers, who train and supervise more volunteers, to carry out the tasks usually reserved for staff. While Hillary Clinton’s campaign was organising money, the Sanders campaign was organising people. By the end of the nomination process, more than 100,000 people had been recruited. Between them, they ran 100,000 events and spoke to 75 million voters.

His bid for the nomination was a giant live experiment, most of whose methods were developed on the job. Those who ran it report that by the time they stumbled across the strategy that almost won, it was too late. Had it been activated a few months earlier, the volunteer network could have abandoned all forms of targeting and contacted almost every adult in the USA. If the techniques they developed were used from the outset, they could radically alter the prospects of any campaign for a better world.

When, after reading a book by two of Sanders’s organisers, I argued in a video for the Guardian that this method could be used to transform the prospects of Jeremy Corbyn’s Labour party, I was widely mocked. But it turned out to be true. By adopting elements of the Sanders strategy, Labour, supported by Momentum, almost won an election that was widely predicted to be a Conservative landslide. And the method that propelled this shift is still in its infancy.

I believe it could become still more powerful when combined with some of the techniques identified by former Congressional staffers in the Indivisible guide to influencing Members of Congress. These people studied the methods developed by the Tea Party movement and extracted the crucial lessons. They discovered that the key is to use local meetings with representatives to press home a single demand, film and share their responses on social media, then steadily escalate the pressure.

The Tea Party honed this technique until its requests became almost impossible to resist. The same thing can be done, though without the harassment to which that movement sometimes resorted. Supported by the Big Organising model, using its proliferating phone-bank teams and doorstep canvassing, the Indivisible methods could, I believe, be used to flip political outcomes in any nation that claims to be a democracy.

But none of this will generate meaningful and lasting change unless it is used to support a new, coherent political narrative.

Those who want a kinder politics know we have, in theory at least, the numbers on our side. Most people are socially-minded, empathetic and altruistic. Most people would prefer to live in a world in which everyone is treated with respect and decency, and in which we do not squander either our own lives or the natural gifts on which we and the rest of the living world depend. But a small handful, using lies and distractions and confusion, stifle this latent desire for change.

We know that, if we can mobilise such silent majorities, there is nothing this small minority can do to stop us. But because we have failed to understand what is possible, and above all failed to replace our tired political stories with a new, compelling narrative of transformation and restoration, we have failed to realise this potential. As we rekindle our imagination, we discover our power to act. And that is the point at which we become unstoppable.

George Monbiot’s book Out Of The Wreckage: a new politics for an age of crisis, is published by Verso.