If everything is getting better, why doesn't if feel that way?

The right sets great store by selling us optimism. But in this bleak climate what people really need is hope

As the last turkey leftovers are served up as sandwiches or madras, it has long been traditional to audit the year that is ending and to survey that which is about to begin. In recent times, the right has seized such calendrical opportunities to remind us that, contrary to the jeremiads we are routinely fed by the media, things are, as the Beatles sang, getting better all the time.

And not just the Beatles. As the Tory MEP Daniel Hannan wrote in a ConservativeHome post on Christmas Eve: "Whether we measure literacy or longevity, infant mortality or sexual equality, the world in 2015 was a better place than in 2014; and the world in 2016 will be better still." All Hannan asked of his readers was to follow his empirical tour d'horizon: globally, according to UN statistics, the number of people dying violently has fallen by 6% since 2000; there has been a drop of eight percentage points in the number of people suffering from malnutrition since 1990; satellite imagery suggests that green spaces have grown by 14% over the past three decades; and so on.

Like everything Hannan writes, the lucidity of the post was compelling and made well a claim that other Tories have advanced with less poise and eloquence. But the core arguments is invariably the same - things are improving, thanks to the operations of the market, technology and digital culture. Though this insistent school of optimism has many roots, it acknowledges two key texts. First, Matt Ridley's The Rational Optimist: How Prosperity Evolves(2010). a beautifully written account of the interaction between trade and progress in human history. And second, Steven Pinker's The Better Angels of Our Nature: The Decline of Violence in History and its Causes (2011), which amasses a huge stock of data to chart and explain the diminishing resort to violence. The author attributes this to a range of social imperatives including the "intensifying application of knowledge and rationality to human affairs".

Why is this school of thought especially popular on the libertarian right? Because it recasts the victory of the west in the cold war and the consequent triumph of globalization as a historical moment that has had a glorious, if unacknowledged, dividend for humankind. It has become fashionable to say, with Le Carre's Smiley, that the right side lost the cold war but the wrong side won.

The Ridley-Pinker optimists argue, with some force, that the benign impact of globalization has not received the recognition it is due and that the argument has been contaminated by the left's sour grapes. In a pointed example of historical cross-dressing, the right has inherited the Enlightenment tradition of optimism: the belief that reason will invariably conquer its enemies, and that history therefore has an underlying direction as greater prosperity gradually reduces crime, indigence and ill health. Implicit in this position is an impatience with the left for refusing to acknowledge the reams of statistics deployed by the right and the role of trade in making the world a safer, healthier, less violent place.

Yet that is not the end of the matter; patently not. If everything is getting better, why doesn't it feel that way? To survey the prominent movements of our time is a bleak experience indeed: fury at "the 1%" finds expression all over the world, as does a more general sense of injustice at entrenched inequality between and within nations. Contempt for politicians is now matched by distrust of the professions. Religious violence and millenarian bigotry are on the rise, a domestic threat and a geopolitical force: on Boxing Day, Abu Bakr al-Baghdadi, the Islamic State leader, made his first public statement to warn "crusaders and Jews" not to attack him on his own terrain. Meanwhile, in the US, still the world's only hyper-power, the front runner in the Republican presidential primaries is a shameless hate-monger who exacerbates every fear that lurks in the depths of the American psyche, and, incredibly, proposes to keep Muslims out of the country he aspires to govern.

How to reconcile the feelgood statistics paraded by the new optimists with this bleak panorama? The media may play a part in stocking doubt and fear, but it is lazy to pretend that the disparity between global statistics and psychological reality is to be blamed exclusively upon television, web and press.

Part of answer, I think, is that people are not circuit boards governed by rationally constructed algorithms. They are passionate souls, complex and irrational. One of the philosopher Isaiah Berlin's great insights was that optimistic rationalism was inherently limited by human nature, foible and emotion. This was what the Romantic thinkers - Vico, Herder, Schelling - had understood.

Beneath this clash of perspectives lies a crucial distinction - between optimism and hope. Optimism can be tabulated and illustrated on a graph, presented as an objective position. Hope, in contrast, is subjective, empathetic, personal and easily flipped over to become its opposite: fear. As Rabbi Jonathan Sacks has written, "Optimism is a passive virtue, hope an active one. It takes no courage to be an optimist, but it takes a great deal of courage to have hope."

How can you tell a lone parent at a food bank, looking for viable work in an area where Sure Start centers suitable for her child are closing, that the world is getting better? How dare we even suggest that a widow whose operation has just been cancelled stop complaining and recognize global health trends? How can one take seriously the financial sector's claims to have learned its lesson when even the bailed-out banks are paying out shares and staggered bonuses to evade the government's ceiling of 2,000 pounds per employee? In such circumstances, politicians have no right to insist upon optimism. But part of statesmanship is the ability to urge citizens to experience the much more complex emotion of hope.

The political class has lost this talent, at the worst possible moment. For the year ahead will be one in which hope goes into battle with fear; when decency squares off against bigotry. Regrettably, but inevitabley, the furore over migration policy will become almost indistinguishable from the struggle over Britain's membership of the EU. The "in" campaign will be obliged to address popular fear, without appeasing it. There are many great challenges ahead for this country but our future in the EU (or outside it) looms like a new moon in the winter sky. I think the British will see sense and stay in. How do I know? I don't. But I hope so.