The Chinese government’s ongoing attempts to create a social credit system aimed at rating the trustworthiness of people and companies have generated equal measures of fascination and anxiety around the world. Social credit is depicted as something uniquely Chinese – a nefarious and perverse digital innovation that could only be conceived of and carried out by a regime like the Chinese Communist Party.

The proposed system will draw on data gathered from individuals and businesses to provide social credit scores based on both economic and social behaviour. While government proposals provide little detail about how scores will be calculated, current pilots use both online consumer behaviour and the scores of others in a person’s network.

The stated aim is to “provide the trustworthy with benefits and discipline the untrustworthy … [so that] integrity becomes a widespread social value”. The official documents are light on detail, but have suggested various ways to punish untrustworthy members of society with low scores, such as through restrictions on employment, consumption, travel, and access to credit.

It is seen as a signal of a dystopian future, but one that could only exist in China’s authoritarian context. But China’s plans to “build an environment of trust” – as the Chinese government’s State Council puts it – using the data generated from digital activity is not uniquely Chinese. And the country’s experiments are a natural extension of a global trend where data is being used to control society.

Pilots underway

The impetus to create a social credit system in China came largely from the country’s dearth of a credit rating infrastructure; most people have no credit score. In 2014 the government outlined plans to create a nationwide social credit system by 2020.

Today, the Chinese social credit system is far from unified or centralised. Like most new policies in China, it is being subjected to China’s distinctive policy modelling process. Local governments produce their own interpretations of policies, which then vie to become national models. Currently, over 30 local governments have been piloting social credit systems, using different approaches to arrive at their social credit scores.

In contrast to other policies, however, until recently social credit was also being piloted by eight large Chinese internet companies. The most well-known of these is Alibaba’s Sesame Credit, which uses opaque algorithms to arrive at social credit scores for their customers drawn from data provided by an affiliate company called Ant Financial. Those with high scores – the top score is 950 – have been able to access a range of benefits from other Alibaba businesses and their partners. In mid-2017 the government declined to renew the licenses for the private pilots over conflict of interest concerns. However, the door has been left open for these pilots to be merged into the ongoing attempts to construct a nationwide government-run system.

The expansion of these social credit systems fits neatly into moral and economic narratives that are now prominent in China. These are linked to a percieved “trust deficit” in the country. High profile cases of economic and social fraud regularly go viral on Chinese social media. This has resulted in a popular discourse of the decline of morality and the inability to trust other people or companies.

Social credit is also presented as a way to improve financial inclusion. Supporters argue that expanding financial services and credit to previously excluded groups is good for socioeconomic development. They believe intrusive methods of assessing creditworthiness are justified in order to reduce the risk to lenders. In this way, Chinese social credit mirrors financial inclusion initiatives elsewhere, such as the use of psychometrics and other personal digital data to determine whether someone is eligible for a loan.

Social credit beyond China

Socio-economic credit systems are not confined to China. Most industrialised nations have relied on credit ratings for decades to quantify the financial risks associated with countries, firms, and individuals. But social factors are increasingly being included to make more accurate predictions.

In China, Alibaba’s Sesame Credit factors in credit scores of a debtor’s social network. This means that those with low-score contacts will see a negative impact on their own scores. In the US, Affirm, a San Francisco-based lender headed by PayPal co-founder Max Levchin, combs through a wide range of sources, such as social networks, to evaluate the default risk of a creditor. And Lenddo, a Hong Kong-based company, took an even bolder approach and informed debtors’ friends on Facebook when they didn’t pay instalments in time.

So, credit scores based on social action aren’t just unique to China and authoritarian regimes. However, this doesn’t make them less concerning.

The primary downside of public rating systems like social credit is the far-reaching consequences of low ratings. In China, a recent administrative regulation dictates that defaulting debtors will be listed and shamed on online platforms. Similar measures have taken place in the US. The New York Post, for instance, filed a Freedom of Information request and published the performance ratings and names of teachers evaluated by a “secret algorithm” in an attempt to shame those deemed not to be performing as well.

Rule by algorithms

Experiments repeatedly confirm that data and algorithms are as biased as society is and reproduce real life segmentation and inequality. In her book the Weapons of Math Destruction, American mathematician Cathy O’Neil warned that we need algorithmic audits.

Algorithms that measure social credit and trustworthiness could theoretically be fair – but those in positions of power may well find a way to circumnavigate them. And even though social credit schemes are supposed to extend access to financial resources to previously excluded populations, it’s likely that credit solutions driven by big data will exacerbate existing social divides. In the US, for example, 45m Americans do not have a credit score because they lack a credit history and minority groups and low-income neighbourhoods have a disproportionately high rate of credit invisibility. For China, it’s likely something similar could emerge.

Social credit has the potential to cement and exacerbate existent power imbalances in societies, while simultaneously closing down spaces of resistance. The social credit system in development in China is a phenomenon that belongs to a global trend with a transformative potential that is just as threatening in Western democracies. It is but one harbinger of a likely common digital future marked by a shift away from the rule of law and towards rule by algorithms.

Stefan Brehm, Researcher, Lund University and Nicholas Loubere, Associate Senior Lecturer in the Study of Modern China, Lund University.

This article first appeared on The Conversation.