Journalist, author and trainer Nicolas Kayser-Bril will speak at the “Super-Scoring?” conference on 11 October about “Personal Scoring in the EU: Not quite Black Mirror yet, at least if you‘re rich”. The essay can be downloaded here as a PDF-File.
By Nicolas Kayser-Bril
In October 2016, the British dystopian TV series Black Mirror aired Nosedive. In that episode, citizens use their cell phones to review each other on a 5-star scale after each social interaction and everyone’s average score is visible to everyone else. In such a world, any unpopular behavior or opinion leads to lower reviews and, from there, to social ruin.
This work of fiction has since widely been used to illustrate what a centralized, permanent and public personal scoring system would look like. In the countries of the European Union, such a system is very unlikely to appear for technical and legal reasons. This does not mean that a large share of the European population is not or will not be subject to invasive scoring mechanisms. Reality is just slightly more pernicious than fiction.
The need for national identification numbers
From a technical perspective, ubiquitous and centralized scoring requires a unique and immutable identifier for all citizens, from cradle to grave, called a national identification number (NIN). This allows for fast and reliable operations between the databases that store the information used to compute the score of each citizen.
The potential use of NINs for generalized scoring was clear from the early days of digitization in the public sector. In the 1970s in both France and Germany, civil society erupted over plans that would have made it possible to access personal data in several government databases at once (Steinmüller, 1973, for Germany and Chignard, 2018, for France). As a result, Germany’s plans to introduce a NIN, the Personenkennzeichen, faltered, and the French parliament passed a law that banned any automated processing of personal data[1], effectively preventing citizen scoring. (France, however, does have a NIN since 1945, the social security number).
While some countries did introduce NINs successfully, such as Danemark in 1968 and Estonia in 1992, others followed in Germany’s footsteps. The constitutional court of Hungary, for instance, declared NINs illegal in 1991 (Pouloudi & Kalliamvakou, 2011). Austria generated a NIN in 2006 to make its census more efficient but the authorities involved took great care to scramble it so that it could only be deciphered by the data protection authority (Kronbichler, 2011).
In light of global trends, where many countries – incentivized by vendors and international aid – rush to create NINs (Kayser-Bril, 2019, for an overview of five African countries), many countries of the European Union are exceptional in that their legal frameworks often prohibit them.
Experiments in citizen scoring
While NINs were a precondition for citizen scoring in the 1970s and still make it easier today, advances in database software and computing power make the identification of individuals across databases very possible without them. The United Kingdom, where many examples of citizen scoring can be found, does not have a NIN, for instance.
In several countries, local authorities run experiments in citizen scoring, most commonly to detect welfare fraud or potentially problematic behavior. In the Netherlands, the Systeem Risico Inventarisatie (SyRI) merges data from the tax office, immigration authorities and others to detect welfare fraud. In Trelleborg, Sweden, an algorithm fetches data from several databases, such as the tax agency or the bureau for housing support and decides whether or not applicants can receive social benefits. In Denmark, a point system tries to detect children in vulnerable circumstances. Parameters such as mental illness (3000 points), unemployment (500 points), missing a doctor’s appointment (1000 points) or dentist’s appointment (300 points) contribute to the score of each family in the area where the system is deployed. In France, intelligence services deployed algorithms that detect anomalous behavior from internet users. In Mannheim, Germany, surveillance cameras will alert the police when their algorithm detects certain behaviors from individuals (all examples from AlgorithmWatch, 2019). In Spain, an algorithm decides if tenants are eligible to subsidized electricity prices using income and rent data (Belmonte, 2019).
While not all examples of automated decision-making lead to the production of a publicly available score, they all need to reduce several inputs of personal data to a single number or measure that allows the algorithm to reach a decision. As such, citizen scoring and automated decision-making that involves several sources of personal data are essentially the same thing.
As far as we know, no country of the European Union deploys or is planning to implement a single score for all its citizens. It is unlikely that they ever will. As Antonia Hmaidi (2018), an economist, said of the Chinese social management systems, the very concept of a single score is utterly useless. By merging inputs from all possible sources, the resulting number cannot be used for any purpose. One can have stellar bank statements (used for credit scoring) and terrible parenting skills (used to detect vulnerable children) or vice-versa; merging the two will produce an average score – and a meaningless one, too.
The current experiments in citizen scoring in the European Union, however, do have massive impacts on all persons affected. The obtention, or the amount, of welfare benefits, are life-defining items for the poorest Europeans. Exactly how many citizens are currently affected by scoring is impossible to say, not least because many of the automated decision-making processes are intransparent or secret. Interestingly, a large proportion of the examples that AlgorithmWatch (2019) collected aim at automatically restricting access to social services. A few counterexamples came from tax authorities. The French government announced in 2018 that it would merge several databases to detect tax avoidance (Berne, 2018), though no update was published regarding the implementation of the scheme. The Slovenian tax authorities automatically rank tax-payers according to their risk of tax avoidance, but it is unclear what data is used to this end (AlgorithmWatch, 2019).
Governance through numbers
The current wave of citizen scoring begs the question: why now? It was very clear that the digitization of government data did allow for such practices as early as in the 1970s. It was, technically at least, possible for public authorities to engage in citizen scoring forty years ago in countries that did have a NIN, such as Denmark or France. No major piece of legislation entered in force that would have made citizen scoring easier, so that it would be hard to argue that a change in the legal systems caused these new practices. On the contrary, many of the citizen scoring initiatives, such as the Trelleborg system in Sweden, or a Slovenian system to automatically detect ‘risky’ travellers at the border, are suspected to be illegal (AlgorithmWatch, 2019). Nor can the efficiency of public services be invoked as the cause for the flurry of citizen scoring. In the United Kingdom for instance, conservative estimates show tax avoidance to be 30 times larger than welfare fraud (Ball, 2013). An efficiency-driven government would have targeted areas where the pay-off would have been greatest. Instead, citizen scoring is used largely to target vulnerable populations.
This fits the theory of French legal scholar Alain Supiot, who posits that “governance through numbers” allows for the depoliticization of decision-making. By handing over decisions which are political in nature, such as deciding who is entitled to benefit from national solidarity, to algorithms (which use citizen scores), those holding political power are able to present their policy as neutral and objective – as numbers are supposed to be. The dream of a harmonious society run by the rule of numbers is as old as civilization. It is built in opposition to a society run by the rule of law, where each political decision is debated first by lawmakers, then judges. As such, citizen scoring in Europe is a new development in the long list of “governing machines” that popped up in Western thought at least since the Enlightenment (Supiot, 2015).
Governance through numbers contributes to the “politics of inevitability”, a concept developed by historian Timothy Snyder to describe the process through which political power can be taken away from the public debate. By making political decisions appear without alternatives, the political personnel fosters an intellectual environment without effective political options, thereby ensuring their grip on power (Snyder, 2018). Because the technicalities of automated decision-making, often shrouded in complex terms, are very rarely understood by the political personnel and their entourage of political journalists and think-tanks, they might not be cognizant to the political choices that underlie the machinery of citizen scoring. The drive towards algorithmic-driven politics of inevitability might not be based on a conscious decision.
Opacity by design
The fears of Black Mirror’s Nosedive episode are irrational. On the one hand, an all-encompassing citizen score would be impractical and useless. Because it would aggregate dimensions that are unrelated to each other, and because it would be extremely unlikely for an individual to score evenly across all of them, a comprehensive citizen score would merge information from different areas that would even each other out (to take an extreme example, a jailed criminal might be positively viewed by prison personnel because of her good behavior).
On the other hand, most experiments with citizen scoring nowadays aim at reducing the number of political options that can be publicly debated, for governance through numbers does not allow for political debate.[2] This political project does not require a ubiquitous score for all citizens but only for those with the least power or those who might challenge the political order.
In this light, clearly laid-out rules for scoring, as used in the Danish system that detects vulnerable children, are not beneficial to those who decide on the implementation of citizen scores. On the contrary, the opacity of the scoring process reduces the power of the individuals being scored, who cannot regain agency to control or feel in control of their score. It is not by accident that for most of the examples cited here, the rules of scoring are not published. This is a feature of citizen scoring in the European Union.
References
AlgorithmWatch (2019). Automating Society: Taking Stock of Automated Decision-Making in the EU [report].
Ball, J. (2013). Welfare fraud is a drop in the ocean compared to tax avoidance, The Guardian. Archived at http://archive.is/XOx6o
Belmonte, E. (2019). La aplicación del bono social del Gobierno niega la ayuda a personas que tienen derecho a ella, Civio. Archived at http://archive.is/f7K0c
Berne, X. (2018). Contre la fraude fiscale, le gouvernement veut recourir davantage au «data mining», NextINpact. Archived at http://archive.is/g94s2
Chignard, S. (2018). «Safari, la chasse aux Français», 40 ans après [blog]. Archived at http://archive.is/PloNt
Hmaidi, A. (2018). “The” Social Credit System [presentation]. Accessed on March 31, 2019 at https://media.ccc.de/v/35c3-9904-the_social_credit_system
Kayser-Bril, N. (2019). Identity-management and citizen scoring in Ghana, Rwanda, Tunisia, Uganda, Zimbabwe and China.
Kronbichler, K. (2011). „Von der traditionellen Volkszählung zur Registerzählung.“ GW-Unterricht 2: 123. Archived at http://archive.is/HisuB
Pouloudi, N., & Kalliamvakou, E. (2011). Tracing Diversity in the History of Citizen Identifiers in Europe: a Legacy for Electronic Identity Management? In Emerging Themes in Information Systems and Organization Studies (pp. 333-346). Physica-Verlag HD.
Snyder, T. (2018). The Road to Unfreedom: Russia, Europe, America. Tim Duggan Books.
Steinmüller, W. (1973). Datenschutzrechtliche Anforderungen an die Organisation von Informationszentren. GI-BIFOA Internationale Fachtagung: Informationszentren in Wirtschaft und Verwaltung, 186–205. doi:10.1007/bfb0019346
Supiot, A. (2015). La gouvernance par les nombres. Fayard.
[1] Loi n°78-17 of 6 January 1978, article 2.
[2] If it did, the whole construct enabled by citizen scoring would fall apart. If the way in which the score is computed could be debated, the political nature of the score would be obvious to all. Its intricacies would then need to be defined in law and through the judicial system, thereby nullifying the supposed benefits in efficiency and objectivity brought by governance through numbers.