Prepared for the interdisciplinary symposium “Super-Scoring? Data-driven societal technologies in China and Western-style democracies as a new challenge for education.” Cologne, Germany; October 11, 2019. The essay can be downloaded here as a PDF-File.
By Larry Catá Backer
Introduction
Black(white)(red)lists have been an instrument of regulatory management for a long time.
“The city was filled with murder and there was no counting the executions or setting a limit on them. . . .Finally one of the younger men, Gaius Metellus, ventured to ask Sulla in the senate at what point this terrible state of affairs was to end. . . . “We are not asking you,” he said, “to pardon those whom you have decided to kill; all we ask is that you should free from suspense those whom you have decided not to kill.” Sulla replied that he was not sure yet whom he would spare, and Metellus at once said: “Then let us know whom you intend to punish.” . . . Then immediately, and without consulting any magistrate, Sulla published a list of eighty men to be condemned. (Plutarch, The Fall of the Roman Republic, Sulla, p. 31).”
In the West it has long been common for leaders of states (or those whose leadership guides the state) to establish proscription lists. The most famous are those produced by Marius and Sulla during the first of the great Roman civil wars in the century before the collapse of the Republic. The consequences of being listed ranged from loss of status, to loss of property and life. But it is equally well established to confer privilege; these (white or red) lists opened opportunity and signaled status. These lists were administered by any person, public or private entity with the ability to use the lists as a means of punishing or rewarding those on it. Chinese history is also full of lists created by officials and others. They are all were used to similar effect—to identify individuals or societies for the purpose of reward or punishment. That was accomplished either by listing or ranking; the former where the list itself contained the restriction or privilege; the latter where the list permitted others to use rank to determine consequence.
These lists have changed little in form, or function even by institutions officially leery of their use. By 2010 even the EU produced its “Visa Information System” for border management. These lists have always been instruments — these are the “middlemen” of regulation the operative for of which is expressed in a designation. That is, these lists merely articulate judgments (placement on the list) that is the product of applying data to an analytical model useful for separating those who belong on the list from those who do not. In effect, these lists are the product of an analytics — certainly crude by contemporary standards in the era before “Big Data,” but always becoming more potent as technological levels and the taste for using these structures increased all over the world. Lists are the way in which ratings are memorialized by reference to a threshold. They are a means of scoring respecting the aggregation of conditions necessary to produce a judgment of inclusion or exclusion from a list. Lists, however, in themselves are merely passive conclusions; they acquire potency only when circulated, and, in the wake of circulation, when they produce a timely social, political and economic consequences undertaken through and supported by the state. These consequences, themselves, must be keyed to societal values that resonate in ways that create incentives to support or condemn particular practices or behaviors that served as the basis for placement on the list. In other words, there must be a connection between that facts that produce assessments on which list are created, and the consequences, related to that data for placement (or non-placement) on that list.
The value of lists, then, lies in their utility for managing effects. Hierarchy, status, and privilege are the key elements around which black(white)(red) lists are activated. In contemporary societies, color coding merely makes the ultimate object of the list ore apparent; that is, the color is meant to match the consequences—black for punitive, white/red for privilege or reward. But while a single list may be useful means of actualizing a score, super-scoring becomes possible only by the development and coordination of a larger aggregation of lists, each potent within its own narrow field, but together capable of producing a coherent means of managing any aspect of societal expression (and its underlying beliefs).
It is in that context that it is possible to think about lists, and a developing list universe system, at the center of China’s Social Credit (CSC) system. This essay considers the role of lists in the construction of CSC conceived as a vast super-scoring system that effectively displaces law and administrative regulation as the engine for ordering society through government. The essay first very briefly describes the CSC system. It then considers two questions: (1) how does one build a super scoring system through the structures of CSC?; and (2) what role do lists play within that framework. It ends with a short consideration of what may be the principle challenges for political and general education that now arise in the context of these digital regulatory measures.
1. The Chinese social credit system
The CSC represents a new regulatory methodology which seeks to displace the traditional system of enforcing law and encouraging approved behavior, for a complex and interrelated system of rewards and punishments. That system of rewards and punishments is based on scoring the way that every social, political, cultural and economic actor in China conforms to laws, rules, and other expectations, and on the way that they organize their social, political and economic relations in conformity to expectations. The CSC system, then, does not displace the power of law, rules, norms, and the like to describe behavior expectations, but it transforms implementation by disconnecting the specific misconduct with a related punishment. For example, a violation of a court order to pay a fine may place an individual on a list of untrustworthy persons which then is used to prohibit the individual from purchasing airplane tickets. In effect, a CSC system is meant to rate behavior across all aspects of societal life. Rating every aspect of life is then the means by which behavior can be regulated (the law becomes the aggregation of actions which affects the ratings) and in this way to steer the behavior of all social actors—but principally of individuals, business, and eventually government officials. At its most developed, CSC as a regulatory system will merge into and become the way that law itself is expressed.
The CSC system has a moral dimension as well, which deeply informs its regulatory and enforcement dimensions. It was created, in part, to fundamentally steer the culture and practices of people in virtually every aspect of their lives. To those ends, the Twelve Cor Socialist Values unveiled in 2012 plays an important role. At the same time, CSC has as its objective to enhance the objective of achieving a rule of law society, that is, a compliance culture, by combining a power to rate compliance with rewards and punishments to touch on the ability of the individual or firm to function effectively in society. More importantly, it was meant to delegate that task to the state, under the guidance of its vanguard.
To that end, the list represents the end product of a process of data driven analytics, producing a conclusion related to each list (compliance with court order lists; student misbehavior lists; subway misbehavior lists; community service lists; financial responsibility lists; etc.) based on the factors weighed (e.g., spitting, eating, loud music playing etc. on the subway for the subway misbehavior list; timely payment of bills, traffic tickets, utility bills for a financial responsibility list) ) in order to determine whether a threshold quantum of conduct has occurred that merits inclusion on the list. The end product of that analytics—the list—then serves as the signal necessary to either encourage or compel other social, political, and economic actors to act (reward or punish) on the basis of inclusion. That is what gives list its power, something unchanged from the days of the Roman proscriptions under Sulla, and shifts regulatory power from the rules (ostensibly the object of all of this fuss) to the decisions about what rules will be given what weight to produce what sort of lists. One here moves far from an ordinary conception of ratings as a means to an ends. CSC ratings serve the critical element of data driven governance founded on analytics that makes it possible to regulate and steer behavior through data and analytics rather than through law and traditional enforcement. When produced in multiple ways by multiple ratings organs and coordinated by and through the state, one has in sight the possibility of a super-scoring mechanism that itself could point to a new way of organizing law.
2. Building a CSC super-scoring system
The Chinese Social Credit system understood as a complex network of coordinated scoring (rating) behavior in every aspect of organized life would be fairly useless if it merely the aggregation of the product of such scoring by a multitude of actors without an overall design. The Chinese system, in contrast, has over the course of the last several years, increasingly evidenced not just its coordination, but also the unity of a very ambitious conceptualization, an approach of fractured experimentation as the system moves from conception to implementation, and ultimately the bureaucratization and loose centralization of emerging scoring systems providing the platform necessary for “super” scoring.
A. Conceptualization. The initial conceptualization of CSC as a super-scoring system was first widely publicized in the now famous 2014 State Council Planning Outline for the Construction of a Social Credit System (2014-2020). The CSC was founded on the application of emerging developments in Leninist theory that began to embrace the idea of the need to overcome the governing methodologies of liberal democratic states which were inextricable from liberal democratic principles and culture. It followed that an emerging system of socialist market economy under a socialist political model grounded in socialist culture required a socialist approach to governance suitable for the times. This conceptual development ran parallel to the development of what would become by the time of the 19th Chinese Communist Party Congress New Era Thought. Its opening paragraph, in retrospect, provides an excellent summary of the principles underlying the model:
A social credit system is an important component part of the Socialist market economy system and the social governance system. It is founded on laws, regulations, standards and charters, it is based on a complete network covering the credit records of members of society and credit infrastructure, it is supported by the lawful application of credit information and a credit services system, its inherent requirements are establishing the idea of an sincerity culture, and carrying forward sincerity and traditional virtues, it uses encouragement to keep trust and constraints against breaking trust as incentive mechanisms, and its objective is raising the honest mentality and credit levels of the entire society.
The 2014 Planning Outline outlines an ambitious agenda—to remake society in line with the new era made possible by the success of the generation long project of Reform and Opening Up. To that end new methods are necessary to shape society. Rules are important (in the form of law, regulation, social norms, principles and the like), but they remain abstract and remote unless they can be internalized. But internalization is possible only by changing cultures of behavior and behavior expectations. As such, in order to internalize a new weltanschauung , it is necessary to inject it from the outside. It is here that super scoring moves from its normative objectives (the new era integrity society framed by the twelve Core Socialist Values) to its methodologies (data driven governance, ratings, punishment and reward). Law moves to the sidelines—it is reduced to the means by which the factors necessary for the production of scores, or rankings, can be determined.
From this core, the Chinese authorities have sought to implement Social Credit as a system. In the usual pattern of substantial reform in China, that required first a space for experimentation divided among a large class of principal actors (the scoring part). But it also required a space for the development of mechanisms for coordination and for the construction of appropriate punishment and reward systems (the “super” part of scoring). These are discussed next.
B. Fractured experimentation. Chinese authorities have traditionally embraced a “think centrally and experiment locally” approach to new initiatives. More importantly, the state authorities have tended to de-link public authorities from direct control of such experiments. The effect has been to produce the great centralizing conceptual vision at the highest level of the central government, but to implement the vision strategically in stages, first by delegating initial experiments in implementation to closely coordinated non-governmental groups and low level officials, and then by ensuring that those experiments take place either at a very local level near Beijing, or elsewhere quite far from the seat of the central government — southwest and southern China have since the last quarter of the 20th century served those ends, with Shanghai the center of market-finance oriented experimentation. In this way success could be incorporated into the building of a comprehensive system, and failures could be blamed on local officials or private enterprises who could be punished without affecting the “integrity” of the central government. In effect once the central authorities produced the premises and principles within which a system is to be constructed, code writing and application was fractured and spread out to key experimental centers. Once useful code was produced and successfully applied, it could be either expanded or the seeded elsewhere under systems of coordination in which the central government would again assume a leading role.
Thus, the super-scoring elements of CSC lies in central control at both the initial stage (conceptualization and vision-parameters; the normative structures and system objectives) and at the final operational stage (coordination, management, direction through rules based administrative discretion systems guided or controlled by a central administrative apparatus). The success of the super scoring system, however, is also premised on the fracturing of experimentation at the crucial initial implementation and “code writing” stage. Central to this effort was the objective of transforming scoring, or ranking, from an autonomous object to the expression in a simple and easy to use form of a complex analytics that applied the overarching conceptual principles of the 2014 State Council Planning Outline directly onto the bodies of its targeted individuals and enterprises (including eventually state officials as well).
This is precisely the model followed in the creation of CSC in the specifically Chinese context. The initial conceptualization was marked by the 2014 State Council Planning Outline. The Planning Outline was not the first but rather the last expression of a long process of working through the core conceptual elements that has been traced by some to provincial experiments in scoring in the early 20th century and then through the development of the Twelve Core Socialist Values and the New Era Theory of Xi Jinping. The conceptualization document itself then produced a timeline within which, from 2014 through 2020, a large set of more or less free experimentation would be encouraged, managed, or permitted, operated by both private firms (in strict coordination with the state) or lower level administrative units, especially outside of Beijing — plus the Beijing municipal authorities). In the meantime, as experiments produced successes or (useful) failures, the central authorities could begin refining (and also experimenting with) approaches necessary to weave these experiments together into something that could emerge as a coordinated comprehensive and self-reflexive system (after 2020).
The fractionalization stage is what has tended to fascinate Western observers. That in part follows because it has been the most visible element of this long and coordinated (and unfinished process). But perhaps also because the fractionalized experimentation phase is the one most accessible to Western oriented analysts because it can be analogized (and thus misread) along the lines of Western conceptual frameworks — the market, competition, and consumer choice incentive models at the heart of recent governance efforts including regulatory governance, markets based management and international soft law disclosure frameworks. It was marked by a number of high profile and quite well publicized efforts chronicled in both Chinese and Western press outlets. These included a number of initiatives. The most well-known of these include the licensing of eight companies to develop a mechanics of social credit scoring — that is to code data that could be usefully converted through analytics into judgments against which algorithmically produced consequences could be attached. The most well-known of these are the credit scoring systems developed or overseen by China Rapid Finance (Tencent) and by Sesame Credit (Alibaba). In simplest terms — they were charged with developing lists of individuals to be rewarded or punished in accordance with meta-objectives of the universal principles to be advanced by the systematization of CSC. Such lists could be single purpose — the lists of individuals and enterprises that failed to comply with judicial orders could be transformed into a list (subject to elementary analytics, e.g. threshold amounts, time lapse between order and payment, etc.). In the case of what is now known as social credit scoring, the effectiveness of the lists depends on their aggregation and weighting. That process of weighted amalgamation (the incorporation of a moral-normative measurability of data required second order analytics — a super scoring — that was meant to blend through a process of weighting the product of multiple lists and to convert the result not into a binary (on-the-list versus off-the-list) but into a score. For that the West proved useful — it seemed a simple matter to take financial credit scoring already refined in the West and used to rate governments, enterprises, and individuals, and deploy it for the more comprehensive objectives of the 2014 Planning Outline. These scores could then be used to simultaneously reward and punish.
A central objective of these fractionalized projects was the development of complex targeted analytics that could be reduced to lists of individuals reflecting conformity to one or more of the elements of the trustworthiness principles set out in the conceptual guidance from the central authorities. While the actual algorithms remain secret, at the initial stage, these private enterprise experiments in scoring did disclose categories of data that contributed raw material to the super scoring analytics. These included (1) credit history; (2) fulfillment capacity (compliance with public and private obligations); (3) personal characteristics; (4) Behavior and preferences; and (5) interpersonal relationships. Added together, the categories leave virtually nothing subject to data harvesting and analysis. For example, the fifth category permits harvesting data about social networks and rating based on the strength of interpersonal connections and the ratings of those with whom one has a relationship (friendship circles can raise or lower scores). Likewise, shopping habits under the fourth category can be used for a similar purpose (buying diapers may raise scores while buying too many video games may reduce scores based on an assessment of commitment to societal objectives). Similarly, the “quality” of social media postings could also find their way into the analytics of scoring. The third category also nudges. Scores can depend on where one lives and the sort of connections one has with society, including mobile phone and computing. The second category speaks to trustworthiness in commercial and private relations. But it can also be easily coordinated with public lists — for example lists of compliance with court orders, or lists of losing defendants, or lists of the rate of police complaints filed against an individual. The first category is the simplest. It includes the same utility and challenges as encountered in the West, but here it is not deployed only autonomously but is blended with all other activities to produce cross category behavior nudging effects. For example, if failure to pay bills on time reduce credit scores, it may impact the ability of the individual to obtain a visa to travel abroad, rent a car, utilize certain services, or send their children to a specific school. But the converse is also true—produce positive contributing data and privileges become available—cheaper loans, travel, schooling, credit, housing, faster internet speeds, and the like.
There were a number of other well publicized initiatives. These included the 2017 announcement by the Supreme People’s Court of the construction of a blacklist (Supreme People’s Court’s Judgment Defaulter List; 关于印发对失信被执行人实施联合惩戒的合作备忘录的通知) that started with almost seven million names of people determined to have engaged in threshold exceeding misdeeds who would be banned from taking flights. The Supreme People’s Court Judgement Defaulter List was coordinated through a memorandum of understanding with several dozen administrative departments that also issued no fly lists. Inclusion in the various no fly lists was tied either to misconduct related to air travel or broadly interpreted “untrustworthiness” as determined by the agency within the scope of their jurisdiction. For the Supreme People’s Curt that centered on judgement defaults; for the State Taxation Administration inclusion was triggered by a failure to pay taxes; for the Ministry of Finance on a finding of financial fraud, or certain overdue debt obligations; for the Ministry of Human Resources and Social Security inclusion was triggered by a finding of fa or failure to cooperate in investigations; and for the Securities and Futures Commission the trigger was failure to pay fines or failure of public companies to perform public commitments.
It was reported that a related effort was undertaken in Henan Province (Zhengzhou) where a shaming announcement was substituted for a dial tone on the phones of people who failed to pay debts per judicial order. In Hubei (Wuhan) the targets were students who could be blacklisted for committing an excessive number of rule infractions (cheating, unpaid tuition, and the like). In Shangdung (Rongcheng) the target was public social behavior of persons (jaywalking, littering) and in Beijing it was behavior on the subway. Not all experiments were targeting individuals. Business, especially private business was also an object of scoring. In Sichuan (Luzhou local authorities sought to implement a social credit scoring system for liquor businesses focused on regulatory compliance. Business ranking is likely to expand to foreign enterprises operating in China as well.
C. Bureaucratization, coordination, and loose centralization. The third stage, and the critical one for the rolling out of an “all around” CSC system is now currently in its initial phase. It is also among the more difficult (for Westerners) elements of the move form conceptualization to system to understand on its own terms rather than through a Western the conceptual lens. Very briefly, the State Council began this 3rd phase of the construction of the Chinese CSC (super scoring) system as early as 2016 with its “Warning and Punishment Mechanisms for Persons Subject to Enforcement for Trust Breaking.” The object was to begin the process of inter-institutional coordination of data and analytics. More importantly, it set the tone for consequences—while conformity could bring rewards, non-conformity must also produce punishment. Punishment was not to be penal or civil (as inevitable within a traditional law-administrative system) but rather serve a nudging purpose. The character of punishment then was transformed to a system of restrictions. The more one failed to conform, the lower one’s score (or the more likely the placement on a blacklist) and the more comprehensive and severe the restrictions that followed. The 2016 State Council Warning has a step in the direction of creating an administrative apparatus for the management of the scoring and rating systems being developed during the period of experimentation and localized implementation between 2014 and 2020.
These have been followed by provincial and local regulations seeking to implement portions of the State Council guidance and from 2019 on several important new directives from the central government that are meant to complete a conceptual architecture for the roll out of a nationally coordinated CSC by the 2020 self-imposed deadline. Among these are the Ministry of Commerce’s 17 July 2019 Notice on Printing and Distributing “Management Measures for the List of Business Credit Joint Disciplinary Objects. This one focused on the coordination and management of credit lists for businesses and sought to implement a portion of the State Council’s 2016 guidance. It centered jurisdiction of business social credit within the Commerce Ministry and its provincial apparatus, and established rules for compiling lists (under ministry oversight and rules). It specified the scope of data from which lusts may be created, including data on business legal or rule compliance, judicial decisions, and an open-ended category of “other laws, regulations, and regulatory documents.” It specified identifying information to be included on the lists and the reasons for inclusion. Also included was a specification of a quite broad range of restrictions that could be imposed on those listed, and the conditions for removal form the listing.
Also relevant was the 10 July 2019 distribution by the State Administration of Markets of its “Measures for the Administration of Serious Illegal and Untrustworthy Lists (Revised Draft for Comments). It specified the competent authorities for the management of specific categories of lists (and its underlying scoring) related to violation of the laws and regulations of market supervision and management including drug supervision and intellectual property management. It applies to enterprises, individual industrial and commercial households, other organizations, and natural persons holding specific positions within these institutions or who participate in market operations. It invites the establishment of a threshold based on “the subjective malice, illegal circumstances, and harmful consequences of the subject” and vests responsibility for guiding and organizing the lists on the State Administration of Market Supervision. It specifies a procedure for listing those enterprises and individuals subject to scoring and placement on the list that includes thirty-six circumstances on which guidance is provided. It also establishes a process for removal of subjects from the list. Lastly, it establishes ten categories of restrictions that may be imposed on those who are included in the lists. Provision is also made for coordination with other government and private organizations. Governmental organs are encouraged to develop strategies of joint punishment; relevant industry associations, professional service organizations, platform-type enterprises, and the like are encouraged to implement social co-governance. Finally, because ultimately the object of scoring is to rectify behavior, substantial attention is paid to the mechanics of rehabilitation.
Perhaps most important was issued 16 July 2019 by the State Council General Office on Accelerating the Construction of the Social Credit System — Guiding Opinions on Building a New Credit- Based Regulatory Mechanism. It speaks to further innovation in the organization of linage of credit supervision and the expansion of the application of credit reports. It also directs the strengthening of the chain of credit supervision. These include enhancing data warehousing as well as better targeting data harvesting, as well as disclosure system of lists. The latter is meant to provide enhanced publicity of inclusion on lists for the imposition of societal repercussions as well as the official restrictions that may be imposed by law or rule based administrative discretion. Provision for enhancing self-reporting is encouraged, with the suggestion that voluntary reporting can itself improve credit scoring. This parallels developments in the West where, for example, in the United States Justice Department exercises of prosecutorial discretion may be guided by the extent of the willingness of subjects to cooperate by complying with DoJ rules for establishing compliance and reporting systems. It calls for a greater development of a national credit information sharing platform that is standardized compatible. Related to this is the encouragement of cooperation among (and thus the approval of the operation) of cross regional, cross industry and cross disciplinary mechanism. It also called for a more efficient application of restrictions designed to induce approved behaviors. The Guiding Opinion also pointed to the need to improve mechanisms for identifying potential subjects of listing and connecting that to violation of laws and rules. The State Council divides this into two tracks—a market track for business and a personal track for individuals in their social behaviors. The discipline of government personnel is noticeable by its absence. But measures for “credit repair” are emphasized.
Importantly, the Guidance speaks to the important role of the “’Internet +’ and big data on credit supervision . . . [to] effectively integrate public credit information, market credit information, complaints and reports, and Internet and third-party related information, and make full use of next-generation information technologies such as big data and artificial intelligence to achieve comparable credit supervision data.” Protection of data integrity is also emphasized. The State Council also emphasized the construction of supervisory mechanisms and organizational leadership of social credit mechanisms and workable credit scoring systems. Important among these system creating measures are control of the narrative of social credit: “so that operators can fully understand and actively cooperate with new credit-based regulatory measures. Strengthen guidance and training for grassroots and frontline supervisors. Organized extensive coverage of news media, actively promoted credit supervision measures and their effectiveness, and created a good social atmosphere.”
Coordinating these elements has been a challenge for the administrative agencies charged with the implementation of social credit systems by the 2020 deadline. Those difficulties expose the ambitions as well as the challenges of building a coordinated social credit system on a national scale. At the same time, it suggests the relative ease of building less ambitious fractured small social credit systems within a smaller community of related stakeholders.
3. At the heart of super-scoring (social credit) systems—The analytics of lists:
The systemic construction of a national, coordinated CSC, then, represents an effort to substitute for law-based systems of behavior management, a system of restrictions and privileges based on a set of behavior models and goals, which is operated through a system of monitoring which is based on conformity to behavior objectives. This is data driven governance articulated through analytics, the consequences of which are established through restriction-reward algorithms. At the center of this system, then, are lists. Lists that follow rating and scoring behaviors (analytics) and provide the basis for the application of judgment (restrictions and privileges). Constructing a list, like the construction of the Social Credit system built around them, then, is the summary expression of the operation of the social credit system itself. To construct these lists requires a tight coordination of at least ten elements. Each of which is briefly considered below.
A. Entity (Subjects). Here the problem is one of authority and jurisdiction. The State Council has moved to organize the list of list-producing entities, but it has done little to organize the jurisdiction of each. Expect much in the way of overlap, and probably the existence of list “gaps.” Moreover, at least with respect to private list creators there is the possibility of conflict of interest or capture. Capture comes where the list producing entity is also a subject of social credit managed by another entity. Conflict comes when the list construction affects the social credit of the listing entity. The issue of entity touches two significant structuring challenges. The first is to align data driven analytics targeting national behavior objectives with the division of jurisdiction which divides authority over persons and activities among a very large number of governmental organs at the national and provincial levels. To some extent the State Council guidance of 2019 attempts to respond to that challenge. But it does so without disturbing the administrative structure of the state apparatus. The suggests the second challenge—the contradiction between “new era” data driven governance and the retention of traditional structures of state authority that fundamentally misaligns the character of regulation with its administrative structure.
B. Class of persons or institutions that might be included in the (color) list. Every list includes a universe of “subjects” which may be included on the list. This misalignment inevitably creates challenges of overlap (multiple administrative units with authority over an entity or an aspect of entity operations), coherence (multiple state organs applying different standards on the same entity with worst case irreconcilable measures, see below), and governance gaps (entities are not included in the regulatory universe). That suggests two significant consequences. The first is that every list necessarily excludes certain actors who exist outside a specific list system. It also produces a system in which different subjects will be faced with behavior standards from different sets of lists. This poses both coordination problem as well as a transposability problem. As importantly, it also suggests the likelihood of traps for the unwary (application of lists to unaware subjects) and the probability of inconsistent application of list analytics to the same individual data sets by different list creating entities. While behavior X may get one on List Y, it may be insufficient to get onto List Z. More interesting is the circumstance of contradictory analytics—where behavior P will get an individual on List B but keep that individual off List C.
C. The objective of list production (e.g., promote on of the 12 Core Socialist Values). The the moment this issue remains largely undefined. At its best it is meant to promote the great ideological objectives developed under the guidance of the CCP and reduced to obtainable specific norms. At its worst—where it is limited only to the process of law or lawful decision making, it becomes rudderless. It is clear that legal compliance and moral adherence to the 12 Core Socialist Values are contemplated. More generally important is the attainment of the CCP Basic Line as expressed in specific policies and interpreted in accordance to official approaches to the currently central ideas now organized as New Era Thought. But the translation of these great principles into operational objectives—that is into commands that can be coded by behaviors, becomes a difficult project. At the moment we have coding but less inclination to match coding to principle.
D. The sector of class of conduct around which data tied to the objective is limited. The State Council has already suggested an ecology of data harvesting and list making. There are divisions between individual and commercial sectors, between different industrial and commercial sectors, between regions and the like. The result creates potential incompatibilities among lists and their underlying analytics. It also produces another area of substantial challenges to coordination and unified management. This also touches on a related issue—who is harvesting data. The State Council has already spoken to the issue by focusing on greater efforts at self-reporting. But self-reporting can produce data bias and require multiple levels of (self) monitoring.
E. Data—bits of information that connects behaviors or activities. This points to the general problem of the identification of data that is useful. There are a number of issues; for purposes of this essay two are worth identifying. The first is the connection between data choices and rulemaking. That is choosing data identifies conduct with significance. Unchosen data suggests the reverse. People will conform their behaviors to comply with this hierarchy of importance which may have a perverse effect the consequences of which may not be apparent until it occurs. That has the effect of rulemaking. The second is the conflation of data and ideological perception. Individuals are only capable of recognizing data that aligns with their cultural conceptions of meaning. One “sees” race because culture has infused certain characteristics with meaning. That set of cultural constrains will inevitable corrupt the identification and organization of data. More fundamentally, data is transformed in CSC systems from an object (a thing one harvests for normative ends) to the way in which the normative ends themselves are defined. The choice of data serves as ther definition of the conduct one seeks to regulate and the behaviors one seeks to manage in a particular direction or with a particular outcome in mind. But that had been the traditional role of law and administrative regulation. Those are not necessary where they may be displaced by decisions about the character of the data to be identified and collected—and, as discussed in the next subsection, on the analytics applied to that data.
F. Analytics—the development of the process of producing meaning from data related to the objectives. This is fairly self-explanatory. Unless the process for choosing data is (deliberately or unconsciously) utilized as a form of hidden analytics, the data that is generated through monitoring carries no inherent meaning. Analytics is the process by which data is organized and is given meaning. But the coordination of multiple list systems grounded on multiple (and secret) analytics create challenges. This is especially the case where analytics permits discretionary choices that may vary among list making entity. The issue of significance looms large. What data or aggregation of data is significant? How does one measure significance or justify it against objectives or principles? To what extent is uniformity or predictability sacrificed in the development of a vocabulary of signification that may vary from list to list? These are questions that appear only lightly posed and largely unanswered. The great challenge here (and in the West as well) is secrecy. Societies are moving toward data transparency—there is no choice especially if data transparency serves the same function as the publication of law and administrative regulation (to give notice of expected conduct). But the analytics (usually termed ambiguously algorithms, a term that includes both the analytics and the application of consequences depending on the results of analysis) is viewed as property. Privacy, in this model, is then transformed from the protection of the subjects generating data (with respect to the use of that data) to the protection of the analytics (and algorithms) which then transform data harvested into consequences applied to the data producing subject. For Chinese CSC, a more significant challenge arises—the ability of the state or other organs to monitor the system itself; that is to ensure that it is operating properly and aligned with the normative objectives for which it was created—becomes far more difficult. Chinese CSC has yet to deal decisively (or at least publicly) with this issue.
G. Judgment—line drawing; what combination of data in what manner triggers decision to include ort exclude; can be as simple or complex as the analytics necessary to utilize the data. Analytics gives data meaning. It provides significance to data. But it does not produce judgement or consequence. That is the function of algorithm or administrative discretion applied to interpret not the meaning derived from the analytics but rather its consequences.
H. Broadcasting. Lists lose their power when they remain secret. But lists that are well distributed also likely reduce the power of the list creating entity to control its effects. Sometimes that is desired. When low personal social credit scores are widely broadcast, it is likely that a large universe of individuals and entities that come in contact with that individual will make decisions about their relationships based in part on those scores. That augments both restrictions (punishments) and privileges (rewards). But it also reduces the power of the entity to control those effects (unless those effects are in turn control by the mechanics of social credit). This becomes more problematic where an agency in charge of a particular industrial sector publishes lists of businesses over which it has oversight which empowers other agencies to impose restrictions without consulting and perhaps in an effort to expand authority at the expense of the listing entity.
I. Coordination. A solution to the challenges of broadcasting lies in coordination. Coordination is also a central objective of a national comprehensive social credit system. But complexity makes effective coordination difficult. And the realities of contests for power, influence, money etc. among administrative agencies, factions, officials and the like will make effective coordination a more difficult objective to meet. Though here there is a place for law, most likely it will be filled with contract. Coordination in China tends to be a function of coordinated Memoranda of Understanding, rather than of regulatory coordination. This is not unique to CSC; China’s Belt and Road Initiative is also built on contract. Nor is it unknown in the West; the administrative state is quite comfortable with interagency MOUs as a means of recasting webs of regulation into something that resembles coherence. Here on encounters a consequence of SCS on administrative practice in the sense of its incentive to make traditional legal structures more remote. And remoteness here also moves the apparatus of constitutional norms meant to protect the polity against governmental excesses or arbitrary conduct more to the margin: in place of rechtsstaat there is administrative discretion constrained by contract.
J. Consequences—beyond the list. Creating a social credit system, like the creation of a law-regulatory system in its contemporary form over the last several centuries, has proven to produce a broad range of unexpected consequences. It is not clear that large bureaucracies may be nimble enough to respond effectively when these consequences emerge. One of the more interesting emerging elements being developed is predictive blacklisting (forecasting trustworthy or untrustworthy conduct). Related to this is the “quality control” issue, and a mechanics for perhaps using super scoring to score the stakeholders in a social credit system. Chinese authorities have begun to recognize this problem. In 2019, the National Development and Reform Commission and the People’s Bank of China were tasked with the establishment of a tracking and evaluation mechanism to assess the construction of social credit systems among a set of specially designated demonstration cities and to adjust the model as necessary. Details, of course, are not available.
The entire enterprise of listing (black, white or red) cannot sit well with academics, government officials, political people, and officials committed to the principles of liberal democracy and markets as currently organized around its early 21st century orthodoxies. Some have argued that in the U.S. context some forms of big data-based triggering or screening initiatives constitute a liberty-depriving constitutional harm within the American constitutional order. Many more worry about the privacy implications and data protection. European might view the entire enterprise through a several year’s worth of policies at the national and national level—from Artificial Intelligence and ethical principles to the EU’s General Data Protection Regulations (GDPR). Still others worry about bias, implicit or explicit in data driven governance—from data gathering to analytics to the algorithms, with much made of the ability of programs to learn biases that reflect inherent in the coding through which programs are constructed, operated and their products analyzed and interpreted. Statutory structures in the West, like those in China, have begun to weave in pace quite distinctive ideological overlays for the management and discipline (as well as the application) of their distinctive variants of CSC. In the U.S. a marker of that difference may be found in the courts where, for example, in August 2019, the California appellate courts permitted the certification of a class action against Facebook by plaintiffs who allege that Facebook’s facial recognition technology violates Illinois Biometric Information Privacy Act, with implications for common law privacy standards and US constitutional protections (Patel v Facebook, Inc., US 9th Cir. Ct Appeals, 8 Aug. 2019).
But that is the point. The centrality of lists within CSC nicely spotlights the way that political ideology drives both the form and function of lists within these complex systems of restrictions and privilege which are meant to substitute micro-managerialism for the command enforcement model of law-regulatory systems. But it does more. It also embeds the ideology of Chinese Marxist Leninism, including its core legitimating principles into the regulatory model. These include the prominence of the leading role of the vanguard party, the centrality of the collective in social and political life, the prominence of advancing socialist market economy principles as a central objective of the state, and the imperative of embedding the twelve Core Socialist Principles into every aspect of life. There is little space left for the narratives of government function and the rights based normative principles on which liberal democratic systems are organized within this construct. That makes Chinese Social Credit systems difficult to transpose as a normative model to the West—but also easily transposable as a mechanics of management if undertaken by non-governmental entities, or reframed by the state to conform to Western normative standards. This, however, will require conscious effort.
At its root, Chinese CSC and the more privatized and uncoordinated Western approaches at subject to the same fundamental challenge—to ensure the integrity of data-driven governance systems as measured against its conformity to the core values and principles, that is to the “higher law” values of economic-social-political model. That means, in some respects, that Western and Chinese CSC are incomparable because their baseline integrity principles are founded on quite different ideologies each in its own sphere viewed as legitimate and authoritative. But it also means that while both systems may share and align the technical or methodological structures of their systems, the objects for which they are deployed are unlikely to converge. Therein lies a danger for both systems—to the extent that methodology is heavily embedded within ideological presumptions, even the borrowing of technique may pose challenges for the integrity of the system into which it is imported.
4. The challenge for political and general education in the context of these digital transformations.
The thrust of this essay might trouble academics, government officials, political people, and officials committed to the principles of liberal democracy and markets as currently organized around its early 21st century orthodoxies. Yet that troubling is necessary. Chinese Social Credit System construction is deeply embedded within the Leninist political model of China advancing the normative principles of Marxism with Chinese characteristics, a model with a structure of legitimacy and political objectives substantially incompatible with those of liberal democracy and its normative constitutional systems. It is to a fundamental understanding of those differences that any educational reaction ought to be focused. The opposite has generally been the case. Chinese CSC is usually studied through the lens of both Western political ambitions for China (e.g., that its system is flawed and must be nudged (an ironic use of the term here) toward transition toward some sort of liberal democratic model), and Western ideological premises. Those premises provide an ecology of verities against which the Chinese effort is understood as flawed and dangerous (to Western principles) with no effort made to connect Chinese CSC efforts to Chinese authenticating ideology. That is not to suggest that either ideological basis for system building is “right” or “erroneous,” only that systems become comprehensible only within the ecology of its own ideological framework.
A good example of the difference may be gleaned by a summary review of the OECD Principles on Artificial Intelligence were adopted on 22 May 2019. It consists of five normative principles (what the OECD terms “values based”) grounded in the sustainability enhancing notion of responsible stewardship that has gotten much traction in the business context among influence leaders in recent years. These principles inform AI in the West but are easily transposable to the Western perspective on the data driven governance framework of Chinese CSC. They include five principles. The first is that AI should benefit people and the planet by driving inclusive growth, sustainable development and well-being. The second is that AI systems should be designed in a way that respects the rule of law, human rights, democratic values and diversity, and they should include appropriate safeguards – for example, enabling human intervention where necessary – to ensure a fair and just society. The third is that there should be transparency and responsible disclosure around AI systems to ensure that people understand AI-based outcomes and can challenge them. The fourth is that AI systems must function in a robust, secure and safe way throughout their life cycles and potential risks should be continually assessed and managed. Fifth, that organizations and individuals developing, deploying or operating AI systems should be held accountable for their proper functioning in line with the above principles.
These five principles are then directed to the state, as is the habit of the OECD regulatory form. That direction is summarized in five recommended actions that states can take. The first is that the state should facilitate public and private investment in research and development to spur innovation in trustworthy AI. The second is that the state should foster accessible AI ecosystems with digital infrastructure and technologies and mechanisms to share data and knowledge. The third is that the state should ensure a policy environment that will open the way to deployment of trustworthy AI systems. The fourth is that the state should empower people with the skills for AI and support workers for a fair transition. And the last is that the state should co-operate across borders and sectors to progress on responsible stewardship of trustworthy AI.
While there are points of convergence—for example the focus on trustworthiness, even those notions are understood in substantially different ways in Western and Chinese data driven systems. Consider the differences even within the realm of AI itself. In its July 2017 New Generation AI Development Plan . (新一代人工智能发展规划的通知, the Chinese State Council noted “Artificial intelligence brings new opportunities for social construction. China is now in the final phase of building a well-off society in an all-round way. Challenges such as population aging and resource and environmental constraints are still severe. Artificial intelligence is widely used in education, medical care, pensions, environmental protection, urban operations, and judicial services, which will greatly increase public awareness.” (新一代人工智能发展规划的通知 machine translation). The primary challenge for education, then, is the ability to detach analysis of foreign systems from the biases inherent in our own. But this should not be taken as a call to abandon our values; quite the contrary. Such an exercise makes it possible to be more rigorous in the understanding and advancement of our values. But it also makes it possible to understand areas of convergence, areas of compatibility, and areas of conflict in the construction and operation of these systems. This is important especially when Western systems inevitably bump up against Chinese CSC. This is inevitable as global economies become more closely networked. And it becomes pressing as Chinese projects its Social Credit System through its Belt and Road Initiative.
This produces the second suggestion. To think of Social Credit as a problem of technology is to miss the most important point of the exercise. System building is not to be left to engineers. Its normative element is essential. Thus, CSC and more generally data driven governance becomes, at heart, a problem for law, for ethics, for values and for the principles to be advanced in the operation and delegation of authority to states and governmentalized non state actors. Perhaps better stated, they become an issue over the language of law and its mechanisms; where that language and those mechanics migrate to analytics it raises the further issue of the transposability of rule of law and constitutional principles to that transformed regulatory enterprise. What impedes this approach, of course, is the stubborn classical taxonomy of knowledge that continues to plague the scientific approach to knowledge. This perhaps may be the greatest challenge of social credit for education—in order to effectively study and advance a science of social credit, it may be necessary to rethink the current taxonomy of knowledge so deeply embedded in the West.
At its core, then, one might consider the following as the core challenges for education in the wake of these digital transformations. Two core implications are offered. There are no doubt others.
First, as painful as it may seem, academic culture must change. To insist on partitioning the study of data driven governance along the classical taxonomies of academic scientific organization is to invert the possibilities of knowledge production and reduce its value. The object of the study of these transformative changes in collective human behaviors is not to advance the greater glory of classical academic fields, whose turfs are myopically protected. Rather it is to advance knowledge in ways that reflect facts. The ideology of education itself, then may get in the way. For states and other actors, that may pose an issue—if academic institutions are no longer capable of efficiently advancing knowledge, then perhaps other means must be developed for those ends. This is particularly true with respect to CSC which requires a merger social science, law, philosophy and logic, mathematics, computing, and modeling.
Second, ideology must be exposed as a central element of scientific study. The object is twofold. The first is to ensure that scientific study controls for ideology to the extent necessary to advance knowledge in specific context. The second is to ensure an alignment of judgements about consequences—interpretation—with reference explicitly to such ideologies. While “facts” may have no ideology, the choice of facts, and the interpretation and application of facts is heavily embedded in ideology. This does not suggest cultural, scientific or political values relativism. It suggests rather that a central element of scientific study must be to identify “bias” (values which shade or nudge interpretation) either to correct for that bias (when dealing with systems with different bias sets) or to better align the objectives of that study to the support of that bias (for example in the West by interpreting scientific study against a baseline of liberal democratic values). The process must, however, be explicit.
5. Conclusion
My object in this essay was to encourage fresh thinking among Westerners by detaching them from their world view conceits as the lens through which analysis is undertaken, and to focus others on the challenges for creating a CSC system that itself mirrors the integrity advancing principles which is its principal objective. China serves not only as a harbinger of great transformations in technology, but in a technological revolution that is changing the relationship of law (and the state) to the advancement of its core principles. It asks the questions: if legal codex (the algorithm that is the self-referencing codes of modern states) is transformed into a “Super” Scoring Codex (an accumulation and combination of data from a variety of sources that is then actualized through analytics with nudging consequences) have political communities replaced one set of super algorithms (law) for another (scoring)? And if so, what consequence?
The essay deliberately turns the conventional way of thinking about data-driven governance upside down from the Western perspective but right side up from the economic political model of the People’s Republic of China as articulated and guided by its vanguard Party in accordance with its own normative documents. It starts from a set of premises, first that governance is driven by data; second that data has displaced the law; third, that the law has been transformed in function from an ends-in-itself to a means-to-an end. Only from those foundational premises is it possible to understand how data selection is itself a product of definite normative premises or structured to lead to identified objectives. From those premises, however well developed, it is then possible to understand the inherent and unavoidable bias of these systems however much its minders seek to present them as objective or “scientific”.
This is no indictment of China’s Social Credit system. Instead it suggests that the Chinese have been far more conscious of these fundamental premises and have sought to use CSC instrumentally to augment these biases which for them encapsulate the great political objectives for which they have a responsibility to advance. On the other hand, it serves as a reminder to Westerners who still believe that objective systems can be created that the structural foundations of such systems are meant to advance rather than reduce bias—and that a conscious articulation of those biases (seen as the positive principles and objectives to be advanced) is essential to the construction of any CSC whether in China or it the West.
These points are factually accurate yet likely will be received differently. Those who build systems, and therefore reason in terms of technology will agree with it. Those who build models, and reason in terms of the eternal truths of Mathematics will be disappointed and likely to be uncomfortable with this perspective. The discomfort emerges form an unwillingness to entertain the possibility that “objectivity” and “truth” in a sense have moved from the ideological premises of governance systems to the very act of modelling something. No longer produced by ideology, “objectivity” and “truth” are born when the axioms of a mathematical model are defined. The consequences for education are clear—as are the challenges for Western constitutional orders. For the Chinese economic-political model, on the other hand, the development of a comprehensive national social credit system fits in nicely with evolving principles of Leninism and with the fundamental principles around which the Chinese political model is organized. Fr China, Social Credit may well represent a more authentic expression of law closely tied to (and constrained by) the great political principles the administrative apparatus has a duty to apply.
For all that, the Chinese Social Credit system still has a long way to go to achieve its ambitions—the operation of a comprehensive mechanics for the management of all social actors through a system of restrictions and privileges based on compliance with legal standards. Those standards will reach substantially all aspects of organized life deemed important to the state. CSC are constructed from the inter-relationship of relevant data applied to analytics that incorporate the standards to be advanced or the principles to be applied which produce a scoring (as a function of compliance). Placement along a hierarchy of scoring then permits the construction of lists from which restrictions may be imposed or privileges granted. The list, then, reflects the application of analytics to data that then expresses a judgment in the form of a placement (ranking) along a continuum of compliance expectations. But the development of lists is still in a formative stage, and their integrity remains a subject for study. The coordination of these lists and their integration into the complex compliance system that is Chinese legal structures is still far off. Yet it is in the management of layers of lists that super scoring is possible—though not yet attainable. The resulting governance gaps remain substantial, while the concerns that might have bedeviled such a system in the West—privacy and data protection—are entirely absent. Data protection becomes protection of data integrity under CSC; privacy concerns fail as against the state and in any case the knowledge of list placement or scoring is an essential feature of the utility of the lists. Should it ever mature, it is likely to transform both the character of law and its relationship to managing behavior. In the process China will advance a means of governing that is substantially different from conventional systems identified with the forms and ideologies of Western liberal democracies.
* * *
References
Ahmed, Shazeda. “The Messy Truth About Social Credit.” Logic Issue 7 (May 1, 2019). Available https://logicmag.io/china/the-messy-truth-about-social-credit/
Backer, Larry Catá. “China’s Social Credit System: Data-Driven Governance for a ‘New Era.’” Current History —(forthcoming, Sept. 2019).
———-. “Next Generation Law: Data Driven Governance and Accountability Based Regulatory Systems in the West, and Social Credit Regimes in China.” 28(1) USC Interdisciplinary Law Journal 28(1):123-172 (2018).
Bo, Xiang. “China Boasts World’s Largest Social Credit System: Official.” Xinhuanet (14 Hune 2019). Available http://www.xinhuanet.com/english/2019-06/14/c_138143745.htm
Botsman, Rachel. “Big Data Meets Big Brother as China Moves to Rate Its Citizens.” Wired (UK) (25 Oct. 2017). Available http://www.guicolandia.net/files/expansao/Big%20data%20meets%20Big%20Brother%20as%20China%20moves%20to%20rate%20its%20citizens.pdf
Cheng, Evelyn. “China Wants to Track and Grade Each Citizens’ Actions: It is in the Testing Phase.” CNBC (25 July 2019). Available https://www.cnbc.com/2019/07/26/china-social-credit-system-still-in-testing-phase-amid-trials.html
China Law Translate. “Market Regulation Blacklist Overview.” Policy Papers 1 Aug. 2019. Available https://www.chinalawtranslate.com/en/marketplace-regulation-blacklist-overview/
———-. Core Structural Documents of the Social Credit System. Available https://www.chinalawtranslate.com/en/social-credit-documents/
Daum, Jeremy. “Who did China ban from flying?” China Law Translate 2018/03/21 available https://www.chinalawtranslate.com/en/who-did-china-ban-from-flying/
Daum, Jeremy. “The Redlists are Coming! The blacklists are Coming!” China Law Translate 2018/03/30 https://www.chinalawtranslate.com/the-redlists-are-coming-the-blacklists-are-coming/?lang=en
China, Ministry of Commerce. Measures on the Management of the List of Targets for Credit Joint Disciplinary Action in Commercial Affairs 【颁布时间】2019-7-17 【标题】商务部关于印发《商务信用联合惩戒对象. Available http://www.law-lib.com/law/law_view.asp?id=651440
China, Office of the National Development and Reform Commission, Office of the People’s Bank of China. “Notice on Printing and Distributing the List of Model Cities (Districts) for the Construction of the Second Batch of Social Credit Systems” Development and Reform Office Finance [2019] No. 849 关于印发 第二批社会信用体系建设示范城市（区） 名单的通知 发改办财金〔2019〕849号. Available http://www.ndrc.gov.cn/zcfb/zcfbtz/201908/t20190813_944514.html
———-. Outline of the Social Credit System Construction Plan (2014-2020) The division of tasks and the construction of the social credit system for three years; Notice of key work tasks (2014-2016). 国家发展改革委 人民银行关于印发 《社会信用体系建设规划纲要（2014—2020年）任务分工》和《社会信用体系建设三年 重点工作任务（2014—2016）》的通知. Available http://www.ndrc.gov.cn/gzdt/201501/t20150105_659408.html
———-. Division of Tasks for the Outline of the Construction of Social Credit System (2014-2020) Annex 1 16 Dec. 2014: 《社会信用体系建设规划纲要（2014—2020年）》任务分工 附件1. Available http://www.ndrc.gov.cn/gzdt/201501/W020150105530734502125.pdf
China, State Council. New Generation AI Development Plan (新一代人工智能发展规划的通知) ( July 2017) available http://www.gov.cn/zhengce/content/2017-07/20/content_5211996.htm
———. The General Office of the State Council “Guiding Opinions on Accelerating the Construction of a Social Credit System and Building a New Credit-based Supervision Mechanism.” 2019-07-16. Xinhuanet. 国务院办公厅印发《关于加快推进社会信用体系建设构建以信用为基础的新型监管机制的指导意见》
Available http://www.xinhuanet.com/politics/2019-07/16/c_1124760603.htm
———-. The General Office of the State Council on Accelerating the Construction of Social Credit System: Guiding Opinions on Building a New Credit-Based Regulatory Mechanism; State Office issued [2019] No. 35 国务院办公厅关于加快推进社会信用体系建设 构建以信用为基础的新型监管机制的指导意见 国办发〔2019〕35号. Available http://www.gov.cn/zhengce/content/2019-07/16/content_5410120.htm
———-. State Council Guiding Opinions concerning Establishing and Perfecting Incentives for Promise-keeping and Joint Punishment Systems for Trust-Breaking, and Accelerating the Construction of Social Sincerity Posted on May 30, 2016 Updated on October 18, 2016 GF No. (2016)33 国务院关于建立完善 守信联合激励和失信联合惩戒制度 加快推进社会诚信建设的指导意见 国发〔2016〕33号. English Translation China Copyright and Media Website. Available https://chinacopyrightandmedia.wordpress.com/2016/05/30/state-council-guiding-opinions-concerning-establishing-and-perfecting-incentives-for-promise-keeping-and-joint-punishment-systems-for-trust-breaking-and-accelerating-the-construction-of-social-sincer/
———-. State Council Notice concerning Issuance of the Planning Outline for the Construction of a Social Credit System (2014-2020) GF No. (2014)21. 各省、自治区、直辖市人民政府，国务院各部委、各直属机构：现将《社会信用体系建设规划纲要（2014—2020年）》印发给你们，请认真贯彻执行。国务院2014年6月14日. English Translation China Copyright and Media Website. Available https://chinacopyrightandmedia.wordpress.com/2014/06/14/planning-outline-for-the-construction-of-a-social-credit-system-2014-2020/
European Commission. Communication from the Commission to the European Parliament, the European Council, the Council, the European Economic and Social Committee, and the Committee of the Regions, Artificial Intelligence for Europe COM(2018) 237; {SWD(2018) 137 final} (25 April 2018).
———-. Visa Information System. Available https://ec.europa.eu/home-affairs/what-we-do/policies/borders-and-visas/visa-information-system_en
European Union. General Data Protection Regulation (EU) 2016/679 (GDPR).
van Hotum. “Human Blacklisting: The Global Apartheid of the EU’s External Border Regime.” Environemnt and Planning D: Society and Space 28(6):957-76 (2010).
Hu, Margaret. “Big Data Blacklisting.” Florida Law Review 67(5):1735-1810 (2015).
Kobie, Nicole. “The Complicated Truth About China’s Social Credit System.” Wired (7 June 2019). Available https://www.wired.co.uk/article/china-social-credit-system-explained
Larson, Christina. “Who Needs Democracy When You Have Data?” MIT Technology Review(20 Aug. 2018). Available https://www.technologyreview.com/s/611815/who-needs-democracy-when-you-have-data/
MacSíthigh, Daithí and Mathias Siems. “The Chinese Social Credit System: A Model for Other Countries?. EUI Working Papers: Law 2019/1 (Department of Law). Available https://cadmus.eui.eu/bitstream/handle/1814/60424/LAW_2019_01.pdf?sequence=1&isAllowed=y.
Ohlberg, Mareike, Shazeda Ahmed, Bertram Lang. Central Planning, Local Experiments: The complex implementation of China’s Social Credit System. Merics China Monitor (12 Dec. 2017). Available https://www.merics.org/sites/default/files/2017-12/171212_China_Monitor_43_Social_Credit_System_Implementation.pdf#page=12
Organization for Economic Cooperation and Development (OECD). Recommendation of the Council on Artificial Intelligence (adopted 21 May 2019. Available https://legalinstruments.oecd.org/en/instruments/OECD-LEGAL-0449
Osaba, Osonde and William Wesler IV. An Intelligence in Our Image: The Risks of Bias and Errors in Artificial Intelligence. Washington, D.C., Rand Corp. , 2017. Available https://www.rand.org/content/dam/rand/pubs/research_reports/RR1700/RR1744/RAND_RR1744.pdf
Pasquale, Frank, The Black Box Society: The Secret Algorithms That Control Money and Information. Boston: Harvard University Press, 2015.
Patel v Facebook, Inc., No. 18-15982 slip op. (9th Cir) decided 8 Aug. 2019. Available https://www.documentcloud.org/documents/6248797-Patel-Facebook-Opinion.html
Silberg, Jake, and James Manyika. “Notes from the AI frontier: Tackling bias in AI (and in humans).” McKinsey Global Institute (June 2019). Available https://www.mckinsey.com/~/media/mckinsey/featured%20insights/artificial%20intelligence/tackling%20bias%20in%20artificial%20intelligence%20and%20in%20humans/mgi-tackling-bias-in-ai-june-2019.ashx
Soldo, Fabio, Anh Le, and Athina Markopoulou. “Predictive Blacklisting as an Implicit Recommendation System.” 2010 Proceedings IEEE INFOCOM