Revealed: Facebook’s secret blacklist of “dangerous individuals and organization”

TO WARD OFF accusations that it helps terrorists spread propaganda, Facebook has for many years barred users from speaking freely about people and groups it says promote violence.

(Article by Sam Biddle republished from TheIntercept.com)

The restrictions appear to trace back to 2012, when in the face of growing alarm in Congress and the United Nations about online terrorist recruiting, Facebook added to its Community Standards a ban on “organizations with a record of terrorist or violent criminal activity.” This modest rule has since ballooned into what’s known as the Dangerous Individuals and Organizations policy, a sweeping set of restrictions on what Facebook’s nearly 3 billion users can say about an enormous and ever-growing roster of entities deemed beyond the pale.

In recent years, the policy has been used at a more rapid clip, including against the president of the United States, and taken on almost totemic power at the social network, trotted out to reassure the public whenever paroxysms of violence, from genocide in Myanmar to riots on Capitol Hill, are linked to Facebook. Most recently, following a damning series of Wall Street Journal articles showing the company knew it facilitated myriad offline harms, a Facebook vice president cited the policy as evidence of the company’s diligence in an internal memo obtained by the New York Times.

But as with other attempts to limit personal freedoms in the name of counterterrorism, Facebook’s DIO policy has become an unaccountable system that disproportionately punishes certain communities, critics say. It is built atop a blacklist of over 4,000 people and groups, including politicians, writers, charities, hospitals, hundreds of music acts, and long-dead historical figures.

A range of legal scholars and civil libertarians have called on the company to publish the list so that users know when they are in danger of having a post deleted or their account suspended for praising someone on it. The company has repeatedly refused to do so, claiming it would endanger employees and permit banned entities to circumvent the policy. Facebook did not provide The Intercept with information about any specific threat to its staff.

Despite Facebook’s claims that disclosing the list would endanger its employees, the company’s hand-picked Oversight Board has formally recommended publishing all of it on multiple occasions, as recently as August, because the information is in the public interest.

The Intercept has reviewed a snapshot of the full DIO list and is today publishing a reproduction of the material in its entirety, with only minor redactions and edits to improve clarity. It is also publishing an associated policy document, created to help moderators decide what posts to delete and what users to punish.

“Facebook puts users in a near-impossible position by telling them they can’t post about dangerous groups and individuals, but then refusing to publicly identify who it considers dangerous,” said Faiza Patel, co-director of the Brennan Center for Justice’s liberty and national security program, who reviewed the material.

The list and associated rules appear to be a clear embodiment of American anxieties, political concerns, and foreign policy values since 9/11, experts said, even though the DIO policy is meant to protect all Facebook users and applies to those who reside outside of the United States (the vast majority). Nearly everyone and everything on the list is considered a foe or threat by America or its allies: Over half of it consists of alleged foreign terrorists, free discussion of which is subject to Facebook’s harshest censorship.

The DIO policy and blacklist also place far looser prohibitions on commentary about predominately white anti-government militias than on groups and individuals listed as terrorists, who are predominately Middle Eastern, South Asian, and Muslim, or those said to be part of violent criminal enterprises, who are predominantly Black and Latino, the experts said.

The materials show Facebook offers “an iron fist for some communities and more of a measured hand for others,” said Ángel Díaz, a lecturer at the UCLA School of Law who has researched and written on the impact of Facebook’s moderation policies on marginalized communities.

Facebook’s policy director for counterterrorism and dangerous organizations, Brian Fishman, said in a written statement that the company keeps the list secret because “[t]his is an adversarial space, so we try to be as transparent as possible, while also prioritizing security, limiting legal risks and preventing opportunities for groups to get around our rules.” He added, “We don’t want terrorists, hate groups or criminal organizations on our platform, which is why we ban them and remove content that praises, represents or supports them. A team of more than 350 specialists at Facebook is focused on stopping these organizations and assessing emerging threats. We currently ban thousands of organizations, including over 250 white supremacist groups at the highest tiers of our policies, and we regularly update our policies and organizations who qualify to be banned.”

Though the experts who reviewed the material say Facebook’s policy is unduly obscured from and punitive toward users, it is nonetheless a reflection of a genuine dilemma facing the company. After the Myanmar genocide, the company recognized it had become perhaps the most powerful system ever assembled for the global algorithmic distribution of violent incitement. To do nothing in the face of this reality would be viewed as grossly negligent by vast portions of the public — even as Facebook’s attempts to control the speech of billions of internet users around the world is widely seen as the stuff of autocracy. The DIO list represents an attempt by a company with a historically unprecedented concentration of power over global speech to thread this needle.

Facebook Dangerous Individuals and Organizations List (Reproduced Snapshot)100 pages

Harsher Restrictions for Marginalized and Vulnerable Populations

The list, the foundation of Facebook’s Dangerous Individuals and Organizations policy, is in many ways what the company has described in the past: a collection of groups and leaders who have threatened or engaged in bloodshed. The snapshot reviewed by The Intercept is separated into the categories Hate, Crime, Terrorism, Militarized Social Movements, and Violent Non-State Actors. These categories were organized into a system of three tiers under rules rolled out by Facebook in late June, with each tier corresponding to speech restrictions of varying severity.

But while labels like “terrorist” and “criminal” are conceptually broad, they look more like narrow racial and religious proxies once you see how they are applied to people and groups in the list, experts said, raising the likelihood that Facebook is placing discriminatory limitations on speech.

Regardless of tier, no one on the DIO list is allowed to maintain a presence on Facebook platforms, nor are users allowed to represent themselves as members of any listed groups. The tiers determine instead what other Facebook users are allowed to say about the banned entities. Tier 1 is the most strictly limited; users may not express anything deemed to be praise or support about groups and people in this tier, even for nonviolent activities (as determined by Facebook). Tier 1 includes alleged terror, hate, and criminal groups and alleged members, with terror defined as “organizing or advocating for violence against civilians” and hate as “repeatedly dehumanizing or advocating for harm against” people with protected characteristics. Tier 1’s criminal category is almost entirely American street gangs and Latin American drug cartels, predominantly Black and Latino. Facebook’s terrorist category, which is 70 percent of Tier 1, overwhelmingly consists of Middle Eastern and South Asian organizations and individuals — who are disproportionately represented throughout the DIO list, across all tiers, where close to 80 percent of individuals listed are labeled terrorists.

Chart: Soohee Cho/The Intercept

Facebook takes most of the names in the terrorism category directly from the U.S. government: Nearly 1,000 of the entries in the dangerous terrorism list note a “designation source” of “SDGT,” or Specially Designated Global Terrorists, a sanctions list maintained by the Treasury Department and created by George W. Bush in the immediate aftermath of the September 11 attacks. In many instances, names on Facebook’s list include passport and phone numbers found on the official SDGT list, suggesting entries are directly copied over.

Other sources cited include the Terrorism Research & Analysis Consortium, a private subscription-based database of purported violent extremists, and SITE, a private terror-tracking operation with a long, controversial history. “An Arabic word can have four or five different meanings in translation,” Michael Scheuer, the former head of the CIA’s Osama bin Laden unit, told the New Yorker in 2006, noting that he thinks SITE typically chooses the “most warlike translation.” It appears Facebook has worked with its tech giant competitors to compile the DIO list; one entry carried a note that it had been “escalated by” a high-ranking staffer at Google who previously worked in the executive branch on issues related to terrorism. (Facebook said it does not collaborate with other tech companies on its lists.)

…

Read more at: TheIntercept.com