"EU draft legislation will ban AI for mass biometric surveillance and predictive policing", 11 May 2023

The EU has taken a step closer to enforcing strong regulation of AI, drafting new safeguards that would prohibit a wide range of dangerous use cases.

These include prohibitions on mass facial recognition programs in public places and predictive policing algorithms that try to identify future offenders using personal data. The regulation also requires the creation of a public database of “high-risk” AI systems deployed by public and government authorities so that EU citizens can be informed about when and how they are being affected by this technology.

The law in question is a new draft of the EU’s AI Act, which was approved today by two key committees...

The AI Act itself is a sprawling document that’s been in the works for years, with the explosion of interest in generative AI tools this year forcing a number of significant and last-minute changes.

“It’s overwhelmingly good news,” Daniel Leufer, a senior policy analyst at nonprofit Access Now, told The Verge.

Sarah Chander, senior policy advisor at digital advocacy group European Digital Rights, told The Verge. “Never has the democratic arm of a regional bloc like the EU made such a significant step on prohibiting tech uses from a human rights perspective.”

As Chander suggests, it’s likely these laws will affect countries around the world.

The main changes to the act approved today are a series of bans on what the European Parliament describes as “intrusive and discriminatory uses of AI systems.” As per the Parliament, the prohibitions... affect the following use cases:

• “Real-time” remote biometric identification systems in publicly accessible spaces;
• “Post” remote biometric identification systems, with the only exception of law enforcement for the prosecution of serious crimes and only after judicial authorization;
• Biometric categorisation systems using sensitive characteristics (e.g. gender, race, ethnicity, citizenship status, religion, political orientation);
• Predictive policing systems (based on profiling, location or past criminal behaviour);
• Emotion recognition systems in law enforcement, border management, workplace, and educational institutions; and
• Indiscriminate scraping of biometric data from social media or CCTV footage to create facial recognition databases (violating human rights and right to privacy).

In addition to the prohibitions, the updated draft introduces new measures aimed at controlling so-called “general purpose AI” or “foundational” AI systems — large-scale AI models that can be put to a range of uses. This is a poorly defined category even within the EU’s legislation but is intended to apply to resource-intensive AI systems built by tech giants like Microsoft, Google, and OpenAI.

The draft legislation forces new obligations on large-scale AI systems like ChatGPT

Under the proposed legislation, the creators of these systems will have new obligations to assess and mitigate various risks before these tools are made available. These include assessing the environmental damage of training these systems, which are often energy-intensive, and forcing companies to disclose “the use of training data protected under copyright law.”

The last clause could have a significant effect in the US, where copyright holders have launched a number of lawsuits against the creators of AI image generators for using their data without consent. If the EU forces companies to disclose training data, it could open up lawsuits in the US and elsewhere.

Another key provision in the draft AI Act is the creation of a database of general-purpose and high-risk AI systems to explain where, when, and how they’re being deployed in the EU.

The creation of such a database has been a long-standing demand of digital rights campaigners, who say that the public is often unwittingly experimented on in mass surveillance systems and their data collected without consent to train AI tools.

Certain provisions in the legislation will cause a “major fight” with EU member states

However, the AI Act is still subject to change. After its approval today by EU parliamentary committees, it will face a plenary vote next month before going into trilogues — a series of closed-door negotiations involving EU member states and the bloc’s controlling bodies.

Chander says that some of the prohibitions most prized by campaigners — including biometric surveillance and predictive policing — will cause “a major fight” at the trilogues.

Following trilogues, the AI Act will need to be approved before spring 2024 — a tight turnaround for such a huge piece of legislation.