William Buckley famously described a conservative as a person who “stands athwart history, yelling Stop, at a time when no one is inclined to do so, or to have much patience with those who so urge it.”
While I am not sure that I classify as “conservatives” the judges discussed below, in the first week of February, we witnessed their important decision to stand athwart the historic rise of behavior-monitoring artificial intelligenc and order it stopped. AI’s growing influence on organizations has been taken for granted as unrelenting and inevitable, but we may have seen the first major effective push back against that tide.
On February 5, 2020, a Court in the Hague ruled that a Dutch automated surveillance system for detecting welfare fraud violated human rights and must be terminated. The court did not believe that the government of the Netherlands adequately weighed the system’s intrusion on human rights against the functional value of the system, stating:
“[T]he court is of the opinion that the safeguards provided . . . with a view to protecting the privacy of those whose data can be processed in [the system] are insufficient. Taking into account the transparency principle, the purpose limitation principle and the principle of data minimization, fundamental principles of data protection, the court deems the legislation insufficiently clear and verifiable for the conclusion that the interference that the use of [the system] entails in the right to respect for private life is necessary, proportionate and proportional to the purposes that the [system] serves.”[1]
One of the principal issues for the court was lack of transparency in how the machine learning system determined who was likely defrauding the Dutch welfare and tax offices. The court said that such a black box would not allow for affected citizens to be properly notified that their data was being analyzed or provide enough data for people accused of fraud to be able to defend themselves.
The court wrote:
“The principle of transparency requires accessible and comprehensible information, communication and simple language, and the provision of information to data subjects about the identity of the controller and the purposes of the processing. Irrespective of this principle, further information must be actively provided on the basis of this principle to ensure proper and transparent processing of data and natural persons must be made aware of the risks, rules, safeguards and rights relating to the processing of personal data and the manner in which they are processed on which they can exercise their rights with regard to processing.”
The Dutch court was also concerned about the European rules against profiling, stating that EU law “contains provisions with regard to profiling and a prohibition on automated individual decision-making, including profiling. In Article 4, section 4, AVG, profiling is defined as any form of automated processing of personal data in which certain personal aspects of a natural person are evaluated on the basis of personal data, in particular with a view to his professional performance, economic situation, health, personal preferences, interests analyze or predict reliability, behavior, location or movements.” The system in question, like nearly all anti-fraud software, involved automated processing of personal data evaluating behavior, reliability and performance of the people being checked. So this type of system clearly profiles individual as that term is defined in EU human rights protections.
[1] The quotations from the court are taken from a Google translation into English of the Dutch decision. It is possible that some of the text was clumsily translated and does not catch the essence of the court’s original language. If so, I apologize, and would like to know about any mistakes so I can fix them.
[…]
This is an excerpt. The full article can be read in its entirety at its original location by clicking the link below: