Is that your final decision? Multi-stage profiling, selective effects, and Article 22 of the GDPR

Abstract

Provisions in many data protection laws require a legal basis, or at the very least safeguards, for significant, solely automated decisions; Article 22 of the GDPR is the most notable. Little attention has been paid to Article 22 in light of decision-making processes with multiple stages, potentially both manual and automated, and which together might impact upon decision subjects in different ways. Using stylized examples grounded in real-world systems, we raise five distinct complications relating to interpreting Article 22 in the context of such multi-stage profiling systems. These are: the potential for selective automation on subsets of data subjects despite generally adequate human input; the ambiguity around where to locate the decision itself; whether ‘significance’ should be interpreted in terms of any ‘potential’ effects or only selectively in terms of ‘realised’ effects; the potential for upstream automation processes to foreclose downstream outcomes despite human input; and that a focus on the final step may distract from the status and importance of upstream processes. We argue that the nature of these challenges will make it difficult for courts or regulators to distil a set of clear, fair, and consistent interpretations for many realistic contexts. Data protection law provides a set of rights and obligations in relation to the processing of personal data. Amongst its substantive principles, such as lawfulness, fairness, transparency, and various procedural elements and risk-based measures that apply to personal data processing, it also addresses the use of automated decision-making systems. Provisions around automated decisions are not new, having been part of the data protection toolbox for several decades.1 The Data Protection Directive 1995 regulated automated decision-making;2 however in practice this was largely theoretical and the rights and obligations therein were rarely invoked. However, this previously dormant corner of data protection has begun to awaken in recent years, partly as a result of the rise of automation in the private and public sector, and partly due to new interest, novel concepts, and safeguards in the General Data Protection Regulation (GDPR).3 Article 22 of the GDPR contains provisions which restrict the use of automated decision making systems where they are ‘solely automated’ and have ‘legal or similarly significant effects’. The exact remedies the provision provides4 as well as their utility5 are subject to a debate that this paper does not seek to engage in. Instead, we focus solely on the scope of Article 22. A similarly worded provision exists in the Law Enforcement Directive, the GDPR’s sister instrument for police and criminal justice.6 Furthermore, many other laws internationally share a similar or near identical construction to Article 22.7 Whilst our analysis focuses on the GDPR to study this provision, the issues we raise are relevant more broadly. The relevant section of Article 22, Article 22(1) reads ‘The data subject shall have the right not to be subject to a decision based solely on automated processing, including profiling, which produces legal effects concerning him or her or similarly significantly affects him or her.’ Recital 71, which comments on Article 22, reads (in its relevant section) ‘The data subject should have the right not to be subject to a decision, which may include a measure, evaluating personal aspects relating to him or her which is based solely on automated processing and which produces legal effects concerning him or her or similarly significantly affects him or her, such as automatic refusal of an online credit application or e-recruiting practices without any human intervention. Such processing includes “profiling” that consists of any form of automated processing of personal data evaluating the personal aspects relating to a natural person, in particular to analyse or predict aspects concerning the data subject's performance at work, economic situation, health, personal preferences or or or where it produces legal effects concerning him or her or similarly significantly affects him or her.’ in these provisions have been in and The of of the has not been to any concerning Article 22 or its in the Data Protection The ambiguity of to in this is the nature of it is to a decision on automated other of human or is to an automated decision not solely The most significant that exists on this from the Data Protection to several including that the human should be in a to the and the of the and not they should have the to the automated and they should and are generally in this with the of of private in the of the and where the for decisions to be on automated processing in for safeguards to apply is The ambiguity of decisions have a on the data the examples of credit and in Recital of the has with several other examples of decisions effects as significant, including which from or is on a set of relating to human and These are which human a be or particular decision We these profiling The they are to in and around the of human and they we of these These are as Article 22 is including the in its as decisions its scope a legal basis, which in be difficult to due to the of a or The is as We examples of automated decision-making which multiple effects with different of legal or similar to the We that automated systems in multi-stage profiling and of human We challenges and complications that these in for their as the will these issues be in and ways. We The issues we focus on in this only These are: where are multiple outcomes with different where are different of human outcomes or of the where are multiple at which or human These in the in a decision-making context is is Article 22 does not apply due to the of a profiling to make decisions which to to human is human in at Article 22 does not apply are automated be the for with a which provides to a human in will apply their automated decision-making with significant is a decision to be and outcomes legal or similarly significant effects with human Article 22 apply an automated for to a credit which in the or credit with human in However, in such not the A an automated to analyse to a which are as to make as which are are for human which may result in an manual application are an automated which the application or it to a human in or the from or might human to a set of evaluating an or automated might be to these as and the to an for the the and make a decision the for is human in the an automated is to make the final make this we several in which multi-stage profiling processes be and which form the for the that These around different that automation in These are: to a which to a human decision or to automated decisions from or more human decision These in the we will analyse in to We not the decisions such as the or of a which may systems are the most of multi-stage automated decision systems. to a human decision to make a decision a where they are of The human decision is to the is not is in new are and as a which is to in the the or and use to the or to make a These have potential to to of systems that this systems to or such as the for in use in the to for where the is not to only to to the human in the are systems which result in solely automated where the status of the effects decisions whether they are legal or similarly is is in systems. are and However, in the the decision that the that a a credit which is as to or for human a of this the only which were in a the automated at relating to these and or These The the which to are a potential are subject to The a to in the decision of whether to upon on a The the and and with a of are in a other the to in or with the human The and a which or to to the of these the were to of of (in the of and or in the of may have been human at The systems in in the and in with the on systems the in light of the which that be subject to of are potential that a automatic or a human outcomes be other a of processes where the also a its which are with be to a human A this and is that and are to both the automated and to the human of this of is in the and set of which human for use where human to that are to to on the or their or These examples many more are in systems where are to different or a of different downstream However, these to the we to A final is in which the human decision-making is upstream downstream of the automated or more human decisions or are as and that data is or to an or which is to make a is in of this of with such as or or which are and to which or are The automated the of various human data and to it the an is to make a decision, an such as a an or to a for a based on and which a set of an in a the automated multiple human the may not be a or and on and have which the of multiple human from other and on have which are an of to other and which are to other and in to or from the these is human and in the data which are the final decision is in on the automated and the the is the of human in the data. The automated processes may be they may make in or in their such are not the any set of that with the to a human the or that a human and only as they these systems seek to processes a human have of the issues we may also in systems that seek to both and a that does not seek to also to a or or a more an of human systems such that of human as may the of systems Article 22, and are not as multi-stage systems for our as systems use data in part from human We not these a of systems in this in not to analysis of distinct a of in which automation human and and in a of complications for the scope of Article 22. The decision have to an is the least with to Article 22. decision outcomes are without human it that decisions are not on automated processing and in not subject to Article 22. are a set of challenges in and in to are the rarely or or the or to it might be that the decisions on automated is the in multi-stage decision-making the automated decision-making provisions which has the most public to is that we not has been is the where a in the data exists for decisions more the of the to a that it to be in the for this a of an has been the of decision of the that different of are be for both and for the profiling The be the for a of a to the human with a or the human the recent have this data from decisions in in which the is to make to that to the for use their on in in a very or the decision in other to have been the and in the of were as or human this that such a decision be solely on automated for that a the human to has an at least in relation to the they are to The the Data Protection decision have the and to decisions; that not and that they the relevant be selectively in this of selective automation is for decision is to with a particular they be in and a for an Article 22 decision that the decision has such in the may be a for at this of 22 decisions be this was to a human decision was based on the an the not be in of a and Article 22 may be for rise to such a A is that of where a A decision might be in potential in systems. We might locate it in the of of a be a decision to a subject the the to are a potential are subject to The for it be in ways. be that without on such a an not have been and such that they their the profiling at an the significant not have be in a such as the where both profiling and are part of an a potential in a which was not the data of that might that a profiling was to The of to the of and the of is an in automated is to be as significant in data protection law the of as similarly significant to due to the of the of Article 22 as to we might locate the decision in which only and from which an is an has been a for have the to whether or not to the for any relevant has only been at this we might a that the decision not have for the the potential of it at a might to the decision is of these of the decision we may have a on whether or not Article 22 the automatic is where the decision and it produces the downstream of to we is Article 22 from the for or a provision in as well as safeguards, to be were the decision only the human Article 22 decision not be based solely on automated processing, was a of as in the it is that the profiling a in the significant to where downstream it may not be However, in it will be a in a with significant effects and a profiling it will be that at in a human will in such a as to the significant the is in which the most of systems may for the scope of Article 22. to this in law or the step this is that regulators may to and seek on a they they were the challenges of a of is to be in such a as to be to are not We not a has an of whether a is or not it has been at which we might as our is or potentially we are a decision is and is the of whether ‘significance’ is a that is where is a ‘potential’ of a significant or not a significant has been it is the the of an might be in scope for Article 22. it is only the decision, are for that only with decision such as a should be in scope of Article 22, the for legal in Article and provisions in other be a selective with a particular decision subject a both the and the ‘potential’ have in the Recital of the GDPR examples of Article refusal of an online credit automatic the rights in be processing which an of whether a decision the of Article 22 to in of both the The in the rights with a on also a of this to be at least in of Furthermore, the remedies in Article seek human a of and to a make a decision has been is for decisions on human the decision an Article 22 decision at the of to human is the A data argue that they on in to the data processing for decision-making based on automated processing, such as having and to be seek a on the of automated processing with an significant such as as be to to a where they that a human be were they not that Article 22 a from a data subject to to an automated decision without the which to the of Article 22 as the for automated decision-making in that safeguards on the and other clear, it is to from to other and have in the the ‘potential’ to make more the ‘realised’ is a is a of to credit to be a to their of credit to potentially in of might for the may be on a will not be in a that is not will require significant to or with the data protection more broadly. our the ‘potential’ is the only that A decision should be it is as such for be subject to are to the of that a as potentially significantly for it to an automated decision a A data protection impact is the where such decisions should be and a data and a this be the of whether a or the ‘potential’ to be more the interpreting in this in a more for data subjects have Data systems, it to this as an Article 22 a potentially have their as a result of the automated processes to their on a systems such as where of automated are to it such as terms of of legal obligations and the a be to a in relation to of that for is a very of a in or that to or be with the of to at least in part upon The of refusal of this to this might in a an of effects as with an of that effects the this that to as is a and not in scope and in of a legal having as for human is not a significant decision has been at that may be potential for an automated decision with as is potential for an automated decision with a are only a human which is the scope of Article it that was to a human to was the for the to Article 22, Article of the Data Protection Directive 1995 the scope to decisions which an legal or similarly with the law which this However, such it the or the their of their the Article that significant effects be or a the of their and on the the GDPR with its sister instrument for the Law Enforcement Directive, which in to the GDPR that only effects as well as the of in the of the Data Protection Directive, as that the the provision to apply to significant effects of their The that the of in relation to be as a the for a be an this not to have their be and it they are it be that for is that they should to be more the in other such are to be more and to A does not to be a from a to be at a as a of to the of Article 22 of processing where rights and are most at for the of that having an a significant that the in produces significant the has been as an a human for more and has their has an automated decision been it be that a human has their to the automated the decision is not on automated However, this be to without the at the automated the with not have their for the the not have been that the be in was have that of the automated as it is an of both is not the a where are outcomes The step is an automated to of of human A or and are significantly different to outcomes and human A only to or and human only to or These systems are not have might in systems which are to the and in a or for human to measures or such as or also a Such a may where an automated the part of a that an is for and a in that automation automation this the automated which to the A or is to that or a they might be for a will it be in or from the in that the human to any of the this the human are in their decision-making the automated is to that an automated decision has not been be that are decision outcomes in the the in which an will a and the the exact on decision-making that be a in that upstream has the potential to be a use of Article 22. However, aspects of the decision that as the of the outcomes and not be is that with a the the as well with with this the outcomes in the have been to the credit is that an to as is that the is the or other in this make a to as a to a The automatic and manual human might more personal be The of a have application of human to only that result in a might to an of the human to for their in be a significant Furthermore, automatic in where are may the of have to for human to the may be as a result of the of in a particular other for at a the it in the to as a significant not from the impact on for the impact on an decision-making may have its the selective application of manual processes. complications we the in which automated and human elements of the decision-making are to in the of this it is to of the decision in a decision-making as final not only in the and also as the of decision-making is that we might to the final step in a decision-making and that step is automated, the to be a human making the final decision the However, as of the will not processes which with an automated step are the examples of A which to and to a decision an may be the final step in a the upstream human be to the not solely provides an to the that the final step in a to be a human in for the decision to not be solely was for this that the decision has a this is only in the decision be personal aspects relating to him or will be in which the automated step more a or of data. a which natural processing to in to a on a of their make the final automated processing step a decision in its where the on the of and the right of and the final step automated should not the solely the automated has to on and decision be processes which with a human step may decisions which are solely The examples of processes which foreclose outcomes and human this might be the outcomes a final human the outcomes to the human to have been an automated at an automated be to have a significant the a human the of whether the final step in a decision-making is human or automated is not a to the status 22. is other to be on the final step in a as the of decision-making a in which Article 22 is only multiple automated decision-making are together an might a on the it may apply a to an or the to a human to the upstream analysis upstream analysis this are the automated step does not result in any solely automated decisions which legal or similarly significant effects; such decisions are only a this any upstream automated processing from the scope of Article The to which human at the final step upstream automated processing of scope as i

Cite this work

No public copy is stored for this record, so only the metadata is cited.

Stored only when both papers are in the corpus — 65 citations exist upstream.