Artificial intelligence (AI), and its impacts on privacy and other human rights, have been the focus of much attention in the past two months. From the European Commission considering a new AI-specific law, to the Australian Human Rights Commission (AHRC)’s final report on Human Rights and Technology, legislators and policy-makers are starting to grapple with the implications of this particular form of technology.
Why? Because from setting insurance premiums to deciding who gets a home loan, from predicting the risk of a person re-offending to more accurately diagnosing disease, algorithmic systems – especially those turbo-charged by AI – have the ability to re-shape our lives. Algorithms are increasingly being used to make predictions, recommendations, or decisions vital to individuals and communities in areas such as finance, housing, social welfare, employment, education, and justice – with very real-world implications. As the use of algorithmic systems increases, so too does the need for appropriate auditing, assessment, and review.
The AHRC for example has recommended, amongst other things, mandatory human rights impact assessments before AI is deployed in administrative decision-making, a ban on the use of ‘black box’ algorithms by government, a moratorium on the use of facial recognition technology in law enforcement settings, and the creation of an AI Safety Commissioner.
Privacy and consumer protection regulators are wielding their powers over AI developers under existing laws too. The US Federal Trade Commission has warned in a blog post that companies need to hold themselves accountable to laws requiring truth, fairness, and equity in the use of AI … “or be ready for the FTC to do it for you”. In April the South Korean Personal Information Protection Commission (PIPC) imposed sanctions and a fine on the developer of a chatbot for breaching rules on the processing of personal information.
So government and corporates alike have been put on notice that they need to lift their game when it comes to the development and deployment of AI, and other automated decision-making systems.
What is the privacy officer’s role in this? Where should a privacy professional start, when assessing the privacy considerations raised by algorithmic systems?
As a privacy professional, you cannot apply the law or assess risk without a basic understanding of algorithmic systems, and how they interact with information privacy obligations. You don’t need to be a technical expert, but you do need to know the right questions to ask of technical experts, so that you can assess the legal and reputational risk of what they propose.
To help privacy professionals get a handle on this topic, Salinger Privacy has just published our latest guide: Algorithms, AI, and Automated Decisions – A guide for privacy professionals. Here’s a little taster.
Are we talking about AI or algorithms?
At its simplest, AI is an umbrella term for a range of techniques and computer systems capable of performing tasks which would normally require human perception and intelligence, such as recognising speech, or recognising patterns in data. An AI system capable of recognising patterns in data is typically running an algorithm which was developed, and is continuously refined, by using machine learning (ML) to process vast amounts of ‘training’ data, to come up with correlations, so that the machine can ‘learn’ to recognise images, and be able to distinguish between – for example – an image of a cat and an image of a carrot.
An algorithm is basically a set of rules or sequence of instructions telling a computer what to do. An algorithm will typically instruct the computer to take certain inputs, and then perform a calculation in order to generate an output, which could take the form of a classification, a prediction, a score, a recommendation, or an automated decision. For example, an algorithm might examine credit history data and predict the credit risk posed by a customer; or examine electricity usage data and recommend the best product for that customer; or examine recruitment applications en masse and make automated decisions to reject certain applicants.
However not all algorithms are developed using AI. Algorithms can also be created by humans, writing code. These are known as rule-based systems. They can be as simple as ‘if X then Y’.
The trouble is, algorithmic systems do not come with the contextual understanding that humans have. This is sometimes purported as a good thing: if a machine makes a decision, then it won’t be as biased as a human one, right? Unfortunately it’s not that simple. If the rules encoded into an algorithm are unfair, or the data in which the algorithm is applied is biased, so will be the outcome. And removing a human from writing the code doesn’t necessarily solve this issue.
A machine learning model which classifies a picture as being an image of a cat or a carrot might seem like a simple task (and for humans, it is), but its ability to do so will depend on the data it has ingested, the way it was designed and developed, and the context in which it is deployed. If the algorithmic system was only trained on pictures of black and white cats, what will happen when it is deployed and has to classify a picture of a ginger cat? This is a silly hypothetical scenario, but real-world harm can be caused by mistakes such as these – and risks can be introduced at any point across an algorithmic system’s life cycle.
Which systems create privacy risks?
Algorithmic systems which raise privacy issues will include systems using AI, but also human-written rule-based code, which use personal information to make predictions, recommendations, classifications, scores or decisions about humans.
Systems which were developed using AI may pose particularly high privacy risk, but in our view all types of algorithmic systems should be considered. A recent example that demonstrates this is the Australian Government’s Online Compliance Intervention, more commonly referred to as ‘Robodebt’, which used an algorithmic system to automate a process with the goal of recovering debt. This system did not use AI, but its human impact – AUD$1.5B in unlawful ‘debts’ – was significant nonetheless.
How to assess privacy risk in an AIA
Evaluating the privacy risk of algorithmic systems via an Algorithmic Impact Assessment (AIA) is not just a matter of testing for legal compliance. In order to understand, identify, and mitigate against privacy-related harms, you need to think about concepts such as fairness, ethics, accountability, and transparency (when taken together, sometimes abbreviated to ‘FEAT’), which are vital factors to consider when assessing algorithmic systems. However we would also encourage privacy professionals to think more deeply about how to design trustworthy systems, by looking at both risks and solutions through the lens of what we call ‘The Four D’s Framework’.
The Four D’s Framework
The Four D’s offers a framework for assessing algorithmic systems in order to minimise privacy-related harms throughout the lifecycle of an algorithmic system.
The build of an algorithmic system comprises four stages:
- design
- data
- development, and
- deployment
Responsible algorithmic systems feature privacy risk management long before any personal information is used in a live scenario. Ideally, consideration of privacy issues should start as early as the design stage.
One of the first things to consider for any algorithmic system are the design objectives. It should go without saying that building an algorithmic system that processes personal information ‘because we can’ will rarely meet community expectations. Asking the question: ‘what is the problem we’re trying to solve’ is a popular place to start to ensure there is a clear understanding of why an algorithmic system is being pursued at all. But there are many other questions that are also important to ask at this stage, such as:
- Who will benefit from the development of this system?
- Who has this problem, and how do we know it is actually a problem for them?
- What do they say they need?
- Who is most at risk or vulnerable?
- Will this system meet the need/solve the problem better than what is already in place?
- How will we know if the system is ‘working’?
Organisations need to take care when considering the types of data that will be used to develop, train, test and refine their algorithmic systems. Data often only tells part of the story, and when processed without due consideration, can lead to misleading or even harmful outcomes.
For example, historic bias might be found within a training dataset, or bias could be introduced as an AI system ‘learns’ post-deployment. There have been numerous examples of this: the use of training data which reflected historic bias has led to racial discrimination in facial recognition algorithms, and gender bias in the Apple Credit Card.
Some questions to consider regarding data include:
- When, where, how and by whom was the data initially collected?
- Who is represented in the data? Who is not represented in the data?
- How will we organise the data?
- How will we test for bias in the data?
The development stage of an algorithmic project includes building and testing the algorithmic system.
If you are procuring a product or services, you will need to play an active role in the building and testing stage, or ensure that off-the-shelf products are rigorously examined and tested before deployment. Questions of accountability also need to be considered, as organisations cannot simply outsource their responsibility through procurement and hope to avoid liability for any harms caused.
For organisations developing their own algorithmic systems, now is the time to put the design thinking and considerations around data into action.
When testing algorithmic systems, organisations should first determine a baseline for each metric that they deem to be the minimal acceptable result. For example organisations should be aware of, and mitigate for, the possibility of evaluation bias, which can occur when the test data do not appropriately represent the various parts of the population that the system is planned to be deployed upon. Also, when developing an algorithmic system, organisations will want to test to see how well it is ‘working’, against metrics such as accuracy, recall and precision.
Once the algorithmic system is deployed into the real world, organisations cannot simply wash their hands of the system and let it run wild. Examples of machines ‘learning’ to be biased post-deployment include a recruitment algorithm which learnt that Amazon preferred to hire men, and Microsoft’s Tay chatbot which all too quickly learned from Twitter users to make offensive, sexist, racist and inflammatory comments.
The system may need to be changed to deal with real-world inputs and requirements for interoperability and interpretability, and there may be real-time feedback that needs to be integrated back into the system.
Another key area for privacy professionals to consider at deployment is how individuals will be meaningfully informed of how the algorithmic system is being used in practice, and how they may seek more information or exercise review rights. Decisions made or supplemented by algorithmic means should be explainable and auditable.
Conclusion
Privacy professionals should ensure they are involved in assessing privacy and related risks throughout the lifecycle of an algorithmic system’s development. In this blog we have introduced the Four D’s Framework as a way of conceptualising the breadth of issues to be considered in an Algorithmic Impact Assessment. Our new guide offers more detail on the legal and ethical risks posed by algorithmic systems, how to utilise the Four D’s Framework to guide system development, and a set of indicators which can be used to measure the trustworthiness of an algorithmic system.
Algorithms, AI, and Automated Decisions – A guide for privacy professionals can be purchased as a single eBook, or along with other resources in one of our value-packed Compliance Kits – see the ‘PIA Pack’ or the ‘Everything…’ option for your sector.
Photograph © Shutterstock
