The UX of explainable AI

Artificial intelligence (AI) is a transformational opportunity for every business today. From algorithms that can assist doctors in diagnosing illness to the robot pickers that work in warehouses, AI is empowering and changing industries across the board. In day-to-day life, people are interacting with AI more and more – talking to AI customer service reps via mobile apps, using voice recognition with virtual assistants such as Alexa, or getting more powerfully attuned search results from search engines. All these applications and many more are powered by the ability of learning machines to analyse datasets, whatever the size, and identify complex patterns.

Yet, as AI becomes more sophisticated and better able to handle complex problems, it can also tend to more opacity in how and why particular outcomes arose. The ‘black-box’ conundrum stems from the unfortunate general correlation between the power and the opacity of many machine learning algorithms. That is, in general, algorithms that demonstrate greater performance in prediction, planning, or recommendations tend to be more complex and difficult to interrogate; even for expert users.

This has led both policy makers and AI developers to examine ‘explainable AI’. In a nutshell, explainable AI seeks to give users confidence in outcomes, by offering a rationale for how the algorithm arrived at its recommendation or prediction in a language and at a level that makes sense to the user. For expert AI developers, that might be a complex, mathematical explanation that allows them to change the functioning of the algorithm. For a layperson getting a decision on a bank loan, that should be a clear outcome-based explanation that outlines the reasoning behind the decision.

Explainability vs usability

The motivation for making AI explainable is clear. It allows for the development of more accurate algorithms and allows those algorithms to be understood and interpreted. This, in turn, allows for processes of transparency and accountability, so that users and developers can question decisions, seek second opinions or analyse for inherent biases.

However, explainability has to be weighed against usability. Take, for example, the implementation of the ‘cookie law’, part of the wider data privacy law known as the European General Data Protection Regulation (GDPR), which became enforceable in May 2018. This legislation requires websites to get consent from visitors to store or retrieve any information on a computer, smartphone or tablet. The law was put into place to protect online privacy, ensuring that consumers are aware of how information about them is collected and used online, giving them the choice to allow it or not. While most users are no doubt thankful of the greater control over their data, the result is a web plagued with thoughtless implementations of accept/reject dialogue boxes that make for an unpleasant user experience (UX).

Making explanations user-friendly

A broad programme of explainability in all AI algorithms would clearly be ineffective, rendering many applications unusable. Consider the use of AI in the predictive functionality of your smartphone’s keyboard. If you type in “L”, “I”, “F” with the language set to English, the program may predict that the next letter is likely to be “E” or “T”, because those are the most probable outcomes given the preceding letters typed and over time, your learned writing style. This prediction is unobtrusive, low-stakes and contributes to the UX. As such, any additional presentation of the reasoning behind these predictions is likely to only make typing very tedious. If the keyboard doesn’t suggest the word or letter you’re looking for, it’s not a big deal to correct it and move on, you’re rarely left baffled.

The difficulty is that more complex, higher-stakes algorithms need to be made transparent, while retaining a comparable UX.

In healthcare, physicians are beginning to make use of AI as an aid to interpreting medical images and thereby diagnosing diseases. The potential for enhancing the speed and quality of healthcare is enormous, in that the need for patients to have access to specialists would be greatly reduced. Instead, the patient could have the image taken by a technician at any hospital or acute care centre and then uploaded and would only need the time of the expert after diagnosis. Patients in rural and smaller urban areas would not have to wait weeks and then travel to larger urban centres to be examined, and the time spent in diagnosis could instead be used by the expert physician for treatment.

However, in order for a diagnostic aid to be useful for medical professionals, it must be transparent, providing clear reasoning for the predictions and suggestions that it is providing.

In 2018, Google’s DeepMind and Moorfield Eye Hospital undertook research that would allow doctors to better understand their AI system’s rationale¹. Using a database of over 14,000 retinal scans, the researchers created a system that could detect retinal disease. Importantly, they also created a second system that would analyse the decision of the first system and offer a percentage on the accuracy of the analysis. The first system essentially offered a list of features that supported its decision on the presence or absence of disease, which the second AI could show to the physician, along with its probability analysis.

The doctor could then look at the areas on the image that led the AI to identify disease themselves, and have the benefit of computer-backed probability of accuracy. This led to ‘the referral of 50 different eye diseases for further treatment with 94% accuracy.^'2, which equals or improves on medical professionals. In addition, AI also provides safeguards in the diagnosis process. ‘Not only can this technology provide confirmatory data to initial readings done by these professionals, but it can also catch things that might be missed due to human error, providing a second barrier of defense and a safety net for these diagnostics.’³

Beyond explanability to transparency and fairness

It's not just complexity that can prove challenging when developing usable and explainable AI. The opacity of 'black box' algorithms can also raise issues of fairness and bias. The problem with bias tends to arise because algorithms learn from datasets in which historical human decision-making could contain bias. A well-known example is the use of AI in criminal justice to identify potential offenders or re-offenders. For a time, researchers hoped that AI could in some way be inherently fair, since they believed a machine couldn't hold its own bias. However, it quickly became apparent that bias had been introduced to the system by the human data from which it had learned.

As an example, it wasn't enough for these programs to exclude protected characteristics such as race or gender that could unfairly influence decisions. Other data such as postal addresses or the type of job a person held could suggest their likely race or gender, and without the ability to interrogate the AI system for an explanation, it would be difficult to see if those biases were influencing the result.

In 2019, the Partnership on AI released a report documenting “serious shortcomings of risk assessment tools in the U.S. criminal justice system, most particularly in the context of pretrial detentions”.⁴

“To the extent that such systems are adopted to make life-changing decisions, tools and those who operate them, must meet high standards of transparency and accountability. The data used to train the tools and the tools themselves must be subject to independent review by third-party researchers, advocates, and other relevant stakeholders. The tools also must receive ongoing evaluation, monitoring, and audits to ensure that they are performing as expected, and aligned with well-founded policy objectives. In light of these issues, as a general principle, these tools should not be used alone to make decisions to detain or to continue detention,” the report concluded.

There are methods that developers can employ to adjust for human bias. These safeguards – often in the form of statistical tests – must be put in place to ensure that this bias is either eliminated from the input data or accounted for. On top of this, AI systems must have tests in place that continually monitor their output for any bias against particular protected classes.

User experience, accountability and explainable AI

There are no easy answers here. For businesses, there will always need to be a balance of explanation with usability. Lengthy disclaimers containing every possible negative consequence would be unwieldy and render many systems unusable. At the same time, leaving users in the dark about the intention of AI programs and their recognised unintended consequences is unacceptable.

For relatively benign, high-volume applications such as predictive text, an opaque, yet accurate algorithm is the commercially optimal approach. However, the use of AI for life-altering decisions in finance, healthcare, and safety critical systems are high stakes for both businesses and individuals, and users need to be able to interrogate the rationale.

In providing users with appropriate explanations, on top of diligent statistical analysis of models, and transparency of the processes that ensure fairness and accountability, it is vital that system designers are thoughtful about how these explanations are seamlessly woven into the user experience.

Business benefits of explainable AI

According to a Forrester study for IBM⁵, explainable AI has the potential to increase profits in the range of $4.1 million to $15.6 million. But by focusing on explainability, developers could also increase the accuracy of models by between 15% and 30% and reduce their monitoring effort by 35% to 50%.

There are significant business benefits in building explainability into AI systems. The EU's data privacy law, GDPR, is just one example of a growing body of regulation around the use of data and AI. When businesses choose AI systems to support their operations, they need to stay up-to-date with the evolving regulation of AI and explainability will be core to that. The European Commission has already proposed the creation of the first-ever legal framework on AI⁶ for its member states, which aims to provide AI developers, deployers and users with clear requirements and obligations regarding specific uses of AI.

As well as helping address growing regulation, explainable AI shows good ethical practice and will help businesses meet obligations for accountability. Banks will need to show why they granted or withheld a loan and doctors will need to be able show why they treated or discharged a patient. Where life-changing decisions are made, there’s accountability and there’s liability too, with the attendant risks of legal action.

For businesses with in-house developers, investing in explainability from the outset is likely to lead to building better algorithms.

In every case, AI only works if it is readily adopted by users. Businesses need to give their customers confidence that in the AI systems they’re using to encourage them to interact with them. Failure to instil that confidence could lead to a costly mistake, where significant resource investment in deploying new tech is wasted.