How to see inside the AI black box

Companies turn to AI for automation, but as its outputs come from complex algorithms, can we really understand them?

Featured faculty: Kim Keongtae
Written by Putro Harnowo

Almost a quarter million Uber drivers in Europe have recently filed a legal action against the company, alleging that its algorithms lowered their earnings. A similar reason has also prompted delivery riders in Edinburgh to demand that platforms like Deliveroo, Uber Eats, and Just Eat open their AI systems.

For these service partners, AI operates as a “black box”, where nobody actually knows how the machine makes decisions. This issue is not unique to the gig economy. The same complaints have made headlines in resume screening, rental properties, insurance claims, and more. The worst part is that even the companies deploying the AI have no clue how it works.

A recent survey from advisory firm Grant Thornton found that almost four in five companies lack confidence in passing an AI governance audit as they are oblivious to how AI makes decisions. In a 2024 interview, Twitter co-founder Jack Dorsey said that even if an AI model is open source, it effectively remains a black box because the model is fundamentally unpredictable.

Nevertheless, the EU has required AI systems to be designed with enough transparency for users to interpret and leverage appropriately. China has also released an AI evaluation framework to strengthen explainability requirements and address the black box problem.

A few methods have been developed to help human users better decipher AI outputs, known as explainable AI, or XAI. However, Kim Keongtae, Professor in the Department of Decisions, Operations and Technology at the Chinese University of Hong Kong (CUHK) Business School, argues that these techniques do not necessarily help user comprehension.

For instance, a rejected loan applicant may receive an AI explanation saying that a short credit history is the negative factor. While transparent, such explanations do not address practical questions like what the applicant should do to fix their shortcomings.

“Simply providing information for AI users may not build enough understanding of complex issues,” Professor Kim says. “Explanations work best when their style or format matches what AI users need to understand or do for a specific task.”

How to find the best answers

Cognitive Fit Theory suggests that humans make better decisions when they see an answer that matches the problem they want to solve, while Construal Level Theory argues that people think in detail about concrete, immediate situations, and in abstract terms about broader, distant problems.

Drawing on these theories, Professor Kim collaborates with Francis Costello of Nova School of Business and Economics, Park Jooyoung of Peking University HSBC Business School, and Jason Bennett Thatcher of the University of Colorado Boulder to introduce an advanced XAI method in their latest study, Gen(X)AI: boosting explainable AI for all.

Leveraging generative AI (GenAI) to produce explanations, Professor Kim and the team developed a conversational layer called GenXAI that translates AI outputs for laypeople. By combining GenAI and XAI, GenXAI translates technical AI outputs into interactive, understandable explanations that let users ask follow-up questions and explore the reasoning behind AI output.

“Simply providing information for AI users may not build enough understanding of complex issues.”

Professor Kim Keongtae

To test this idea, the team built a GenXAI chatbot and deployed it in two experiments asking participants to solve loan application tasks. They used real-world cases from a US peer-to-peer lending platform, LendingClub, to create a set of challenging loan default prediction tasks. Participants then chose whether each loan would default or be repaid in full.

The first experiment showed GenXAI contributed to 26.6 percentage points higher decision accuracy and greater confidence than standard XAI. The second experiment then split GenXAI participants into two: the abstract explanation group, who received broader patterns and narrative summaries, and the concrete explanation group, which emphasised explicit causal relationships and a structured presentation.

Participants with concrete explanations achieved the highest accuracy, up to 87.5 per cent, followed by the abstract explanation group at 75 per cent, and the regular XAI group had 67.6 per cent accuracy. Overall, GenXAI with concrete explanations increased the odds of answering correctly by 3.56 times compared to normal XAI.

Not only that. AI users exposed to concrete explanations process information more efficiently, spend more time thinking about the answers, and ask more structured follow-up questions.

Walking users through AI answers

Linguistic explanation style is a critical parameter in XAI. Explanation effectiveness depends on how well the explanation format matches the questions. Concrete and analytically structured explanations help users interpret and understand AI outputs significantly.

“Concrete explanations, characterised by specific numerical values, explicit comparisons, and structured reasoning, produced the greatest understanding,” says Professor Kim. “Participants who initially struggled with the task benefitted more from concrete explanations, suggesting this style may be especially helpful for those struggling to interpret AI outputs.”

Well-designed GenXAI explanations can also cultivate active engagement. While some may worry that easy-to-digest AI output may discourage analytical thinking, concrete GenXAI explanations show efficient information extraction followed by extensive deliberation, consistent with competency boosting rather than blind acceptance.

The primary beneficiaries

GenXAI would have a significant impact in financial sectors. Specifically, GenXAI improves AI users’ understanding and confidence in analytical decision-making, particularly in credit risk assessment, insurance underwriting, fraud detection, and other tasks where AI models evaluate multiple risk indicators.

Other industries with complex AI-driven decision processes can also make the most of it. GenXAI aims to bridge the gap between computational representations and human reasoning across domains, including resource allocation, market forecasting, and business strategy formulation. This asset is valuable in fields where specialists need to understand AI recommendations without being overwhelmed by technical details, such as in healthcare and engineering.

“Essentially, any sector where complex AI systems are used to make important decisions, and where human users need to understand, evaluate, and trust those decisions, would benefit from GenXAI,” Professor Kim adds.

What makes it outstanding is that GenXAI is designed as a conversational layer that sits atop existing AI models and their technical XAI outputs, without replacing the underlying structure. It basically takes the technical logic and outputs of existing XAI methods and translates them into interactive, natural-language explanations.

In essence, GenXAI acts as a conversational interface that ingests technical outputs from other AI models and their associated XAI tools, processes them through its own GenAI capabilities, and then delivers interactive explanations that clarify the AI’s reasoning.