How do education authorities end up firing ‘failing’ teachers based on an algorithm using flawed data and questionable assumptions? How do Bangladeshi farmers lose livelihoods to a flood predictor that gives lots of false warnings? Why are untested applications of data science infecting infrastructure, environmental alerts, policing, aid distribution, and health checks? Why don’t the heads of big banks know how reliable their programmes that pick who to lend to are?
Maybe it’s the hype around big data and AI, or the desire for easy automated solutions, but we are currently facing a lack of quality checks and standards in data science, and no one wants to question it.
Data scientists don’t explain and the rest of us feel too dumb to ask, rather like medicine 50 years ago. We are dangerously left to assume that someone somewhere in the development of the solution has asked the right questions. Have they?
We set out what those questions are for everyone, expert or not in Data Science: A Guide for Society in 2019. Since then Sense about Science has started work on seeking a significant global change. We have drawn from lessons (painfully learned) in medicine and in civil engineering, to propose the concept of Responsible Handover. It’s similar to what multiple contractors and commissioners do on a large infrastructure project – communicating what has been done, the extent of testing and limits of where it works at every point in development and application; every time it is ‘handed over’ to a different person, organisation or context. Questions about that can be asked by everyone – government procurement, politician, journalist, consumer.
We have made big changes in system and culture before. Success has depended on mobilising citizens and setting out the public interest case.
We need advocates and supporters to animate our examples of what is at risk, and for sector leaders to explain what this would mean practically in their area. We want major global influencers to take on and advocate for the principle of responsible handover and work out the systems for implementing it.
Thanks to a small grant from the Good Thinking Society, we are now documenting examples and concerns – to help people to understand the breadth of the problem and the need for responsible handover. We welcome collaboration and involvement. Please contact Errin Riley, Research Manager with your suggestions and views, or to speak in confidence.