Using maths to resolve a trade-off in disease testing
How Jean Pauphilet and Kamalini Ramdas used probability theory to develop a method that could slash the cost of testing
- There is a fundamental trade-off between accuracy on the one hand and cost and speed on the other in medical diagnostic testing
- In Covid-19 testing, this problem has resulted in huge resources being wasted on purchasing tests that then proved unfit for purpose
- Groundbreaking research proposes a methodology for policymakers to decide which cheap tests to use and how to combine their results for increased predictive accuracy
- Methodology has many potential broader uses, such as recruitment and machine maintenance
According to the European Centre for Disease Prevention and Control (ECDC), in October 2022 the pooled EU/EEA notification rate of Covid-19 cases among people of all ages had increased for three consecutive weeks, with 17 countries reporting an increasing trend. The ECDC report also noted, however, that forecasts of cases are “increasingly unreliable due to changes in testing criteria and reporting procedures”, and that, as a result, “all current forecasts, in particular case forecasts, should be interpreted with caution”.
One of the main causes of that unreliability is that, in diagnostic testing, there is a fundamental trade-off between accuracy on the one hand and cost and speed on the other hand.
Trying to resolve this trade-off was the motivation behind a groundbreaking piece of research entitled ‘Robust combination testing: Methods and application to Covid-19 detection’ by Kamalini Ramdas and Jean Pauphilet of London Business School, together with Sanjay Jain at the University of Oxford and Jonas Oddur Jonasson of the MIT Sloan School of Management.
Testing and economic activity
As the paper highlights, accurate diagnosis in epidemics, such as the ‘gold standard’ reverse transcription polymerase chain reaction (RT-PCR) test for detection of Covid-19 infection, can be very costly. This is because they typically require expensive lab capacity and a supply chain connecting the labs to testing locations. In contrast, point-of-care (POC) lateral-flow antigen tests are cheap and fast, but “largely fail policymakers’ accuracy criteria” – a drawback that has resulted in huge resources being wasted on purchasing tests subsequently proved not fit for purpose.
Nonetheless, there is a strong argument for deploying large-scale testing schemes to enable resumption of economic activity, such as by enabling safe re-entry into public spaces.
While far-reaching in its implications for testing for Covid-19 and similar disease epidemics, the new research was based on basic probability theory that has been around for a long time.
Dr Jain and Professor Ramdas, together with Professor Lord Darzi of Imperial College London, first proposed the idea of using mathematical probability to resolve the trade-off between Covid-19 test cost and accuracy in a Nature Medicine piece published in May 2020.
Dr Pauphilet says, “They realised that, where tests are cheap but may not be very accurate – for example, with the antigen test, the key issue was false negatives – potentially you could apply multiples of them and get a better result. If there is a 20% chance of a false negative on a test and you do the test three times, the odds of getting a false negative decrease greatly. Like when you are tossing a coin, the probability of getting all heads decays exponentially as you increase the number of tosses.”
Discover fresh perspectives and research insights from LBS
"Many of the theoretical ideas were already there – it was about adapting the theory to the reality of data"
Based on robust optimisation techniques, the research proposes an analytical methodology for policymakers to learn from data which cheap tests to use and how to combine their results for increased predictive accuracy.
Although the theory was not entirely new, its application to the disease-testing context was highly innovative. Dr Pauphilet says, “Many of the theoretical ideas were already there – it was about adapting the theory to the reality of data. There are many tests available and many ways to combine them, so you need data to select the few that will give the most meaningful outcome.”
Key contributions
The research uncovers two key methodological techniques that enable the solving of problems that were not addressed by existing methods, Dr Pauphilet reveals. “The first concerns test selection – out of all the tests available on the market, how do you identify the few that would make the most accurate combination? That’s where the algorithm we developed kicks in; by identifying not necessarily the tests that are individually the best performing, but the three (or more) tests that work best together, so you’re selecting which tests to combine in a smart way.
“Our second contribution was in finding a way to dispense with the assumptions underlying the tests – in probability language, whether they should be independent random variables or not – and to start from the data.
“We reached out to medical institutions that had conducted studies to compare the performance of tests and used their data without making any assumptions about the maths that generated them. The insight here was that we could use robust optimisation methodology to work directly from the data (with all its limitations) instead of making idealistic and unrealistic assumptions.
“In fact, what surprises me the most is the impact that a simple idea like this can have – the data was available very early in the pandemic, but no one had developed the theoretical lens to analyse it the way we did.”
AI meets crowd wisdom
The insights hint at a more fundamental wisdom: that of the crowd. Dr Pauphilet explains, “Again, it’s the idea of combining output from multiple individual models. In machine learning, for example, there are a host of methods called ‘ensemble’ methods. These combine multiple learning algorithms to give better predictive performance than you would obtain from any of the individual learning algorithms alone. The basis of this idea is that you train different models, interrogate all of them, and average the answers – it’s a kind of crowd wisdom applied to algorithms.”
If the idea had been around for a long time, the key innovation was to apply it to the use case of testing for diseases. The researchers believe the study is the first “to explore the value of optimally combining multiple diagnostic tests through analytical modelling and empirical analysis.”
Their findings also have direct and immediately applicable policy implications. The most obvious application is testing for Covid, as the paper makes clear: “Given our results, policymakers should consider approving and deploying combinations of tests in settings where RT-PCR testing is too costly or too slow, but individual POC tests do not provide enough diagnostic accuracy.”
This suggests immediate benefits in terms of public health. By combining multiple POC tests, widespread Covid-19 testing could be “simultaneously accurate, timely and cost-effective – and thus could help curb transmission while reducing unnecessary isolation.”
The methodology also allows policymakers to decide whether the need is for sensitivity (for example, for vulnerable populations for whom false negatives are costly), or specificity (for example, population-level seroprevalence studies, where the need is to detect the level of a pathogen in a population) by selecting the appropriate classification rule. The Foundation for Innovative Diagnostic Testing in Geneva (FIND), a WHO Collaborating Centre, tests samples using a battery of tests from different countries; this could provide the data needed for running the algorithmic approach tailored to country requirements.
Direct policy implications
And, because the research is based on an application-agnostic methodology, it has many potential broader uses, Dr Pauphilet says. “The methodology can be applied in any setting where data on different predictors is collected on the same subject. This could be used in HR recruitment, for example, or machine maintenance. The research demonstrates the value of structuring the data-collection process to enable both head-to-head comparison of different diagnostic tests and the development of combination-testing methods. In effect, it demonstrates the power of optimisation as a tool to integrate statistics and operations.”
Jean Pauphilet is Assistant Professor of Management Science and Operations at London Business School
Kamalini Ramdas is Professor of Management Science and Operations; Deloitte Chair in Innovation and Entrepreneurship at London Business School
