Part 2 -- Modelling COVID-19: So, what does an effective approach to simulation look like?

BLOG POST 2 of N

This post is the 2nd of several devoted to addressing the complex challenges of modelling the coronavirus as a public health issue. It is also about clarifying for a wider audience how and why such modelling is important, as well as the value and power of complex systems thinking and computational modelling for public health policy. The first post -- Modelling the coronavirus: why all public health models are not the same -- can be read by CLICKING HERE.

So, what does an effective approach to simulating COVID-19 look like?

The focus of the first post was to explain how models influence public health policy and why some models are better at modelling COVID-19 than others, given the challenge of complexity. The post ended asking the question: So, what does an effective model look like? In response I said I would review two of the models getting the most attention, which are very good in their particular ways. The first is the simulation model by Ferguson and colleagues at Imperial College London; the second is complex network model by Alessandro Vespignani and colleagues at Northeastern University in the States. 

Before we turn to these models, however, we first need to understand what a public health infectious disease model looks like in the first place. NOTE: I am not going to reinvent the wheel here, as there are a ton of video tutorials, books, articles, workshops, white papers, and on the basics of simulations and complex network models. In short, I will keep to the basics and provide links for more in-depth review elsewhere.

Public Heath Infectious Disease Models: The Basics

1. Modelling is more about learning and less about predicting

Of the various things to get clear first is that there is a fuzzy divide between those who think public health models are created to make accurate predictions about the future; while others (myself included) who see them more as learning tools that can help us think better about a topic -- albeit with data forecasting still being incredibly important! And the latter -- that is modelling to learn -- which works even better when 'learning' is done through a participatory and co-production approach. (For more on participatory approaches, see the INVOLVE project.)
 

Bottom line: when it comes to highly complex systems, the fact is that agent-based models, complex networks, survival models, microsimulations are not particularly great at telling us what exactly will happen -- in fact, nothing is! 

However, they do seem rather good at telling us what sorts of public health outcomes might happen and how we might respond to these outcomes and so forth. In other words, they seem better when used as learning tools (albeit as well as forms of cautious prediction). And that is, it seems to me, exactly what a lot of the public health models these past several weeks (circa March 2020) have done: they have challenged a certain way of thinking in the government of focusing on mitigation alone, and probably really helped to put us in the right direction, even if their predictions on mortality and rate of spread are not (or will not be) exact.

   

2. Modelling multiple trajectories, not a single outcome!

The reasons all of these models fall short of accurate predictions is because complex social systems are, by definition, comprised of multiple and different trends that evolve simultaneously.

Also, as I mentioned in my first blog post, the agents in social systems are learning animals and therefore interact with and respond to changes in the system, including policy interventions. At the aggregate level, these responses and their intersecting interactions self-organise into a larger emergent pattern that forms a whole that is more than the sum of its parts. 

Predicting these emergent aspects and the multiple and different trends upon which these patterns are based is further complicated by the fact that the outcomes will also differ for different groups and trends (a point I made in the first post relative to case-based configurational thinking). For example, if a certain set of guidelines for behavioural change are suggested to mitigate COVID-19, they may work much better to the advantage of affluent communities (which have the resources and healthcare to carry out the recommendations) as opposed to a poor community lacking in resources with an already-strained healthcare system. 

Example of multiple trends and emergent patterns in depression/physical health study

3. Thinking about complex causality in modelling

When running any model of a complex system, such as pandemic like COVID-19, please take the following caveats into consideration when thinking about causality: 

(CITATION: The following points were developed from my reading of Moore, G. F., Evans, R. E., Hawkins, J., Littlecott, H., Melendez-Torres, G. J., Bonell, C., & Murphy, S. (2019). From complex social interventions to interventions in complex social systems: future directions and unresolved questions for intervention development and evaluation. Evaluation, 25(1), 23-45.)

First, we need to be clear that no evaluation method, including a complex systems approach, “will ever be able to address the almost infinite number of uncertainties posed by the introduction of change into a complex system” (Moore et al, 2019: 36).

However, adopting the type of case-based systems lens suggested by complex systems approach (which I outlined in my first post) may help to “drive the focus of evaluation (i.e. which of the multitude of uncertainties posed by interventions in complex systems do we need answers to in order to make decisions, or move the field forward)” (Moore et al., 2019: 36). It can also help to “shape the interpretation of process and outcomes data” (Moore et al., 2019: 36). 

Second, “complex interventions in complex social systems,” including exploring such strategies via a complex systems approach “pose almost infinite uncertainties and there will always be much going on outside of the field of vision of an individual study” (Moore et al., 2019: 37).

“However, a focus on discrete impacts of system change” as if often done with complex systems approach, “does not necessarily betray a naïve view of how systems work, but may simply reflect a pragmatic focusing of research on core uncertainties” (Moore et al., 2019: 37). And this is, for me, one of the most powerful provisions of a computational modelling perspective (particularly simulation, network analysis and machine learning) given its focus on interventions in complex clusters and the configuration of factors of which they are comprised, be such a study cross-sectional, pre-post, or longitudinal.

Third, we need to strongly emphasize that the complex systems approach is a learning environment that bridges the computational/quantitative/qualitative divide, as it requires users to be in direct and constant (i.e., iterative) interaction with the complex systems approach environment and their respective theories of change, be they sitting implicitly in the background of their minds or formally outlined and defined. For example, as Moore et al (2019) state, “Of course, it is never possible to identify all potential system level mechanisms and moderators of the effects of an evaluation,” (Moore et al, 2019: 39), even in the case of complex systems approach. Additionally, “no evaluation would be powered to formally model all of these. However, combining quantitative causal modelling [as in the case of complex systems approach] with qualitative process data [in the form of user-engagement with the simulation platform] can play a vital role in building and testing theories about the processes of disrupting the functioning of complex social systems to optimize their impacts on health” (Moore et al., 2019: 39).

In short, putting all of the above points together, the goal here is not necessarily about identifying some underlying causal model, as much as it is about exploring and learning how various interventions or strategies might play out for a given policy and the larger complex system in which it is situated. And such a goal is, while humbler, nonetheless very important.  

4. What, then, can models tell us and help us do?

Given the above realities, the value of modelling is that models (no matter the form) are useful heuristic tools for exploring the implications of what might happen in relation to different sets of governance interventions. One can think of such interventions as control parameters which can operate to determine (in the sense of set boundaries not exact specification) the future state of the system. This seems to be what governments around the world are doing in response to the implications of the Imperial College model and other such models.

In other words, the best that can be achieved is a tentative forecast of several possible futures. However, that would require policy makers and citizens to actually implement the strategies described in such a model. But, even then, if implemented at a threshold of 100%, most of these possible futures will not occur anyway. As such, the value of modelling, when done well, is that they allow policy makers, scientists, healthcare providers, public health officials, businesses, and the general public to talk about scenarios for which precautions could be taken.

And so, to reiterate, as one of my colleagues recently pointed out in an email -- it is crucial for modelers to ensure that model users (policymakers, healthcare providers, government officials, the general public, etc) treat these models as decision support tools, not prediction machines. Unfortunately, this is not often made clear or understood by users. It is also not made explicit by model developers in their well-intentioned enthusiasm to help. 

(For more on this point, see (1) Computational Modelling of Public Policy: Reflections on Practice; (2) Ideal, Best, and Emerging Practices in Creating Artificial Societies; (3) Using Agent-Based Modelling to Inform Policy – What Could Possibly Go Wrong? See also, the Journal of Artificial Societies and Social Simulation and also, relative to policy, the Centre for the Evaluation of Complexity Across the Nexus.) 

So, to summarise: what does an effective approach to modelleing COVID-19 look like?

• It is one that embraces complexity.
• Takes a case-based configurational approach, similar to current compuational modelling.
• Focuses on the complex sets of conditions and wider context in which the model is situated.
• Related, while being country or population or region specific, it is taking into account the wider global picture.
• Is aware that complex social systems are comprised of humans, who react and learn in response to change, including public health interventions, and so culture, politics, economics, and collective psychology must be taken into account, along with likelihood of people adhering to the things being asked of them.
• understands the role of human and social interaction, including interaction with wider socio-ecological systems in which people and the rest of life on planet earth live.
• Searches for diferences -- differences in groups, populations, trends, trajectories, reactions, and outcomes.
• Recognises that the intersecting complex systems in which COVID-19 is playing itself out are nexus issues and wicked problems that bring with them a host of knock-on effects.
• Makes it clear that the purpose of modelling is less about prediction and more about helping people learn and explore. In other words, modelling is a powerful learning tool.
• Still predictions are helpful and very important within certain parameters and as long as they are done soberly.
• Demonstrates multiple and different possible scenarious and outcomes, many of which will not actually happen, but which allows users to understand the COVID-19 pandemic in more nuanced and sophisticated ways.
• Related, they allow for the exploration of counterfactuals -- which could prove the model to be wrong or partially wrong, or wrong for different groups, or require further development.
  
• Evaluates public health policies, strategies and interventions in a spirit of democratic co-production and participatory research, including the citizens it is meant to serve.
• Makes clear that its purpose is not necessarily just about identifying some underlying causal model, as much as it is about helping users explore and learn how various interventions or strategies might play out for a given COVID-19 policy and the larger complex system in which it is situated.
• Is able to adjust to new information, new data, new criticisms, and alternative viewpoints.
• Is designed, run and understood as one of an ansemble of other forms of scientific evaluation and modelling. In other words, all models are underdetermined by their evidence; all models are in some way wrong, but that does not mean they are not useful.

I am sure there are other points that my fellow modellers would add. And perhaps they can in the comments section below. But I think, overall, you get the basic points! 


So, how do our two key models work and what do they tell us?
Now that we have a good sense of the challenges of modelling (POST 1) and also a good sense of what an effective model of COVID-19 looks like, it is time, finally, to review two of the incredible models getting the most attention. The first is the simulation model by Ferguson and colleagues at Imperial College London; the second is complex network model by Alessandro Vespignani and colleagues at Northeastern University in the States. 

This is the focus of my third blog post!