20/11/2017

Dynamic Pattern Synthesis for Modeling Complex Systems. An Interview with Phil Haynes

The following interview was conducted with Phil Haynes

He is Professor of Public Policy and researches and teaches public policy and management, as applied to a variety of contemporary circumstances. His research focuses on the application of complex systems theory, often using applied statistical methods. His research has been funded by the ESRC and the government and voluntary sector. He has published in a wider variety of journals including Social Policy and Administration and Public Management Review.  He is author of several books including Managing Complexity in the Public Services (2015) now in its second edition.

His most recent book, which is part of our complexity in social sciences series at Routledge, is aptly titled, SOCIAL SYNTHESIS: Finding Dynamic Patterns in Complex Social Systems.

BOOK DESCRIPTION:


How is it possible to understand society and the problems it faces? What sense can be made of the behaviour of markets and government interventions? How can citizens understand the course that their lives take and the opportunities available to them?  There has been much debate surrounding what methodology and methods are appropriate for social science research. In a larger sense, there have been differences in quantitative and qualitative approaches and some attempts to combine them. In addition, there have also been questions of the influence of competing values on all social activities versus the need to find an objective understanding. Thus, this aptly named volume strives to develop new methods through the practice of ‘social synthesis’, describing a methodology that perceives societies and economies as manifestations of highly dynamic, interactive and emergent complex systems. Furthermore, helping us to understand that an analysis of parts alone does not always lead to an informed understanding, Haynes presents to the contemporary researcher an original tool called Dynamic Pattern Synthesis (DPS) – a rigorous method that informs us about how specific complex social and economic systems adapt over time.  A timely and significant monograph, Social Synthesis will appeal to advanced undergraduate and postgraduate students, research professionals and academic researchers informed by sociology, economics, politics, public policy, social policy and social psychology.


--------------------

-->
Thanks, Professor Haynes, for doing this interview…

1. To begin, can you tell us a bit about your academic background? More specifically, how did you end up in policy evaluation and applied social science?

HAYNES: My first degree was in combined social sciences and social work. Over four years it provided a great interdisciplinary foundation. The last two years increasingly focused on social work practice.

It was a fantastic four years. When I graduated, I got a job as a generic court probation officer and then later specialised in developing new services for substance misuse. At that point, I started to get involved in research and training. 

All the new substance treatment programmes had to have evaluation built into them. It was immediately apparent that evaluation was complex and did not easily provide straightforward answers. For example, for the most dependent substance misusers, it was very difficult to estimate which service users would do best with different treatment types. I really enjoyed the research challenge and enrolled for an MSc in advanced social research methods at the UK Open University.


2. What got you involved in the development of methods?

HAYNES: After completing my MSc, I started a PhD examining how to use mixed methods to plan social services. My PhD soon started to show up the severe limitations of using traditional statistical methods for modelling historical patterns in order to plan future services. This took me into complexity theory. I moved permanently into an academic post. This was in the 1990s.

A number of seminal pieces about the application of complexity theory to the social sciences were published at that time in the US, and just beginning to influence Europe.  I was fortunate to have David Byrne as my PhD examiner and he was publishing his important book in the UK, Complexity theory and the social sciences. The late Paul Cilliers monograph, Complexity and postmodernism came out at a similar time.  

After that, David’s approach encouraged me to try methods like cluster analysis and then QCA. This resulted in me succeeding in getting ESRC funding to apply these methods to comparing the social networks of older people alongside different government expenditure patterns. It was a comparative study across several countries. Cluster analysis and QCA allowed the study to demonstrate that there were different patterns within the data and not one aggregate pattern. For example, Scandinavian, Northern Europe, and Southern Europe all demonstrated their own separate patterns, but also with dynamic and evolving changes over time.

In more recent years, I got frustrated with the competing strengths and weaknesses of cluster analysis and QCA and trying to decide which was the best method to use in a given research situation. It then occurred to me, the answer was staring me in the face, to bring them together into a mixed method. Then you could get the best characteristics of each method counter balancing the weakness in the other. That is how Dynamic Pattern Synthesis (DPS) was born.


3. Can you provide us an overview of what you mean by social synthesis? For example, why is social synthesis so important for social science?

HAYNES: Social synthesis is the art of examining social issues and social practices through a more holistic lens rather than a narrow hypothesis. It is founded on the idea from complexity theory that cases and social phenomena are often dynamic and highly interactive with each other. It is closely related to systems theory in this respect. Therefore, experimental and quasi-experimental approaches are extremely difficult to design with regard to knowing what to include and what is left out. Of course, experimental methods can work with replication and incremental adjustments, but that is resource and time intensive and not necessarily the best starting research design. This made me favour initial explorative approaches to large datasets, like using cluster analysis.

There are still limitations. Social synthesis cannot be a ‘theory of everything’, it has to have modelling boundaries, but it starts with the premise that is best to look more broadly rather than to focus its measurements too quickly and too soon into a reduced area of coverage.


4. What is your method Dynamic Pattern Synthesis (DPS) about, relative to this issue of synthesis?  For example, how do you see it as an advance on case-comparative method?

HAYNES: Dynamic Pattern Synthesis starts with an explorative synthesis rather than an explanatory hypothesis (although the latter can be introduced later in the method via QCA, if appropriate). It keeps the focus on being able to identify and compare each case rather than getting aggregate measures that are supposed to represent large groups of cases. It is very much a case based method, but one that tries to maximise the variable evidence for why a case is located where it is.


5. Is there any link to critical realism?

HAYNES: I think the contextual aspect of critical realism is highly relevant. When using critical realism, generative mechanisms and causality are situated in a changing social context. This frames and restricts any attempts at generalisation. It is a realistic and partial perspective on causality.


6. The case studies in your book are excellent.  I found them very useful because of their depth and variety, which helped me to see how your method works in different instances.  How did you happen to choose those case studies?

HAYNES: Because of the pressures of time and resources, my approach to the case studies was pragmatic and based on my previous research with secondary data. I had been involved in some research looking at the relationship of economics with public policy, post the 2008 financial crisis, so the Euro case study emerged from that stream of work. I also have a history of using secondary data to understand the changing demography and care needs of older people.  Similarly, I have focused previously on issues of territorial justice and the differences between local governments.

Probably the most innovative and speculative case study for me was trying to see if DPS made any sense with a small sub sample of micro data about older people. I think it is interesting how the resulting issues are very similar to challenges in qualitative research. It is hard to find meaningful consistent patterns over time at the most micro level. Social patterns seem easier to identify and work with at scale, at the meso and macro level, and that fits with the application to policy studies and evaluating policy at governmental levels.


7. What are the one or two most important things you want readers to come away with reading your book?

HAYNES: I would really like other researchers to try out DPS and to see how it works with different data sets in different contexts. I would also like to see this kind of method taken up in heterodox economics/political economics to reach a better understanding about macroeconomic theory and future interventions in the post financial crisis world. I think there is currently a normative imperative to be adventurous with macroeconomic research, to look for new public policy interventions in the economy.


8. What is the next step in your development of DPS?

HAYNES: I really want to communicate the basics of how the method works and to share the mechanics of this, and to encourage more case studies and more use, and to get other academics to ‘add-on’ to the mix of methods used in DPS. The methodological purpose is clear, to identify case patterns (that are likely to be time and space limited) and what the socio-economic meaning of these patterns is. DPS is not the only way to identify and name these patterns, there will be future evolutions of DPS as a method and better alternatives -  I am sure.  I would also really like to see if I could find and persuade collaborators to attempt to develop R packages in DPS. I do not have the skills and time to do many of these things alone, so I need to network and collaborate.


-->

17/11/2017

COMPLEX-IT A new App for Policy Evaluation at the Nexus


16/10/2017

Public Health is a Complex Systems Problem. When will we finally embrace this fact?


My point is simple enough.  Consider the difference in the following two research questions:
1. How do we help people addicted to opioids overcome their problem?
2. Versus, how do we fix the health systems in which people live so they are less likely to become addicted to opioids?
Or how about this research question?
1. How do we help poor people deal with their health vulnerabilities?
2. Versus, how do we fix the communities and systems in which people live so that poverty is not a vulnerability to their health?
In neither case is the difference between these research questions one of psychology versus sociology.  Instead, it is a difference between a reductionist perspective and a complex systems view. The difference is also a matter of method: conventional variable-focused statistics versus computational methods focused on systems and cases and intersectionality.  Public policy and community health and clinical care research needs to change -- as do the views of people in general.  The world is too complex to keep thinking the way most continue to do.

As a primer, read the following book we wrote in 2015.



See also Battle-Fisher's excellent book:



See also the work being done at CECAN on complex nexus issues, which takes the issue of complexity to another level, at which point one is confronted with how the complexities of one public policy issue (and the changes made to address it) intersect and impact other policy issues and vice versa.
 


And, for an equally exhaustive and wider read on this view as concerns health and healthcare, go to the New England Complex Systems Institute and read through the excellent work by Bar-Yam and colleagues.    













04/10/2017

Wonderful 2 minute Video on Our Globally Interconnected Web of Life

Saw this on twitter and was so impressed with it.  Just a wonderful example of our ecological interdependence and Capra's constant call for us to acknowledge our complex global web of life.

20/08/2017

Scientific Journals, for the love of all that is good ... please grasp that editorial style is not science!

How many times has this happened?  You spend a tremendous amount of time on a study -- designing it, collecting data, conducting your results, writing up your paper, re-examining your results, making sure you've got everything right, getting all of your collaborators to agree the study is finally ready to go -- only to get to the "pick a journal" stage and come to a screeching halt!

And why?  Is it because you cannot find the right journal?  NO!  Is it because your ideas are not sufficiently cutting-edge?  NO!  Is it because your study is no good?  NO!  It's because your study is formatted incorrectly!  You chose the wrong font; your abstract is ten words too long; you need to use some arcane heading series that went extinct a thousand years ago; or you used APA when you should have used some bizarre hybrid referencing system that only the journal you want to publish in uses.   THAT IS WHY.  

So, you do what is asked, you convert everything to the exact style required -- none of which has anything to do with intellectual content whatsoever -- and then you submit the article; only to get an email back a few days later saying, "SORRY WE COULD NOT MOVE YOUR ARTICLE TO THE REVIEW STAGE BECAUSE YOU USED A SOFT RETURN AT THE END OF YOUR PARAGRAPHS."

What?!

Please Journal Editors and Publishers, stop the ridiculousness and do what your more enlightened colleagues do.  Here for example is the GUIDE FOR AUTHORS of a well-known journal with a high impact rating:
Your Paper Your Way
We now differentiate between the requirements for new and revised submissions. You may choose to submit your manuscript as a single Word or PDF file to be used in the refereeing process. Only when your paper is at the revision stage, will you be requested to put your paper in to a 'correct format' for acceptance and provide the items required for the publication of your article.

Does that not make sense?

Now, do not get me wrong.  I am actively involved in the field of visual complexity and take the issue of writing and presenting my work very seriously -- in fact, I probably spend too much time on it. Such concerns are different, however, from demanding that each and every time we submit an article we have to entirely reformat and sometime significantly rewrite it to meet the idiosyncratic needs of editorial style!  Please, science is hard enough without such nonsense.

Let people submit their studies, keep the formatting to the basics, and stick to the intellectual argument and the science.  And, if the paper is worth publishing, then run us through the hoops of meeting your arcane table formatting requirements.  But, until then....

On a humorous note, I leave you with this brilliant cartoon in The New Yorker by Tom Cheney (Published February 8th, 2016)






 









04/07/2017

JULY CECAN WORKSHOP: COMPLEX-IT and the SACS TOOLKIT: A Case-Based Computational Modeling Platform for Data Mining Complex Issues in Policy and Evaluation


 CECAN Training Workshop

 

 

 

COMPLEX-IT and the SACS TOOLKIT: A Case-Based Computational Modeling Platform for Data Mining Complex Issues in Policy and Evaluation


When: – Friday 7th July 2017 (1 day)

Location: – University of Surrey, Guildford, UK

Purpose: The complex socio-technical arenas (nexus issues) that government seeks to improve (e.g., health, food, water, safety, infrastructure) are not driven by a single factor or consequence.  Instead, they are driven by multiple factors at multiple levels, which lead to different trends or outcomes for different areas/groups of people.
The challenge is how to model such diversity and complexity?  The complexity sciences, data mining and big-data offer some useful solutions.  The challenge, however, is stitching these methodological solutions together into a user-friendly platform and APP, which policy makers, social scientists, evaluation commissioners and civil servants can use – hence our creation of COMPLEX-IT and the SACS TOOLKIT.

Intended audience:  This workshop is for anyone involved in evaluating the impact of policy (and its improvement) on complex nexus issues and would like to explore new software and mixed-methods options for doing so.

Level of prior knowledge of subject required:  For policy makers and evaluation commissioners, it is helpful to have a basic sense of statistics and an interest in data mining and the complexity sciences.  For researchers and methodologists, it is helpful to have an understanding of the latest developments in interdisciplinary mixed-methods, computational modeling and data-mining big-data.
Participants are strongly encouraged to bring to the workshop a policy issue or research concern (e.g., modeling multiple trajectories across time, dealing with large numbers of variables, etc) that they would like to use COMPLEX-IT and the SACSTOOLKIT to explore.

At the end of this course, participants will:

GOAL 1: Understand the theory behind case-based computational modeling, including
  • Having a basic sense of the principles guiding case-based complexity.
  • Understanding the philosophy behind data mining and computational modeling.
  • Developing a working knowledge of COMPLEX-IT APP and SACS TOOLKIT.
GOAL 2: Learn how to apply case-based computational modeling to their nexus topic, including how to:
  • Build a complex systems model of their nexus issue.
  • Explore how policy impacts different groups or areas across time/space.
  • Use this information to create your study’s case-based profile.
  • Identify major and minor case-based clusters and key causal factors.
  • Identify major and minor cluster trends (for longitudinal data).
  • Identify key global-temporal dynamics, such as spiraling sources and saddle points.
  • Use network analysis (where appropriate) to explore cluster links and structure.
  • Examine how different clusters and trends lead to different outcomes.
  • Run simulations to explore how policy can change outcomes.
  • Compare resulting model to original theoretical formulation.
GOAL 3: Learn how to use the COMPLEX-IT APP, including how to:
  • Download and install the software.
  • Run the software, including the R Studio environment in which it works.
  • Upload the case study database.
  • Identify key variables for case-based profiles.
  • Explore how to deal with missing data and errors in variable choice.
  • Use k-means cluster analysis to identify initial clusters, including how to identify. optimal solutions and run k-means for trend data.
  • Use the SOM neural net to corroborate clusters and identify possible sub-clusters.
  • Use SOM and k-means to identify underlying causal model.
----------------------------------
COMPLEX-IT
GitHub - Cschimpf/Complex-It: Complex-It Development

LINK TO R STUDIO
Click here to download R Studio

COMPLEX-IT AND AGENT-BASED MODELING
RNetLogo - ...and two worlds are yours
R Marries NetLogo: Introduction to the RNetLogo Package | Thiele | Journal of Statistical Software
GitHub - NetLogo/Mathematica-Link: allows Mathematica to control NetLogo (and not vice versa)
Georg-August-Universität Göttingen - Agent-based/individual-based simulation tools
CRAN - Package RNetLogo
CRAN - Package gafit
CRAN - Package GA
How to load the {rJava} package after the error "JAVA_HOME cannot be determined from the Registry" | R-statistics blog
Agent Based Models and RNetLogo | R-bloggers
Facilitating Parameter Estimation and Sensitivity Analysis of Agent-Based Models
NetLogo User Community Models: SegregationExtended
SELF-ORGANIZING MAP SUPPORTING MATERIALS:
Blog overview of SOMbrero R Package
Stochastic Gradient Descent
Wikipedia Article on Stochastic Gradient Descent
The art of running the SOM and choosing the map size
Discussion about setting the random seed for reproducible results
  
WALES DATABASES (INDEX OF MULTIPLE DEPRIVATION)
This is the main one we used for our Workshop.

Welsh Multiple Index of Deprivation

Another data website source.

Full dataset, which is a very useful case study for exploring how to use the COMPLEX-IT App
(The entire dataset is shown for all authority areas, across multiple indicators)

This provides multiple databases for multiple levels of analysis

This goes with the above, as it is their annual data release website for IMD

MAP OF WALES BY AUTHORITY AREA FOR DEPRIVATION

LSOA MAPS for Wales (Lower Super Output Areas, roughly N=1,500 people each) for a total of about 1,896 LOSAs in Wales http://gov.wales/docs/statistics/lsoamaps/lsoa.htm

NOTE: Area maps for each LSOA in Wales are available from the links below. LSOA is the geographic unit used in the Welsh Index of Multiple Deprivation (WIMD). LSOAs are built from groups of Output Areas (OAs) used for the 2001 Census. There are 1,896 LSOAs in Wales each with a population of about 1,500 people. Because the size and boundaries of LSOAs have not changed since they were created in 2004, the same areas are analysed in the three recent WIMD updates (WIMD 2005, WIMD 2008 and WIMD 2008: Child Index). The maps can be used alongside each of the three updates to identify the area covered by each LSOA.


BIOGRAPHICAL BACKGROUND:
Brian Castellani, Ph.D. is Professor of Sociology and Lead of the Complexity in Health and Infrastructure Group at Kent State University, as well as Adjunct Professor of Psychiatry, Northeast Ohio Medical University and co-editor of the Complexity in Social Science series, Routledge.  Trained as a sociologist, clinical psychologist and methodologist, Brian has spent the past ten years developing a new case-based data mining approach to modeling complex social systems, which he and his colleagues have used to help practitioners and policy makers address and improve complex public health issues such as community wellbeing, stress and coping (allostatic load), comorbid depression in primary care, addiction, medical education and grid reliability. Recently, Brian received a systems science scholarship from the Robert Wood Johnson Foundation to present at the 2016 AcademyHealth Conference – the leading organization in the States for health services researchers, policymakers, and health care practitioners and stakeholders. For more information, including publications on case-based complexity, see Brian’s website at www.personal.kent.edu/~bcastel3/

Corey Schimpf, Ph.D. is a Learning Analytics Scientist at the Concord Consortium, a not-for-profit company that develops curriculum and software for K-12 science, technology, engineering and math learning, just outside of Boston.  He received a Ph.D. in Engineering Education and a M.A. in Sociology from Purdue University and has several years of programming and software development experience. One avenue of Corey’s work focuses on the development and analysis of learning analytics that model students’ cognitive states or strategies from fine-grained computer-logged data from students participating in open-ended technology-centered science and engineering projects. I n another avenue of Corey’s work, he has been the lead or team member developing software to assist researchers dealing with complex, high dimensional problems and data-sets, such as an interface and infrastructure to integrate several methodological tools or a multi-purpose data processing tools for high volume data with limited structure.




16/06/2017

GROWING INEQUALITY: Bridging Complex Systems, Population Health, and Health Disparities (A BOOK REVIEW)


A new book has been published by George Kaplan and colleagues, titled, appropriately enough, GROWING INEQUALITY: Bridging Complex Systems, Population Health, and Health Disparities

The edited book is the result of a handful of years that Kaplan and a working group of interdisciplinary colleagues spent struggling to figure out how to more effectively think about, model, and address growing health inequalities in the States, Canada and, by extension, the world.

The conclusion was that such issues are best viewed as complex systems problems and therefore best modeled in complex systems terms.  So far, so good.

The challenge, however, was how to proceed from there, which allowed the working group to happily descend into the chaos of real interdisciplinary work -- which is not easy by any stretch of the imagination.  Such work requires creating a shared vocabulary, embracing very different ways of thinking about research problems and their solution, and realizing that people often use the same scientific terms (such as non-linearity, for example) in rather different ways, and so forth.  Then there was the second issue of how to define, think about, model and manage this thing called 'complexity' or, more specifically, a 'complex system.'

On the first of these two issues the hard work by Kaplan and colleagues is to be, overall, commended.  In terms of the group's outcome, the book is organized into fourteen chapters -- with the acknowledgement and first and last chapters functioning as meta-reflections on the work the group did over its roughly five years of existence.  The second chapter constitutes a reflection of method, addressing the issue of complexity and multi-agent simulation.  The other twelve chapters focus on various public health issues, from improving health behaviors to the built environment to crime to health and socioeconomic well-being.

As concerns the second issue, however -- that is, how to define and understand and model complexity -- the book seems to fall short, overall, in several important ways:

1. To begin, the definition of complexity embraced remains narrow.  Mainly, while differences among the authors seemed to exist (as in Stange and colleague's chapter, which embraced a wider view) the group mainly employed what the French systems scholar, Edgar Morin calls restricted complexity -- which is basically conventional science, albeit now focused on complex systems.  As Morin states, "The problem with restricted complexity is that it still remains within the epistemology of classical science. When one searches for the 'laws of complexity,' one still attaches complexity as a kind of wagon behind the truth locomotive, that which produces laws. A hybrid was formed between the principles of traditional science and the advances towards its hereafter. Actually, one avoids the fundamental problem of complexity which is epistemological, cognitive, paradigmatic. To some extent, one recognizes complexity, but by decomplexifying it. In this way, the breach is opened, then one tries to clog it: the paradigm of classical science remains, only fissured."  (For more on this issue, see also Byrne and Callaghan's Complexity Theory and the Social Sciences or Jenks and Smith's Qualitative Complexity.)

Here is an easy way to think about the difference vis-a-vis the complexities of health: rather than think about the health vulnerabilities of poor people, we should think about the complex ways the systems in which they live make their poverty a vulnerability.  For example, in our recent book, Place and Health as Complex Systems, we examined how the poverty of the inner-city communities we examined had more to do with the suburban sprawl of affluent individuals moving into the suburbs (and all they take with them in terms of healthcare and its funding and access) than the poverty of the inner-city communities.  In other words, these inner-city communities are stuck in what complexity scientists call poverty (welfare) traps.

2. The second problem follows from the first: given their restricted definition of complexity, the working group primarily employs a reductive multi-agent simulation approach -- which struggles to deal with social structural issues and the macroscopic systems that serve poor and lower income individuals.  Nonetheless, some of the models (as in the chapter on crime and health) are very well done and do offer some useful insights.  Still, the focus, overall, is reductive microscopic modeling.

3. Also, related, as shown on the map of the complexity sciences, there is no wider usage made of complex network analysis, real-data-driven geospatial modeling, dynamical systems theory modeling, and various other computational modeling techniques.  Also, there is no critical discussion about what methods are useful and when -- for example, the chapter on simulation and big data does not address the significant criticism leveled at the issue. In contrast, as Byrne and Callaghan make clear in Complexity Theory and the Social Sciences, not all computational and complexity science methods are equal in their utility for social scientific inquiry.

For more on this issue, see Burrows and Savage's excellent article, After the crisis: Big Data and the methodological challenges of empirical sociology; as well as the SAGE journal, BIG DATA & SOCIETY.  On a positive note, however, Kaplan and colleagues do provide a very good overview in their concluding chapter of the need to ground simulation in real data -- which is a major argument of such journals as Journal for Artificial Societies and Social Simulation.   

4. Still, the final arguments Kaplan and colleagues make in their concluding chapter are not really new.  In fact, they have been argued extensively by others, none of which is really cited (See, for example, the Handbook of Systems and Complexity in Health) -- which leads to the final problem with the book: the insularity of its research and references.  Other than a few citations to the global field of the complexity sciences -- which has been rather highly involved in modeling health and healthcare in complex systems terms -- the references in the chapters were mostly limited to a small set of publications.  For example, in Chapter 2 on method, despite discussing the epistemological issues surrounding simulation, the only scholars cited are mostly those from the early days of systems modeling and cybernetics. The result is a somewhat biased and narrow view of the import of complexity for the fields of health and healthcare.


Still, despite these limitations, the main point of the book remains cutting-edge and clear: if we are to advance our ability to more effectively address the complex health inequalities that now exist on a global level -- and the myriad intersections they have with such global complexities as economy, politics, geography, ecology and culture -- it is imperative that public health scholars and the larger healthcare field (and those they serve) embrace a complex systems perspective.  Oh, and let us also not forget the importance of such an embrace of systems thinking by those civil servants, the world-over, who write the policies....  












    

25/05/2017

Advancing Shannon Entropy for Studying Diversity Or, the Benefits of Case-Based Complexity!

We just published our latest article advancing Shannon Entropy for the study of diversity in complex systems, using our case-based entropy (also read as case-based complexity) approach.





Here is the abstract.  FOR THE ENTIRE ARTICLE, CLICK HERE



24/02/2017

Addressing Complexity in Nexus Issues: A Case-Based Approach to Evaluation



I had the great opportunity to present the methodology on case-based complexity that my colleagues and I have been developing over the past several years to policy makers and researchers in London at the excellent new research and policy initiative, CECAN. 

The video and PDF below present a general audience introduction to the method as a whole.

1. Click here for the PDF of my POWERPOINT

2. Click here for the YOUTUBE VIDEO of my PRESENTATION

3. Click here for copies of the research papers we have published related to my presentation.