This post extends a conversation I began on 19 May 2010, titled testing the validity of complex systems. This is the fifth post on this topic since then.
The argument I am making is that researchers need to do some sort of complete (holistic) test of their topic, to: (1) make sure that the definition of a complex system they are using applies; and(2) make sure that their topic fits this definition.
The question I want to address here is how should such holistic testing be done?
Again, this will take a bit of blogging, but it seems to me that testing can be thought of at two basic levels.
1. Deep/Thorough Testing: The first and most rigorous level would require one or more studies devoted to a sort of deep or thorough testing to determine if one's definition of a complex system applies to a give topic and, related, if that topic can be validly and reliably called a complex system.
This first type of testing is the focus of the community health science study I am doing with my colleague, Galen Buckwlater. For the last couple years, researchers have been explicitly or implicitly treating communities and their health as if these things are complex social systems. Our research question is: is such an assumption valid and reliable? In other words, can one assume that the commonly used definition of a complex system applies to the study of communities and their health and, conversely, can communities and their health be called a complex system?
To conduct this type of test, we did the following. (A) First, we reviewed the literature to determine what the common definition of a complex system is that researchers use. (B) Next, we found a case study that represented the average community researchers typically study and collected data on it. (C) Then, we took each descriptor from the common definition of health and ran a series of tests. For example, a commonly held assumption is that communities are self-organizing. To determine if this is true, we examined if the conception of self-organizing used by these researchers to determine exactly what they mean by this concept. Then, we empirically tested this concept of self-organization to see if our community actually engaged in this behavior. In total, we ran ten individual testsn on the commonly used definition of complex system used in the community health science literature. It was a tremendous amount of work. And, in the process we used a wide arsenal of techniques, including hierarchical regression, curvilinear regression, correlation, k-means cluter analysis, the self-organizing map neural net algorithm, network analysis, qualitative case-based comparative method and computational (agent-based) modeling.
One can think of this first type of testing as helping a field along by increasing the rigor of its concepts and its knowledge of the type of complex system it it studying.
2. Shallow/Preliminary Testing. The second type of testing is what we might expect all researchers to do before and during the process of modeling a particular topic as a complex system. In this case, one would begin by explicitly outlining the particular definition of a complex system one is using. Then, one would conduct some type of preliminary tests to determine if one's topic is, indeed, a complex system.
The testing in this second case is likewise rigorous but it is more background work. Also, it is something that takes place before and during the model building process. The quality of one's results is something that is reported in the methods section of a study.
I have used this type of testing in a couple studies we have done. The first one was my research with Fred Hafferty on medical professionalism and the second was the book on sociology and complexity science that I wrote with Fred as well. In both instances we articulated the definition of a complex system we were using and tested to see if our topic fit it reasonably well.
This second type of testing involves the development of what we call a meta-model, and it is one of the first steps in the SACS Toolkit modeling process--this is the new method Fred and I developed for studying complex systems. SACS stands for sociology and complexity science. For more about our method, see our BOOK
Developing a meta-model (a model of one's model) allows researchers to determine, right from the beginning, if their definition of a complex system is rigorous and if their topic is (empirically speaking) a complex system. In addition to the development of a meta-model, the SACS Toolkit has a total of nine built-in procedures that researchers are expected to use to explore their definition and topic in complex systems terms. My brother John and I are writing a paper on how the SACS Toolkit does this and will be presenting it this summer in Sweden at the International Sociological Association Meetings. I should be done with the paper in the next couple weeks and will post it on here. I also plan to blog more about the SACS Toolkit so that readers can get a better sense of the method.
08/06/2010
Complexity Definitions Need to Best Tested as a Whole
This post extends a conversation I began on 19 May 2010, titled testing the validity of complex systems. This is the fourth post on this topic since then.
Okay, I am getting a bit closer to what I am trying to say about testing. When I say definitions needs to be empirically grounded and tested I mean that the entire definition, as a whole, needs to be empirically grounded and tested. To date, most empirical inquiry in the complexity sciences focuses on parts of the complexity science definition. Researchers study networks or they study dynamics or they study emergence, autopoiesis, self-organization (a.k.a swarm behavior) and so forth. Two things are held as true in these studies. First, that the things being studied are actually complex systems. Second, that the part of the complexity science definition the researcher is studying naturally integrates into the larger complex systems scheme of things. My questions is, how do you know both of these things are true about the topic one is studying?
One way I think researchers can be sure is to do a complete (holistic) test of their topic, (1) to make sure that the definition of a complex system they are using applies and (2) to make sure that their topic fits this definition. For example, if researchers assume that a complex system is self-organizing, emergent, comprised of a large network of interacting agents and open-ended, then these researchers should have a series of tests to validate if this definition (in its entirety) applies to the topic they are studying. Alternatively, such a complete set of tests makes sure that the topic these researchers are studying is actually a complex system, or at least the type of complex system they seek to study.
Okay, I am getting a bit closer to what I am trying to say about testing. When I say definitions needs to be empirically grounded and tested I mean that the entire definition, as a whole, needs to be empirically grounded and tested. To date, most empirical inquiry in the complexity sciences focuses on parts of the complexity science definition. Researchers study networks or they study dynamics or they study emergence, autopoiesis, self-organization (a.k.a swarm behavior) and so forth. Two things are held as true in these studies. First, that the things being studied are actually complex systems. Second, that the part of the complexity science definition the researcher is studying naturally integrates into the larger complex systems scheme of things. My questions is, how do you know both of these things are true about the topic one is studying?
One way I think researchers can be sure is to do a complete (holistic) test of their topic, (1) to make sure that the definition of a complex system they are using applies and (2) to make sure that their topic fits this definition. For example, if researchers assume that a complex system is self-organizing, emergent, comprised of a large network of interacting agents and open-ended, then these researchers should have a series of tests to validate if this definition (in its entirety) applies to the topic they are studying. Alternatively, such a complete set of tests makes sure that the topic these researchers are studying is actually a complex system, or at least the type of complex system they seek to study.
Operationalizing metaphor
This post extends a conversation I began on 19 May 2010, titled testing the validity of complex systems.
I my last two posts I've argued that one should have a way to determine empirically if the topic one is studying is actually a complex system. Related, I've argued that the definitions complexity scientists use to identify a topic as a complex system should likewise be empirically grounded and tested. In this post, I want to comment further why I think doing such things is important.
Two words: operationalizing metaphor. I have read far too many articles and books in the last couple years that are little more than undisciplined, metaphorical labyrinths verging on the same sort of nonsense that took place at the high point of the postmodern movement in the 1990s. I've read articles talking about turning one's business firm or one's educational system into a self-organizing, emergent, agent-based network in order to optimize profits or learning, as if one could make a social system self-organize. Is that not contradictory? How does one make a system self-organize, given that a self-organizing system is one where there is no guiding external force controlling the systems's organization? Or, how about pushing one's business to the edge of chaos in order to profit from its nonlinear dynamics? What does something like this mean? Do these writers really understand what nonlinear (which, last I looked is a mathematical term) means? Related, what is nonlinear management? Or, how about talking about any and all social change as if they were the product of tipping points? When I hear such discussions I am reminded of the first time I heard a politician talk about "deconstructing" some political process to get to the bottom of things. Worse, when I hear such complexity science nonsense, I fear the next Sokal Hoax. Remember how the physicist, Alan Sokal, submitted his completely nonsensical postmodern text to the periodical, Social Text, and got it accepted, only to reveal later that the entire text was garbage. Sokal's hoax was done with complete seriousness. He was not trying to say that postmodernism was useless. Instead, he felt that postmodernism had some important things to offer, but only by increasing its rigor. I'm not saying that some of the complexity science literature has reached this point. But, it is close. If complexity science is going to make important inroads into mainstreet science, many of its new practitioners need to be more empirically rigorous and discerning in the definitions they use and the topics they call complex systems.
I my last two posts I've argued that one should have a way to determine empirically if the topic one is studying is actually a complex system. Related, I've argued that the definitions complexity scientists use to identify a topic as a complex system should likewise be empirically grounded and tested. In this post, I want to comment further why I think doing such things is important.
Two words: operationalizing metaphor. I have read far too many articles and books in the last couple years that are little more than undisciplined, metaphorical labyrinths verging on the same sort of nonsense that took place at the high point of the postmodern movement in the 1990s. I've read articles talking about turning one's business firm or one's educational system into a self-organizing, emergent, agent-based network in order to optimize profits or learning, as if one could make a social system self-organize. Is that not contradictory? How does one make a system self-organize, given that a self-organizing system is one where there is no guiding external force controlling the systems's organization? Or, how about pushing one's business to the edge of chaos in order to profit from its nonlinear dynamics? What does something like this mean? Do these writers really understand what nonlinear (which, last I looked is a mathematical term) means? Related, what is nonlinear management? Or, how about talking about any and all social change as if they were the product of tipping points? When I hear such discussions I am reminded of the first time I heard a politician talk about "deconstructing" some political process to get to the bottom of things. Worse, when I hear such complexity science nonsense, I fear the next Sokal Hoax. Remember how the physicist, Alan Sokal, submitted his completely nonsensical postmodern text to the periodical, Social Text, and got it accepted, only to reveal later that the entire text was garbage. Sokal's hoax was done with complete seriousness. He was not trying to say that postmodernism was useless. Instead, he felt that postmodernism had some important things to offer, but only by increasing its rigor. I'm not saying that some of the complexity science literature has reached this point. But, it is close. If complexity science is going to make important inroads into mainstreet science, many of its new practitioners need to be more empirically rigorous and discerning in the definitions they use and the topics they call complex systems.
Is what you are studying a complex system?
This post extends a conversation I began on 19 May 2010, titled testing the validity of complex systems.
My basic argument is that we simply too often assume that any topic we are studying is a complex system simply because we say so--regardless of the definition we are using.
Now I know that the definition of a complex system is encyclopedic, such that many definitions exist. And, of course, I am not arguing for a single standard by which all topics should be judged worthy of being called a complex system.
But, I am arguing that, regardless of the definition researchers use, they should have some way of testing their topic to see if and how it acts like a complex system.
For example, pretend one assumes that complex systems have the following characteristcs: they are self-organizing, emergent, operating near chaos, and agent-based. Definition in hand, one then goes out to study a local community, a formal organization or some social network. Before one begins, however, shouldn't there be some set of preliminary tests done; some sort of way to determine if what one is studying is actually self-organizing, emergent, etc? Related, what would one look for to determine if such characteristics exist? What tests would one use? What methods would be relevant to conduct these tests? And, what if one finds that one or more of these characteristics is lacking, or only exists in a modified form? What then?
Again, I am not saying that one test or definition fits all. But, I am saying that the definitions complexity scientst use to identify, model and study various topics as complex systems should have a bit more empirical rior. These definitions should be tested and held up to empirical validity and reliability. One should be able to talk intelligently about what one means when one is calling something a complex system.
My basic argument is that we simply too often assume that any topic we are studying is a complex system simply because we say so--regardless of the definition we are using.
Now I know that the definition of a complex system is encyclopedic, such that many definitions exist. And, of course, I am not arguing for a single standard by which all topics should be judged worthy of being called a complex system.
But, I am arguing that, regardless of the definition researchers use, they should have some way of testing their topic to see if and how it acts like a complex system.
For example, pretend one assumes that complex systems have the following characteristcs: they are self-organizing, emergent, operating near chaos, and agent-based. Definition in hand, one then goes out to study a local community, a formal organization or some social network. Before one begins, however, shouldn't there be some set of preliminary tests done; some sort of way to determine if what one is studying is actually self-organizing, emergent, etc? Related, what would one look for to determine if such characteristics exist? What tests would one use? What methods would be relevant to conduct these tests? And, what if one finds that one or more of these characteristics is lacking, or only exists in a modified form? What then?
Again, I am not saying that one test or definition fits all. But, I am saying that the definitions complexity scientst use to identify, model and study various topics as complex systems should have a bit more empirical rior. These definitions should be tested and held up to empirical validity and reliability. One should be able to talk intelligently about what one means when one is calling something a complex system.
Subscribe to:
Posts (Atom)