From: Barry Markovsky Newsgroups: sci.skeptic Subject: Maharishi Effect: New Critique Date: Wed, 25 Jan 1995 09:51:17 -0600 Organization: University of Iowa, Iowa City, IA, USA Lines: 167 Distribution: world Message-ID: NNTP-Posting-Host: blue.weeg.uiowa.edu Mime-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII I just discovered this newsgroup yesterday and spent most of the day reading the various TM-related exchanges that were posted over the last week or so. (Can anyone tell me what happens to older postings?) I am new to this, so let me know if I breach norms or otherwise offend. I know that some subscribers are sick of the exchanges on this topic, but I suspect that's because the focus often drifts toward the ad hominem and away from the "sci." mandate implied by the group's name. The following summarizes a new critique. It consists of around 8 screenfuls of text. I hope this doesn't constitute bandwith-hogging, but I think some readers will find this information valuable. A colleague and I have investigated some issues directly pertaining to the research on the so-called Maharishi Effect (ME). We focused on the 1983 Jerusalem study because its results were published in a legitimate journal--the Journal of Conflict Resolution (JCR)--in 1988. This is a specialty journal with a pretty good reputation serving a multi- disciplinary audience. (I polled 8 political scientists whose ratings averaged 6.8 on a 10-point prestige scale. The journal is probably better known in poli. sci. than any other field.) Notably, however, the journal's editors, referees and readers are not generally qualified to evaluate the physical theory--Maharishi Technology Unified Field Theory--that is the alleged basis of the ME predictions. The article was published there because, as you probably know already, (1) the theory purports to reduce conflicts and to have social, political, economic, as well as other implications, (2) various social indicators seemed to bear out those implications, and (3) reviewers could find no major flaws in the research design or statistical methods. To recap briefly, varying numbers of meditators showed up in Jerusalem for 61 days in Aug. and Sept. 1983. During and after that time researchers gathered data on quality of life, Lebanon war intensity, crime, auto accidents, and fires. Some of the measures were derived from subjective ratings of newspaper accounts. Researchers argue that collective meditation causes changes in a fundamental, unified physical field, and that those changes radiate into society and affect all aspects of society for the better. Analyses showed a statistically significant correspondence in the time series relating numbers of meditators to positive social effects. In 1990 JCR published two critiques, to which MIU authors responded. In my opinion, some issues were satisfactorily addressed, some were not. My colleague and I took a different tack and found problems in the theory, the research design, and the statistical analyses that were not identified previously, and further amplified issues that others touched upon. Here I'll briefly note the types of problems we discussed. I should mention that we submitted our analysis to JCR. It received four reviews, three of which constituted the most positive set of reviews I have ever received for any manuscript I've ever published, or any grant proposal I've ever had funded. The fourth was from the MIU researchers-- 15 pages single-spaced, the sort of rejoinder that JCR would have to print if they published our critique. The editor rejected our paper on two grounds: the journal did not want to publish the Maharishi theory in such an extended form, and the MIU researchers' response would prevent many readers from reaching a clear-cut judgment. We have since revised the paper, keeping much of the critical analysis but placing it within the context of some broader questions: When should heterodox theories and research be published? When should they be conferred scientific legitimacy? It is under review at a sociology journal, and an abbreviated version will be presented at the Annual Meetings of the American Sociology Association in Washington, D.C. (!) in August. Our paper sought to raise doubts about the conclusions of the Jerusalem study, first by pointing out critical ambiguities and logical gaps in the theory. If the theory manifests such problems, then empirical measures cannot be said to support it. Many terms in the theory are offered without any definition. This makes it virtually impossible for outsiders (and, possibly even insiders) to know what the terms mean, or how to go about operationalizing them. The theory is grounded in conjectures from quantum physics for which experimental evidence is not yet available, and is silent on the complex hierarchy of mechanisms that link the micro-level quantum phenomena to the macro levels of social, political, economic, and other phenomena. In short, terminology is obscure and logical integrity is poor. The essence of the researchers' response to these sorts of problems was that no theory is perfect, all language is ambiguous, and their theory is pretty darned well- constructed by social science standards. The problem, of course, is that most social scientific theories do not make paranormal claims. Even if the theory is semantically and logically incoherent, however, the correlation of meditation group size with social indicators is a phenomenon in need of explaining. Even though the Maharishi theory is flawed, if it is the only explanation for the phenomena, it would have to be accorded some merit. I don't know whether or not MIU researchers were aware of this, but if they wanted to choose a sophisticated statistical technique and research design that would proffer a Maharishi Effect when in fact no such effect actually existed, they probably could have done no better than to use a time-series analysis with an unconstrained independent variable. The crux is that (1) the failure to randomize daily meditation group sizes, (2) the failure to control for some obvious historical events, and (3) the openness of their time-series methods to ex post facto explorations, all combine to render the study's conclusions highly provisional at best. Moreover, every other ME study I know of using social indicators uses similar methods that are subject to the same sorts of problems. Regarding (1), the failure to randomize, the authors claim that ethical considerations prevent them from imposing randomization. Artificially reducing the meditation group size on a given day would cause human suffering, they claim. Convenient. There are many ways to respond to this, beyond the obvious one that it assumes as true that which the research program seeks to demonstrate. One could say that performing even one true experiment with appropriate blinds and randomization would convince many skeptics and policy-makers to fund more research and standing meditation groups. In the long-run, more lives would be saved and suffering reduced by one good study than by 20 large, complex, but ill-designed projects. Alternatively, one could say that their ideology (religion, ethic, or whatever) has intruded in such a way as to severely weaken the scientific merits of their research, and to place their objectivity at peril. Regarding (2), historical events, the researchers statistically controlled for weekend effects, several Jewish holidays, and some other factors occurring during the 61 days of the study. They did not control for several other Jewish holidays, and a number of major political and military events that took place during the study period. The problem-- which the authors recognize--is that things like weekends and holidays can affect both the number of meditators that show up on a given day and the social indicators presumed to be affected by the meditators (quality of life, auto accidents, war intensity, fires, etc.). That produces spurious correlation. It would not have been a problem if the numbers of meditators were randomized. Since it was not, it is necessary to control for all variables that may produce the spurious correlates--a virtually impossible task. In response to our identifying the obvious variables (e.g., the other Jewish holidays), the MIU researchers said that it was up to us to demonstrate their relevance, not them. Unacceptable. They are the ones making the claim which, if true, would turn most physical, biological and social sciences on their heads. They are responsible for ruling out (as opposed to merely dismissing) mundane alternative explanations when they are offered. Regarding (3), the researchers filing their social indicators with an independent review board prior to the study is largely beside the point. Their report does not mention filing in advance crucial details of the statistical analysis, allowing for exploratory analysis to be portrayed as confirmatory. For instance, any positive lag effects (correlations between group size and social indicators that were delayed by one or more days) were taken as confirming the hypothesis. In fact, time-series methods are notorious for producing false positive results when a significant relationship at *any* lag is counted as supporting the hypothesis. There are other more technical issues that make time-series methods manipulable--which is why I would like to get those data and see how well-behaved they really are. I have been trying for nearly half a year to get the data from the MIU researchers, to no avail as yet. Any comments? Suggestions? As a whole, my field (sociology) tends to not be terribly concerned with theoretical rigor and non-relativist approaches to the sociology of science, so I can't be too optimistic about our paper being published in a major journal--but we're giving it a shot. Finally, for those who have not yet seen the report, the 1993 Washington D.C. study employed the same types of procedures and analyses as those in the Jerusalem study. ............................................................................ Barry Markovsky / Dept. of Sociology / U. of Iowa / Iowa City, IA 52242 Ph: 319.335.2490 Fax: 319.335.2509 E-MAIL: barry-markovsky@uiowa.edu ............................................................................