My favorite dictionary defines science as “The observation, identification, description, experimental investigation, and theoretical explanation of phenomena”. This seems like an excellent summary to me, especially because it makes no reference to “the” scientific method. An inquiry which incorporates all five of those elements is undoubtedly a science, but many forms of precise inquiry lack some of them. They key factor is experimentation. If there’s no experiment to rely on, then observations, identifications, descriptions and explanations must be built on a much weaker base.
Concepts like “knowledge” and ”truth” become very fuzzy when controlled experimentation is impossible, but that doesn’t necessarily imply that it would be impossible to distinguish between good and bad methods of investigation. It just requires a broader critical perspective on ceteris paribus (“all other things being equal”) conditions, the key element of controlled investigation. In order to observe and describe a limited part of reality precisely, all other things must be held equal.
In the experimental sciences that is achieved with an undisturbed measurement. In the social sciences (broadly construed: economics, sociology, anthropology, political science) there are different approaches toward satisfying the all-other-things-equal requirement. The first one is reliance on pure theory – deductive or otherwise formal models which seek to explain large and complex phenomena, such as the economy or society in general. Such theorizing has no direct ties to empirical study, so it solves the requirement by pure postulation. However, that trick places it closer to the speculative realm of philosophy than to the realm of verifiable science. At the level of pure theory, other things are equal by default.
An alternative approach is the one applied in anthropology, which emphasizes empirical study. Anthropology has a simple premise: instead of theorizing about social phenomena, we should just ask people about them. The anthropologist who visits a remote social group, learns their language, lives with them and comes to understand their life through direct experience can describe their society at a level of detail which no theoretical approach can ever reach. But since such descriptions do not go beyond immediate experience, they are not useful (and often impossible to achieve) in more complex social contexts. While anthropology can be a great starting point for criticizing all-other-things-equal assumptions, it does not yield any general assumptions of that kind. At the level of direct experience, other things are never equal.
But in between pure theory and direct experience there’s a realm of fuzzy social science – partly empirical, partly theoretical. In terms of observation, it is based on statistics. In terms of the identification of phenomena, it is based on reinforcing intuitive ideas, not on surprising discoveries. In terms of description, it is based on an everyday vocabulary where some concepts are defined with greater precision and others are not. In terms of explanation, the field is open for a great variety of entrants.
In this essay my aim is to show that there’s so much interpretive latitude in fuzzy social science that we have good reasons to be sceptical of some claims of exact knowledge. One can seldom decisively prove or explain anything in fuzzy social science. Chances are that most all-other-things-equal assumptions are based just as much on preconceptions and deliberately narrow vision as on rational inference. That doesn’t necessarily render the conclusions uninteresting, but it should make them more tentative.
I will concentrate on political science. It seems to be a field where overambitious goals have led to a particularly bad combination of haphazard research methods and excessive reliance on pseudo-quantification. This can have significant consequences for the way we interpret political explanations in general, as I indicate in the final section. Any references to simple cause-effect relationships are likely to be plain wrong.
Quantifying research
In the following I’m going to refer to a book called Doing research in political science by Paul Pennings and co-authors. The authors’ emphasize the fundamental importance of the comparative method in political science. In essence, the comparative method is what creates the all-other-things-being-equal condition. They outline the comparative method with these three methodological steps:
- Describe the core subject of comparative inquiry. In other words, formulate the question of what exactly is to be explained and how we recognize a need for comparison.
- Develop a view on the theoretical concepts that can ‘travel’ comparatively as well as measuring (sic) what is intended and possessing (sic) a unifying capacity for explaining political and social processes in general
- Discuss the logic of the comparative method as a means to a goal rather than as an end in itself. In other words, which instruments best fits the research questions to be answered by means of what type of research design? (PKK p.25)
Note the strong wording what is to be explained in step 1 and that step 2 essentially is a rephrasing of the all-other-things-being-equal condition. Taken together, steps 1-3 seem like a sensible program, provided that there really are theoretical concepts that can travel comparatively. The authors formulate the all-other-things-being-equal assumption slightly differently on two other occasions:
“In short, a theory of the political process must assume that there exists a mutual and interdependent relation between politics and society, but that its organization is to a large extent independent from society.” (PKK p.36)
“The social and economic configuration of a situation or society is not the primary goal or meaning of comparison; instead capturing the specific differentia of the ‘political’ across situations and across time will be our concern.” (PKK p.42)
In practice, this separation of “the political” from the rest of society means that most of political science focuses on comparing the political systems of sovereign, democratic nation states. Political systems without sovereignty – city governments, for instance – are dependent on the state and hardly suitable for meaningful comparisons. The political systems of undemocratic nation states, on the other hand, have a completely different relationship to the surrounding society and have to be excluded for that reason. So the underlying assumption in political science is that the system of elections, representation and government in democratic countries exhibits universal features which are independent of specific social contexts and thus amenable to scientific analysis.
What I’ve said so far makes political science a difficult enterprise, but not an impossible one. I certainly think that there’s lots to be said about democratic political systems and much to be learned from comparing them. But political science is currently a fundamentally misguided enterprise due to its emphasis on mathematical analysis. This emphasis is evident in Doing research in political science, which teaches students how to quantify different aspects of political systems and how to perform statistical analyses on the resulting data.
Instead of criticizing in detail the idea that political phenomena can be quantified, I think I can simply give a few representative examples of quantification from Doing research in political science to save space. Here they are:
- a market score which adds together: 1) favorable mentions of free-enterprise capitalism (in selected publications) and 2) the need for economic orthodoxy (evidenced by the budget deficit, for example) (PKK p.248)
- a score for government extremism, which is a calculation based on the proportional distribution of the opposition along the left-right dimension (PKK p.274)
- the method of manifesto analysis, which is based on counting and classifying sentences in party manifestos (PKK p.239-240)
Obviously, these quantifications all arise from a certain research context which I have not presented here, so judging them out of context is a bit unfair. Nevertheless, in my opinion they exemplify certain basic fallacies which the context certainly doesn’t correct. In the “market score” we have an example of adding together things that are not commensurable. ”Government extremism” uses calculation based on distributions along a completely subjective left-right measurement axis. And the simple activity of counting words in political pamphlets is just an entirely pointless exercise. I have to say that I feel sorry for the political scientist who believes that the number of words in a pamphlet reveals something objective about the political system.
In general, the quantified concepts presented in Doing research in political science bear little resemblance to sound statistics. There seems to be an awful lot of freedom in political science with regard to what is measured. Researchers can make up their own concepts as they go along, classifying, adding and estimating disparate things according to their subjective judgment. In other sciences, that kind of creative freedom would be curtailed by controlled measurement which separates the good ideas from the bad. But even in that department the venture of political science seems to be fundamentally flawed.
For a good cause
Quantification of spurious variables is only half the story. There wouldn’t be such a desperate urge for quantification in political science unless those numbers were used for something. And what they are used for is of course ”explanations” of political phenomena. I could elaborate at length about the notion of valid explanation in the experimental sciences, but no deeper analysis is necessary. The notion of explanation which the authors repeatedly invoke in Doing research in political science is in fact just the old correlation-causation fallacy: the fact that two variables correlate is taken to mean that one of them explains the other.
The authors of this book do seem to be aware of this fallacy. They even present a clear analysis which shows that correlation is not causation (PKK p.170-171) and state that
“It is an exception rather than the rule that conclusions with respect to theories in comparative political science can be based straightforwardly on statistical tests.” (PKK p.162)
Yet they just can’t seem to resist the temptation to write “explains” when they should write “correlates with”. In regression analysis the variance of one variable is said to ”explain 46%” of the variance in another. Now, I do know that the word “explain” is commonly used in statistical analysis in this manner and that it might just be a slip of the pen instead of a false inference. But the problem with political science is that its whole concept of explanation rests on this equivocation on the word “explain”. If we remove the fallacious interpretation and look for a real explanation, we are left with nothing. The authors actually make this point quite clearly:
“In this section we have examined processes of electoral change along two lines: the type and degree of variation and the explanation of these variations with the help of causal modeling. The two subsequent steps of finding and explaining variations are crucial to comparative research. Without variations we cannot compare, and without explanation the art and craft of comparing loses most of its scientific significance.” (PKK p.237)
This concern with “scientific significance” brings us back to the all-other-things-being-equal assumptions I discussed in chapter 1. If all other things are equal and one variable can be legitimately be considered independent while the other one is dependent, then correlation indeed is causation and the variation in the independent variable explains that of the dependent one. And this can even apply to social phenomena. Say that two football teams play twenty games, then exchange goalkeepers while all other players stay put and play twenty more games. If team performance then clearly correlates with the identity of the goalkeeper, then the goalkeepers probably caused the observed difference.
But in real life it is impossible to keep all other things equal in social phenomena. The assumption that the political system is somehow independent of society, that all other things are equal, is just absurd. How could it possibly be independent when the people which act in the political system – as voters, representatives, civil servants etc. – live in society? When all political issues which are resolved within the system arise from society? When the mission of every political party is to change society in a specific way?
I think that political scientists have become trapped in a vicious circle. On the one hand they know that causal relations can be identified only if all others things are equal. So they must assume that political systems are independent of the rest of society. But this is such an unplausible assumption that it forces them to work with abstract concepts in order to hide the multiple evident connections between politics and society. They compensate for this by calculating all kinds of pseudo-scientific quantities, hoping that the correlations they observe between such quantities will reveal cause-effect relationships. If such relationships were to some degree plausible, they could perhaps justify the initial all-other-things-equal assumption. But they’re always just correlations which don’t allow any inferences whatsoever concerning causality.
The most important consequence of this vicious circle is the misleading language which it brings to political analysis. For instance, the authors of Doing research in political science say that their models ”predict” a certain result when referring to historical data which were gathered long before the model was even conceived (PKK p.185). It’s obvious that there’s no hope of predicting the future accurately with political science models, but it’s still a bizarre misuse of language to say that something “predicts” events that occurred in the past. Clearly political scientists have been misled by their assumption of all-other-things-being-equal into thinking that such small things as the distinction between the future and the past don’t matter anymore.
Some implications for political language
Fuzzy social science can be done in many ways. It always requires a mix of theory and statistical analysis and each field of inquiry has its own peculiarities. I’ve tried to indicate that the field of quantified political science as it now stands does not yield any scientific information. It only yields misleading language. Its facade of accurate analysis hides a basic presupposition which doesn’t make sense: the presupposition that political systems can be studied in isolation from the rest of society.
But does this matter? Outside of the inner circle of political scientists, does anyone really care whether they are doing proper research or fallacious correlation-is-causation research? I don’t know, but if I would have to pick any field of research which practicing politicians could be interested in, political history and political science would be my two primary picks. So I would venture to say that political science does matter for the way we talk about politics.
However that may be, the very existence of seemingly legitimate quantitative methods for explaining political phenomena can have an influence on political discourse even without direct references to scientific results. When things are going well, any government will be more than enthusiastic to reap the publicity benefits by proclaiming that their policies have now been proven successful. Likewise, if things go badly the opposition will not fail to make clear that bad decision-making by the government is the root cause of every problem.
But such statements are essentially just reflect the same correlation-causation fallacy which rears its ugly head in political science. In a complex society we can see that certain things change after policies have been enacted, but we can hardly ever find a strong causal link between the two. Correlations are all we see. This means that most statements of blame and self-gratulation should be discounted as rhetoric rather than truth. Rhetoric is of course a normal part of politics and not necessarily harmful unless it is based on badly distorted facts. I have tried to show in this essay that there are good reasons for being sceptical about a particular brand of facts: those based on the fallacious quantitative methods of political science.
However, that does not mean there are no political facts or that nobody should talk about political facts. It just means that we should not talk about political facts with a language copied from the experimental sciences, the language of cause and explanation. The language of politics should be more cautious. But I will leave a closer elaboration on that subject to my next essay.
References
(PKK) Pennings, Keman and Kleinnijenhuis, Doing research in political science, SAGE Publications 1999.
thcson.com 