About this Resource
Home Research Resources

Thomas König

c.v.

courses

goodness

comparative methods

social movements

papers

Quicklinks

Home

That Research is Crappy: It stinks!

Goodness Criteria in Social Research

Deutsch deutsche Version

Any topic in society or politics that garners some interest in even the tiniest of publics will these days be met with a host of social research. Needless to say, such research will be of very disparate quality. But how can we "cut through the crap" and get to the valuable findings? That is an extremely difficult task, because the disparity of goodness criteria is almost as large as the quality variance in social research. Different theoretical traditions use completely different evaluation criteria, and, what is worse, hardly ever talk about these criteria across paradigms. Scholars that work on the same substantive topics may use completely different (albeit not always incompatible) theories, depending on their paradigm affiliation. Each of these paradigms — be it constructionism, Marxism, rational choice, to name only the few most visible paradigms — has developed its own set of criteria, which decide about the "goodness" or "badness" of a theory. In this seminar we will get to know several goodness criteria stemming from important contemporary theoretical traditions. We will compare these criteria in an initial effort to distill goodness criteria that are valid for all social science research.
The seminar is divided into five sections. After a short overview on the topic, part II will cover traditional goodness criteria, namely validity, reliability, parsimony, and empirical falsifiability. Part III discusses goodness criteria that aim at the substantive content of a theory. Part IV introduces the most common statistical goodness criteria and checks, if these criteria reflect the theoretical criteria developed in the previous weeks. In the last part of the seminar we will finally apply goodness criteria to a number of existing studies.

Organization: All required readings are to be read prior to the sessions to which they are assigned.
Requirements: Students must submit weekly assignments and produce either a term paper or an oral presentation.
Grading: assignments 70%, class participation 20%, presentation 10%; when term paper is chosen: assignments 55%, class participation 10%, term paper 35%
Prerequisites: Knowledge in statistical methodology at least at the level of Statistik 2, basic knowledge of research methodology (e.g., Schein in "Einführung in die Methoden empirischer Sozialforschung"); some knowledge of German, sociological theory, quantitative methodology and epistemology a plus.

Acrobat Document Syllabus (Print Version)

I Introduction

Week 1: Constitutive Session

Presentation of the Syllabus, organization of the class.

Recommended Literature (for the entire term)

Week 2: The State of the Art

In an entertaining and extremely well argued essay, former NORC-boss James A. Davis analyzes the strength and weaknesses of contemporary social research. The questions he raises will follow us through the course of the seminar.

Required Readings
Recommended Readings

II Formal Goodness Criteria

The first part of the seminar deals with the most common formal (processual) criteria for goodness. These criteria have an elective affinity with rational choice theories in the realm of social science paradigms and with liberalism in the realm of political philosophy, but can also fairly easily be adapted to other theories and philosophies.

Week 3: Validity: Theory

(Almost) all students who took a class in introductory research methods know that empirical research should rely on "valid" indicators for the measurement of theoretical concepts. In this class we will elaborate the concept of validity and learn to distinguish different types of validity (ecological validity, construct validity, etc.).

Required Readings
Recommended Readings

Week 4: Validity: Praxis

From an epistemological standpoint, the concept of validity is fairly easily delineated. The trouble usually starts when one tries to apply the methodological guidelines that result from the validity criterion to actual research. Then it turns out that these guidelines are not sufficiently specific to select a definite methodology. In the end, a lot depends on the theoretical framework in the decision, which methodology leads to the most valid results. We will explore how these difficulties are tackled with the example of newspaper data on collective action.

Required Reading
Recommended Reading

Week 5: Reliability

Research results ought to be reproducible. This demand might be fair enough for the natural sciences. In the social sciences we face two serious problems in this respect, though:

How can we nevertheless achieve somewhat reliable results?

Required Reading 
Recommended Readings

Week 6: Parsimony

"You gotta differentiate!" That's one of the key phrases social science students learn in their first semester. But the Pawlowian response to any question is not always the correct one. Contrary to conventional wisdom among sociology majors that differentiation always improves a theory, it might well be the case that simplification makes more theoretical headway. In fact, scientific theories chiefly try to meaningfully simplify empirical reality. Therefore, parsimony is an important criterion for the power of a theory.

Required Readings

Also, please shave with Occam's Razor!

Week 7: Testability …

In the so-called positivism dispute in German sociology, Critical Rationalists around Popper insisted on empirical falsifiabilty as the sine qua non of social scientific theories.

Required Reading 
Recommended Readings

III  Substantive Goodness Criteria

Primarily theories somewhat connected to Neo-Marxism apply (also) some substantive content criteria to social research.

Week 8: … vs. social critism (?)

Frankfurt School emphatically rejected Popper's testability criterion, because a solemn recourse to empirical data would contain an inherently conservative effect (because empirical data inevitably are rooted in the past. Instead Frankfurt School insists that social theory should always be aware of and critical about its social context, i.e. societal totality. With respect to the latter, Popper's "facts" might conceal Adorno's "social totality."

Required Readings 
Recommended Readings

Check out this warped reading on the differences about the role of facts between Frankfurt School and Critical Rationalism.

Week 9: Constructionism I: Theory of Category Construction

Although there are still numerous references to Frankfurt School in contemporary literature, it is safe to say that its heyday has long passed. In social theory, social constructionism/constructivism is the rage of the day in theory circles, identity politics has replaced Neo-Marxism among progressive policy makers. Even though constructionism has been en vogue more than a decade by now, there still is very little systematic criteria that could evaluate constructionist approaches. It is two grand theorists, who do not fall squarely into the constructionist camp, who have developed some fruitful goodness criteria for constructionist research. Niklas Luhmann and Pierre Bourdieu reiterate, for instance the fairly ancient, yet counterintuitive demand that constructionist categories should not be derived from everyday concepts. Even though hardly anyone disputes the utility of the their theses, the many constructionists do not follow their guidelines.

Required Readings 

Week 10: Constructionism II: Concept Construction — Praxis

Taking the concept of society as an example, it will be shown that many theoretical approaches take a fairly naive approach to social reality.

Required Reading
Recommended Readings

Week 11: Constructionism III: The Social Construction of What?

Over the last two decades, analyses of social construction have skyrocketed. Yet, one wonders, what does it really mean, when we speak of the social construction of "marriage", "rocks", "nations", "genders", or "doorsteps"? When is imperative, when is it tautological, when is it sensible to speak of social construction? These are the question Hacking asks in his seminal essay on the social construction of what. BTW, one chapter indeed covers the social construction of what.

Required Readings 

IV  Statistical Measures of Goodness

The last part of the seminar will check, if the theoretical criteria we covered in the previous weeks are applicable to quantitative data.

Week 12:  Measures of Association

When it comes to association measures (e.g. Pearson's correlation coefficient), many social researchers think "the bigger, the better." That motto ignores the specific attributes of different correlation measures and tends to neglect the issue of familiarity with a measure within the scientific community. Buchanan discusses these questions in his article.

Required Readings 

Week 13: Bayesian Criteria I

With traditional statistical measures for goodness of fit like the Χ2 Kolmogorov-Smirnov-Test one can only compare nested models. That is troublesome, because most theoretical alternatives for the explanation of social phenomena rely on entirely different factors and are, thus, non-nested. What is more, traditional goodness criteria also do not consider the complexity of a model sufficiently. These shortcomings have recently triggered a growing advocacy of Bayesian goodness criteria, which do not share these problems.

Required Readings 
Recommended Readings

Week 14: Bayesian Criteria II

Follow-up on last week's discussion.

Required Reading
Recommended Reading

V  Round-up

Week 15: Round-up Discussion

Optionally, the last session will be a weekend session, in which students can present reviews of research they have evaluated with the criteria we have covered throughout the seminar.

Required Reading
Recommended Readings

see Weeks 1 & 2

media methods  > staff > thomas könig > syllabus: goodness criteria contact