Empirical Research: Beyond Methods for Data Analysis
What do we need to consider in preparing for empirical research? The impression sometimes exists that a course on data analysis will do the job. But empirical research entails much more than that.
In recent years, there has been growing interest in the empirical study of law and legal institutions. In line with this trend, law schools have started adding new courses to their existing PhD tracks. And workshops have emerged that introduce legal scholars to the praxis of empirical research. Most of these initiatives have focused on methods for data analysis. Attention to research design, social science theory and the philosophy of science has increased as well, but overall this has been less pronounced. These issues nonetheless warrant – more – serious consideration: they are important for sound and innovative research. Why?
Research designs are the backbone of empirical studies. A research design can be seen as a project plan that combines a research question with a strategy for case selection, data collection and data analysis. Let me give an example. Suppose that we are interested in understanding how the performance of students in law school influences their careers. In order to address that question, we need to decide which research subjects to include in our project, which data to collect and how to analyse the data. We could for instance select a small number of students at one law school, follow them through the programme and interview them extensively afterwards. But we could also collect data on alumni across different law schools, and look for information on law school grades and the post-graduation income of these alumni. What are the pros and cons of these two strategies? And what other approaches could we adopt? These are essential questions that affect the quality of a research project on the one hand, and the extent to which it meets the ambitions or objectives of a researcher on the other. Research designs are thus essential.
But so is knowledge of social science theory. How is that relevant? An understanding of different social science theories allows us to consider a variety of explanations for phenomena that we are interested in. These explanations enrich our studies. But they also help us alleviate methodological threats to our work such as confounding or spuriousness. Let me return to the above example to illustrate the problem of confounding and the value of theory. And let’s also assume in this context that we find a near perfect correlation between law school grades and post-graduation income. What does that tell us? A simple answer would be: grades really matter. But if we think a bit harder, and have read social scientific studies on education and stratification, we may come to realize that the relation is spurious. What does that mean? Well, perhaps research on education and stratification has shown that white students tend to receive higher grades – irrespective of their abilities – and that white students are also more likely to find high-paying jobs – again, not just because of their knowledge or skills. In that case, the near perfect correlation between grades and post-graduation income can be partly traced back to ethnicity. And thus, ethnicity needs to become part of the analysis to prevent distorted outcomes.
The relevance of research design and social science theory is widely accepted among social scientists. But what about the philosophy of science? Does our empirical work benefit from reflections on the nature of reality and our ability to know it? Yes, it does. These reflections help us ground our research, choose our research questions, and facilitate decisions related to research design. Some scholars, for example, consider social reality to be governed by a series of behavioural laws – not legal rules – and believe that we can isolate and identify these laws through research. For those scholars, experimental studies can be very effective. Other scholars may believe that social reality cannot be reduced to a set of behavioural laws. These scholars may prefer participant observation. Studying debates on the philosophy of science thus makes us aware of our own positions on these and many other issues regarding reality and knowledge. And that awareness allows us to attune our beliefs to our work.
This may all sound quite daunting. Indeed, empirical research has many dimensions. It goes beyond statistics and SPSS. The multidimensionality and occasional complexity of empirical research should, however, not stop us from embarking on empirical journeys. Empirical research is intriguing, it nourishes our curiosity, and it allows us to constantly discover new things. But is empirical research not just a hobby for legal scholars? Is it really useful? Interestingly, empirical studies can complement more traditional legal research in different ways. For instance, in addition to asking ourselves how judges should decide based on some set of pre-defined principles, we can examine how these actors do actually arrive at their decisions, or explore which principles should be governing decisions in court according to judges themselves. In short, empirical research has great potential for our work. If we join forces within the law school, and bring people with different skills together, we can realize that potential.