Testing the generalizability of a study

As discussed in Route B: Generalisation, one of the main goals of quantitative research is generalisation; that is, testing to see whether the findings from the study you are interested in replicating hold across a range of populations, settings/contexts, treatments and time. But the justification for generalisation goes further than a basic desire to see how far a study's findings hold. It is a philosophical, theoretical and practical question:

• Philosophy and justifications for generalisation

  Now we don't want to scare or bore you at this stage with the idea of philosophy, or more specifically, the research paradigms that we use to guide our dissertations, often unknowingly. After all, we explain more about these in the Research Paradigms section of Lærd Dissertation. However, simply put, research paradigms are ways of explaining the basic set of beliefs that you have (i.e., at a philosophical level) and how these influence the way you do research (i.e., the practical aspects of doing a dissertation). Some of the research paradigms that you will likely come across when doing your dissertation include positivism, post-positivism, critical theory, constructivism, amongst others. Whilst some students doing dissertations at the undergraduate and master's level do not have to worry about these things, most do.

  Even though we are not going to discuss these in any detail at this point, the important thing to remember is that we all have different basic sets of beliefs, and since these influence the way that we do research, the study that you choose to replicate will have been influenced by the beliefs of the authors involved. This leads to a possible philosophical justification for generalisation:

  For example, imagine that the study's authors made what you would consider to be 'wild assertions' when it came to saying how far their findings could be generalised. To illustrate this, take Study #1, where we examined the relationship between teaching method and exam performance. We concluded that the use of seminars in addition to lectures improved exam performance amongst our population of undergraduate students at a single university. But what if in the Discussion section or chapter of our dissertation write up, we had concluded that: The addition of seminars to lectures improves exam performance amongst university students. We are making the assertion that our results can be generalised not only to the population that we investigated (i.e., undergraduate students at a single university in the United States), but a much wider population (i.e., all types of student - undergraduates, postgraduates, part-time students, full-time students, etc. - and all universities, wherever they may be in the world). Now such an assertion could simply reflect a loose writing style, which could be criticised for being nothing more than that, but it could also reflect a particular basic set of beliefs (i.e., those beliefs that form part of a research paradigm known as positivism, which without going into any detail, are more inclined to support context-free generalisations such as these). If your basic set of beliefs differs from these, and you feel that such assertions cannot be made, this would be a philosophical justification to test the different populations, settings/contexts, treatments and time in which the findings from the original study hold.

• Theory and justifications for generalisation

  When your dissertation aims to make generalisations across populations, settings/contexts, treatments, or time, there are occasions when it is not possible to simply apply the same research strategy used in the original study to your own dissertation. As we stated in Study #2, sometimes the publication of new research suggests that other constructs and/or variables play a role in explaining the phenomenon you are interested in. In other cases, the population or setting/context that you are generalising from has fundamentally different characteristics to the new population or setting/context you want to generalise to. We're not simply talking about the make-up (i.e., characteristics) of the sample used (e.g., the proportion of males to females, professional vs. non-professionals, etc.), but some deeper, theoretical factor that has to be taken into account.
  
  For example, in Study #2, the authors (Boles et al., 2000) wanted to generalise research about retail consumers (i.e., like you and me) to business customers. However, since relationship quality, the construct they were interested in, had some fundamental differences when comparing retail and business customers, they had to add another construct to their research, equity, and a number of additional variables to measure this construct. If the authors had left the construct, equity, out of their research, and just applied the previous constructs in the original study to their new, business context, the study could have been heavily criticised, or even have been flawed. The key point is that theory helps to highlight the ways in which we need to check how far existing studies can be generalised. Replication-based dissertations are important to explore such generalisations, taking into account the role of theory. We explain more about the role of theory in replication-based dissertations in the section: The Route #1 Process.

• Practicalities and justifications for generalisation

  There are many practical justifications to test the generalizability of a study. For example, academics sometimes use student samples because they are easy to access and provide, at the very least, a pilot to help understand the phenomenon they are interested in (i.e., they conduct their research amongst the students in the class or year group that they teach). However, there can be a number of problems with student samples, in particular, the fact that they do not represent the general population. But what if the authors are generalising to the general population, not just students? Clearly, this is a problem, but you will see it quite frequently in journals. Whilst this is just one example, there are many practical issues associated with the sampling strategy that authors select to collect data about the population they are interested in. Now while we explain this in more detail in the section on Sampling Strategy, the important point is that such problems with the sampling strategies of studies can question whether these studies can be generalised. This provides a justification to test the generalizability of the original study by carrying out a replication-based dissertation that tests such generalisations.

  Other practical factors that justify a generalisation-based replication study include the research design that authors select. For example, if the authors used an experimental research design, this has a number of advantages when it comes to strengthening the internal validity of a quantitative study. However, whilst a study must be internally valid and reliable before it can be externally valid (i.e., before we can make generalisations from it), experimental research designs tend not to reflect the real world conditions of the phenomenon you are interested in. After all, just as there are threats to internal validity, there are also threats to external validity; that is, there are factors that undermine the confidence that we have in the generalisations that authors make from their study. For example, in experimental conditions, it is not uncommon for participants to behave in a way that is different from how they would behave in everyday life; something known as a subject effect. Therefore, if a study uses an experimental research design, as opposed to a quasi-experimental or non-experimental research design, this would be one possible reason to question the generalizability of the findings in the original study. It would provide a justification to test the generalizability of the study. We explain more about these types of research design and their implications in the Research Designs section of Lærd Dissertation.

Building the construct validity and reliability of a measurement procedure

As discussed earlier, quantitative research involves the study of constructs and the measurement of variables. The study you choose to replicate will involve such constructs and variables, irrespective of the type of replication-based dissertation you take on.

To briefly recap, constructs are mental abstractions that we use to express the ideas (e.g., ageism, poverty, air pollution), people (e.g., obesity, morbidity, age, etc.), organisations (e.g., service quality, firm survival, outsourcing, etc.), events (e.g., famine, urban regeneration, Jihad, etc.), and/or objects/things (e.g., trees, stem cells, hurricanes, etc.) that we are interested in. We often refer to constructs as mental abstractions because seldom are constructs directly observable (e.g., we cannot directly observe depression, even though we may associate depression with signs such as a person that often cries, engages in self-harm, has mood swings, etc.). Instead, we use variables to operationalize (i.e., measure) the constructs we are interested in (e.g., we may measure the construct, obesity, using the variable, Body Mass Index (BMI)).

When we examine a construct in a study, we choose one of a number of possible ways to measure that construct. For example, we may choose to use questionnaire items, structured interview questions, and so forth. These questionnaire items or structured interview questions are part of the measurement procedure. This measurement procedure should provide an accurate representation of the construct it is measuring if it is to be considered valid. For example, if we want to measure the construct, intelligence, we need to have a measurement procedure that accurately measures a person's intelligence. Since there are many ways of thinking about intelligence (e.g., IQ, emotional intelligence, etc.), this can make it difficult to come up with a measurement procedure that has strong construct validity.

• Construct validity

  Construct validity can be viewed as an overarching term to assess the validity of the measurement procedure that you use to measure a given construct. Now whilst we explain much more about construct validity in the section, Research Quality, the important point at this stage is that you cannot say that a measurement procedure has permanently or absolutely established construct validity. Rather, this is an ideal. With each additional study that shows a measurement procedure to have strong construct validity, especially in a wide range of populations and contexts/settings, the claim of strong construct validity becomes greater. As a result, if the study you are replicating does not build on constructs that have already been tested in a wide range of populations and contexts/settings, you can justify replication on the basis of strengthening such construct validity. Deeper in the Lærd Dissertation site, we explain how to identify whether the measurement procedure in the original study is construct valid.

• Reliability

  Another important consideration when justifying replication is to improve the reliability of a measurement procedure. Reliability, like validity, is a way of assessing the quality of the measurement procedure used to collect data in a dissertation. In order for the results from a study to be considered valid, the measurement procedure must first be reliable. Now whilst we explain more about reliability in the Research Quality section, the important point is that an unreliable measurement procedure can be a significant weakness of a study. If the authors of the study you are interested in replicating (a) did not detail the reliability of their measurement procedure, which is easy to tell because you are looking for some specific things in the Research Methods or Data Analysis sections of their article, or (b) had some inconsistencies in the reliability of their measurement procedure, this acts as a useful justification to conduct a replication-based dissertation.

CONSIDERATION THREE

Specific justifications for replication-based dissertations

In the previous section, we highlighted three of the common justifications for replication-based dissertations, but there are actually a wide range of more specific reasons why a particular study should be replicated. If you choose to pursue this route in your dissertation, understanding these specific reasons in the context of the study you want to replicate will be very important. Whilst we do not go into detail now, we provide the explanation needed to identify many of these more specific justifications for replication-based dissertations within the Lærd Dissertation site. At this point, it is worth more broadly determining whether a replication-based dissertation is right for you, which we do next, in STEP THREE.