Methods and measurement-driven extension: Test whether different methods lead to the same results, and whether existing measures are sufficient

Sometimes, there may appear to be nothing wrong with the measurement procedure used in the original study or the measures selected (e.g., questions in a questionnaire) to measure the constructs you are interested in; at least on the surface. We say that these weaknesses do not appear on the surface because the authors report the findings from commonly used statistical tests that illustrate the findings to be reliable. However, just because they are reliable on the surface does not mean that they are valid. This reflects two types of bias that can creep into studies: mono-operation bias and mono-method bias. We discuss these briefly with examples to show you how these types of bias can present you with an opportunity to extend the original study.

• Mono-operation bias and extension

  Whilst there are many different ways in which a given construct can be measured, it is not uncommon for researchers to only use a single measure for that construct (i.e., a single variable). To explain what we mean by this, and how it can become a threat to validity (or more specifically, construct validity), imagine that the example in Study #3 below is the study that you want to replicate:

  Study #3
  The relationship between background music and task performance amongst employees at a packing facility
  
  The study aims to examine the relationship between background music (i.e., construct A) and task performance (i.e., construct B) amongst employees at a packing facility (e.g., Amazon, Wal-Mart, Tesco, etc.). In these packing facilities, the job of employees is to collect items ordered by customers from the warehouse, package them, stick on a label with the customer's address, and put the package on the delivery line. Each time an employee does this, they complete one task.
  
  The purpose of the study is to find out what effect background music might have on employees' task performance; that is, how many packages (i.e., tasks) they process in a given hour. This is important to firms because if they find that background music has a positive effect on task performance; that is, if background music increased the number of packages processed in a given hour, they may want to rollout a programme of background music in all of their packing facilities.
  
  The independent variable is background music (i.e., construct A), which is a nominal variable because employees are either provided with or without background music. The dependent variable is task performance (i.e., construct B), which is a continuous variable, measured in terms of the number of tasks employees perform correctly per hour.
  
  The independent variable, background noise, consists of a control and a treatment. The control refers to the normal conditions experienced by employees in the packing facility, which in this case, means that employees are not being provided with background music (i.e., employees without background music). The treatment is the intervention that we are making to compare the addition of background music with the normal conditions (i.e., with the control) in the packing facility. In other words, the treatment is providing the employees with background music. It is this independent variable (i.e., background music) that we are manipulating to examine its effect on the dependent variable (i.e., task performance). We use the word manipulating because we are taking the independent variable and changing it (i.e., with or without background music) for different groups (i.e., the control group and treatment group).
  
  So in order to conduct this experiment, we take a sample of employees at the packing facility (e.g., a sample of 100 employees from the total of 400 employees that work there, which is known as the population). We then randomly assign half of these sample employees (i.e., 50 employees) to the control group and the other half (i.e., 50 employees) to the treatment group. At a given day and time, we start the experiment; so the control group continue with their normal day without any music, whilst the treatment group gets to listen to music. The experiment continues for an 8 hour shift. For each of these 8 hours, we record the number of tasks each employee performs correctly, both for the control group and the treatment group. This task performance is our dependent variable (also known as an outcome variable).
  
  Under normal circumstances, we would then statistically analyse our results by comparing the scores on the dependent variable (i.e., the number of correctly performed tasks per hour) between the two groups (i.e., the control group and the treatment group). This should show us whether there are any differences in the number of tasks performed between the control group and treatment group. This would, in theory, tell us about the relationship between background music and task performance amongst the employees at the packing facility. When we perform the analysis on the data from the two groups, we find that (a) there is a difference between background music, (b) the difference is statistically significant, and (c) the difference equates to a 10% increase in task performance. In other words, we found that the addition of background music improved the task performance of employees by 10% compared to the control group that had no background music. Since our statistically analysis shows that the relationship between background music and task performance was statistically significant, we conclude with some confidence that background music improves task performance in the packing facility.

  Under normal circumstances, we may look at these results and feel confident that the original authors did a good job, and that that the findings should be trusted. After all, there were a number of components to the study that gave us such confidence: (a) an experimental research design was used, which as you'll learn more about in the section on Research Design, provides the strongest warrant for knowledge claims; (b) the participants were randomly assigned to each group; (c) the independent variable was manipulated; and (d) the results were statistically significant. Again, without going into any detail at this stage, these are all characteristics of an experimental research design.
  
  However, let's go back to thinking about the potential threat to construct validity when only using a single measure for that construct. Let's just look at one construct from Study #3, construct A, which was background music, our independent variable. We measured background music in a very simple way; that is, background music was presented as a nominal variable because employees were either provided with or without background music. However, the construct, background music, is actually much more complex. Think about the following aspects of music:

  • Type of background music (e.g., chart music, dance/electronic music, easy listening, classical music, etc.)

  • Loudness of background music (e.g., low, medium, high volumes, etc.)

  Since only one type of background music was played during the experiment (e.g., easy listening), which was played at a medium volume (i.e., loudness), we measured our independent variable, background music, using just a single measure. The problem that we face when viewing the construct, background music, in such a simple way, is that we ignore the potential real effect that this construct has on the dependent variable; in this case, task performance. Known as mono-operation bias, this becomes a threat to the construct validity of the measurement procedure we used. We are potentially under-representing the construct we are trying to measure. Indeed, if we had taken into account the different aspects of the construct, background music, including the type of background music, the loudness of the background music, and so on, we would have a much more accurate understanding on the relationship between construct A (i.e., background music, the independent variable), and construct B (i.e., task performance, the dependent variable). This would have improved the construct validity of our measurement procedure.
  
  So imagining that Study #3 was the study we wanted to replicate and extend (i.e., Route #3: Extension), this also presents us with an opportunity to repeat the experiment, but rather than using a single measure for each of the constructs, we can use multiple measures. Now you can't look at everything. You're only doing an undergraduate or master's level dissertation. So you probably can't look at every aspect of background music. You might just want to look at the loudness of background music (e.g., low, medium, and high volumes). Therefore, rather than having just have two groups, one that listens to music and another that doesn't, we add another two groups: one group doesn't listen to music (i.e., the control group), and each of the other three groups (i.e., our three treatment groups) listens to a different loudness of music (i.e., one groups listens to music at a low volume, another at the medium volume, and the third at a high volume). Of course, there has to be a justification for this; perhaps there is a theory about music and the psychology of distraction that can help justify why this would be an appropriate extension of the existing study.

• Mono-method bias and extension

  Just as there are threats to construct validity from a single measure, as discussed in the previous section, construct validity can also be threatened when using a single method to measure a given construct (irrespective of whether the construct is acting as the dependent or independent variable). This is because the method used may introduce bias, changing the scores on the independent or dependent variable. It is known as mono-method bias.

  Before we reflect back on the threat to construct validity from using a single method, let's look at the problems that can arise from using a single method, based on our example in Study #3:

  Study #3
  The relationship between background music and task performance amongst employees at a packing facility
  
  To briefly recap, our study examined the relationship between background music (i.e., construct A) and task performance (i.e., construct B) amongst employees at a packing facility (e.g., Amazon, Wal-Mart, Tesco, etc.). The independent variable was background music (i.e., construct A), whilst the dependent variable was task performance (i.e., construct B). One group of employees listened to background music whilst working (i.e., the treatment group), whereas the other group were not provided with any background music (i.e., the control group). The task performance of employees was measured in terms of the number of tasks employees perform correctly per hour. The method used to listen to background music was a loud speaker (i.e., stereo system), whilst task performance was measured using an e-packing system, which automatically collected data on the number of tasks correctly performed by employees.
  
  Let's imagine some of the multiple methods that could be used to measure these two constructs (i.e., construct A, background music, and construct B, task performance):

  
  

  • Independent variable
    
    Method #1: Listening to music through the loud speaker (i.e., stereo system)
    Method #2: Listening to music using a personal iPod and headphones

  • Dependent variable
    
    Method #1: Data automatically collected through the e-packing system
    Method #2: Supervisor rating the speed of the packer

  Note that sometimes a mono-method (i.e., a single method) is appropriate. For example, Method #1 for the dependent variable (i.e., data being automatically collected through the e-packing system, recording task performance accurately) may be the most accurate measure of task performance in this piece of research. After all, Method #1, where the supervisor rates the speed of the packer is more likely to result from experimenter bias or instrumental bias than an automated system that does not suffer from such bias. For this reason, there would probably be no justification to change the method used to measure the dependent variable in the study you were replicating. However, this is often not the case, and the use of multiple methods reduces the threat to construct validity.
  

  When considering whether mono-method bias has taken place in the study you want to replicated, you need to ask yourself:

  • Would the same results have been recorded if the independent variable, background noise, had been operationalized using a different method; in this case, using Method #2 (i.e., listing to music using a personal iPod and headphones) rather than Method #2 (i.e., listening to music through the loud speaker/stereo system)?

  • Would the use of multiple methods have provided greater insight into the construct than just a single method; that is, would have multiple methods reduced the potential for method bias to affect the scores on the dependent variable?

  Therefore, if the measurement procedure consists of a single method to assess the independent and/or dependent variables, this can act as a threat to construct validity. In order to reduce the threat from mono-method bias, it is useful to use more than one method when measuring a given construct. This can act as a strong justification for extending an existing study. You can extend it simply by using a different method to analyse the independent and dependent variables, or better still, use the old method and add a new one, and the compare the validity of the two methods. This is something that we show you how to do within the Lærd Dissertation site.

The decision to (a) use different or additional variables to measure a construct, or (b) use different or additional methods to those used in the original study, can enable you to make a really interested contribution to the literature through your replication-based dissertation. In this sense, Route C: Extension can allow you to address a potential weakness in an original study, improve the construct validity of the original study, as well as deal with the potential dangers of mono-operation bias and mono-method bias.

Decision

The choice of extension that you can make is very specific to the study that you want to build on. Whilst we explain some of the broader justifications for replication-based dissertations in the section that follows, it is important to understand the specific reasons for making a particular type of extension. As a result, we explain more about making these decisions within the Lærd Dissertation site. However, it is worth noting that Route C: Extension does provide you with many ways to build on an existing study, and potentially make a more original contribution than Route B: Generalisation.