Reliability & Validity

For each experiment below, indicate what the independent variable (IV) is, the number of levels, and what its levels are. Then, identify the dependent variable (DV), the best way to measure this variable (i.e., self-report, observation, physiological measure), and, based on the measurement, the most likely scale of measurement (i.e., categorical/nominal, ordinal, interval, ratio).

 

1. SAMPLE: Researchers are interested in studying whether children who play violent video games are more aggressive than those who play non-violent video games. They randomly assigned children to play a violent or non-violent video game, then observed how they played with other toys in the “waiting room” for their parents.

 

IV: # of levels Names of levels
Type of video game 2 Violent, non-violent
     
DV: Methodology Scale of measurement
Aggression Observation Ratio

 

 

2. Several researchers hypothesized that group study sessions contribute more to learning than individual study sessions do. They tested college students by assigning one group as participants in group study sessions and a second group as participants in independent study. Learning was assessed by performance on an achievement test given after the study sessions.

 

IV: # of levels Names of levels
     
     
DV: Methodology Scale of measurement
     

 

 

3. To assess the efficacy of a new medication, children who were patients at Children’s Hospital were randomly assigned to treatment schedules. One group remained on the old medication, another group was given 10mg of the new medication, and another group was given 20mg of the new medication. Comparisons of white cell counts taken at each hospital visit indicated the effectiveness of the treatment.

 

IV: # of levels Names of levels
     
     
DV: Methodology Scale of measurement
     

 

 

4. The impact of a nutrition and exercise program on weight loss was examined. Random assignment of female adults established groups of generally equal size and weight. One group participated in a daily exercise and nutrition program, another group participated in an exercise program alone, another group participated in a nutrition program alone, and a fourth group did not exercise nor change their eating habits. Weight loss was assessed by measuring the pounds lost in each participant.

 

IV: # of levels Names of levels
     
     
DV: Methodology Scale of measurement
     

 

 

5. Jose wishes to determine which of three new varieties of beer will be liked best by the patrons of the bar. He gives everyone who comes to the bar samples of all three beers, randomizing the order they get them (this is called a “within-subjects” design). He then asks each person to rank the three beers from worst to best.

 

IV: # of levels Names of levels
     
     
DV: Methodology Scale of measurement
     

 

 

6. Xavier decides to investigate whether rats that have learned one maze will learn a different one more quickly compared to rats who have never learned a maze. He randomly assigns rats to either have prior experience with a “practice” maze or to not get any practice, and then times how long it takes to run a new maze.

 

IV: # of levels Names of levels
     
     
DV: Methodology Scale of measurement
     

 

 

Variables Practice – Part 2

 

For this part of the lab, you’ll be considering how to evaluate reliability and validity of the DV from two of the scenarios from Part 1.

· Step 1: Pick a scenario that does not use observation as the methodology.

· Step 2: For this scenario you’ll state the types of reliability that should be evaluated and how you could do that evaluation.

· Step 3: State the types of validity that should be evaluated, and how you could do that evaluation.

(Hint: the types of reliability and validity that you evaluate should depend on the type of methodology used to measure the DV! You should list all types of reliability and validity that you think are relevant to that measure, not just one (unless only one is relevant)!)

 

Sample Scenario # 1

 

Types of reliability that are relevant, and ways to assess each type:

Inter-rater reliability is most important for observational measures, and I could use r (a correlation) to measure it because it is a continuous and not a categorical measure

I could potentially assess test-retest reliability if I went back to observe the same children at a later time.

 

Types of validity that are relevant, and ways to assess each type:

Face validity – I could ask a panel of experts if the things I plan to observe seem to generally fit the concept of aggression

Content validity – I could assess whether what I plan to observe measures all of the aspects of aggression

Criterion validity – I could see if children who score high on my measure end up getting in more behavioral trouble at school for things like fighting than children who score lower

Convergent and divergent (discriminant) validity might be harder to assess because I am observing as my form of measurement, but I could also try to observe something related to aggression to see if scores correlate for convergent (maybe frowning [positive correlation] or smiling [negative correlation]). I could also look for something unrelated to make sure scores don’t correlate for divergent (discriminant) validity (maybe how tall the child is).

 

 

 

 

Scenario # _________

 

Types of reliability that are relevant, and ways to assess each type:

 

 

 

Types of validity that are relevant, and ways to assess each type:

 

 

 

 

 

For 1 pt of Extra Credit , complete an additional assessment of reliability and validity. Make sure that this scenario uses a different methodology (not observation or the methodology used in your previous scenario)!

This is all or none, I will only be grading for completion. For feedback, feel free to come talk to me at Student Hours.

 

Scenario # _________

 

Types of reliability that are relevant, and ways to assess each type:

 

 

 

Types of validity that are relevant, and ways to assess each type: