Three Levels of Research

Levels of research. There are three levels of research. There are also “research” claims that really are not ANY kind of research.

Nonresearch claims.

1. This is writing (e.g., articles) that merely asserts opinions, or beliefs, or “Most educators know that…,” or “Piaget argued that…,” or “According to constructivist philosophy…”

2. There is little or no experimental test of the claims.

3. Readers may be swayed merely because the writing uses emotionally charged and appealing language (holistic, seamless, natural, deep, everyone believes, child centered).

4. Sometimes, the claims are called “theory,” but they really are not theory. They are merely unsupported sentences about the writers’ preferences for how children are taught.

5. A true theory is a set of statements that are connected logically and that form a comprehensive explanation.

Level 1--Basic" research.

1. Level 1 research is "basic research on learning."

2. Correlations, descriptive data and qualitative case studies comprise level 1 research. Examples include field observations (e.g., observing peer reading exercises in class) or it involves some quantitative data (e.g., how many words each peer in the exercises reads correctly per minute when it is his or her turn).

3. The research may be guided by an hypothesis of what the researcher thinks is the case (e.g., peer reading exercises increase reading fluency).

4. The research may identify what APPEAR to be correlations. Or it may show that there are NO correlations.

5. Level 1 research is abundant. However, no theory regarding teaching procedures and materials is testable with descriptive and correlation research (level 1) no matter how abundant it is.

6. Findings from Level 1 research may lead to hypotheses that may be tested in more rigorous Level 2 research.

Level 2--Test of the theory in real classrooms.

1. At level 2 a theory describing how teachers should teach is tested by applying it in the classroom to see if it accurately predicts and gets better results than the practice it replaces.

2. Different teaching interventions are compared at level 2 in controlled research studies to see if students learn more or better in classrooms using teaching procedures based on the theory.

3. Level 2 research is more rigorous than level 1 research.

a. Hypotheses are stated clearly. For example, students who participate in peer reading fluency activities will increase their rate of accurate reading.

b. Variables in the hypotheses are clearly defined (e.g., exactly what goes on in the peer reading exercises [The input or independent variable), and exactly what reading fluency [The outcome or dependent variable] means).

c. Measures, and methods for making the measurements, are developed and tested to see if they are valid---measure what they are supposed to measure. For example, reading experts are consulted on the definitions of fluency and the measures; e.g., each child reads a passage that is 100% decodable (the child knows how to read every word). Each child takes a turn reading. The other child, reading along, marks each error and checks how many minutes the reading took.

In addition, the measures are checked for reliability. That is, if two observers measure the same child’s fluency during an exercise, will the observers arrive at about the same score?

d. Experimental and control groups are formed, and these groups are created by matching or by random allocation to try to ensure that the children are similar on variables that could influence reading fluency. The experimental group consists of students who do the peer reading exercises. The control group might be students who read by themselves and are given strategies for increasing fluency.

e. Fluency (the dependent or outcome variable) is measured at the beginning of the experimental TEST of the hypothesis, during each lesson, and at the end of the series, to see if there is any TREND in each group and to see if (as hypothesized) the experimental group gains more in fluency than the control group.

f. Conclusions are drawn about whether the research hypothesis was supported and whether the null hypothesis (peer readers make no more gains than independent readers) can be rejected.

Level 3--Program Evaluation on a school- or district-wide basis.

1. The same rigorous research is done as in level 2.

2. Level 3 research answers the question,
“Will we find the same thing (e.g., students who work on fluency in peer reading exercises DO make significantly higher gains---between pre-test and post-test---than students who work on fluency independently) when we do this at the level of a whole school or district?”

3. In other words, level 3 research is checking the reliability (repeatability) of the results in different environments (e.g., with different children, and teachers, and different degrees of teacher support).

3. Level 3 research evaluates the effects of the recommended teaching intervention in school-wide or district-wide implementations.
At level 3 scientists are not evaluating one hypothesis regarding one tool, but are evaluating the integration of a whole toolbox full of tools to maximize effectiveness.

4. Level 3 research is what must be done BEFORE writers claim that an innovation works and should be used; and before teachers USE any new method.

These three levels differ in the credibility and therefore the trustworthiness of the findings. Why? Because each next level controls for more possible sources of invalidity that the earlier ones. For instance, Level 1 research may have a very small sample; it may not have validated measurements; it may not be longitudinal. Therefore, the researcher and consumer cannot be confident that findings apply to ANY other students; that the findings are even accurate; or that whatever was found would last. This formulation of three levels of research is in the service of ETHICS. Researchers and consumers should not IMPOSE on children methods that have only been in Level 1 research; they should not impose on a whole school district methods that have only been tested in a few classrooms (Level 2 research).