Linda Suskie

  A Common Sense Approach to Assessment in Higher Education

Blog

Understanding direct and indirect evidence of student learning

Posted on May 10, 2019 at 8:50 AM

A recent question posted to the ASSESS listserv led to a lively discussion of direct vs. indirect evidence of student learning, including what they are and the merits of each.


I really hate jargon, and “direct” and “indirect” is right at the top of my list of jargon I hate. A few years ago I did a little poking around to try to figure out who came up with these terms. The earliest reference I could find was in a government regulation. That makes sense—governments are great at coming up with obtuse jargon!


I suspect the terms came from the legal world, which uses the concepts of direct and circumstantial evidence. Direct evidence in the legal world is evidence that supports an assertion without the need for additional evidence. Witness knowledge or direct recollection are examples of direct evidence. Circumstantial evidence is evidence from which reasonable inferences may be drawn.


In the legal world, both direct and circumstantial evidence are acceptable and each alone may be sufficient to make a legal decision. Here’s an often-cited example: If you got up in the middle of the night and saw that it’s snowing, that’s direct evidence that it snowed overnight. If you got up in the morning and saw snow on the ground, that’s circumstantial evidence that it snowed overnight. Obviously both are sufficient evidence that it snowed overnight.


But let’s say you got up in the morning and saw that the roads were wet. That’s circumstantial evidence that it rained overnight. But the evidence is not as compelling, because there might be other reasons the roads were wet. It might have snowed and the snow melted by dawn. It might have been foggy. Or street cleaners may have come through overnight. In this example, this circumstantial evidence would be more compelling if it were accompanied by corroborating evidence, such as a report from a local weather station or someone living a mile away who did get up in the middle of the night and saw rain.


So, in the legal world, direct evidence is observed and circumstantial evidence is inferred. Maybe “observed” and “inferred” might be better terms for direct and indirect evidence of student learning. Direct evidence can be observed through student products and performances. Indirect evidence must be inferred through what students tell us, through things like surveys and interviews, or what faculty tell us through things like grades, or some student behaviors such as graduation or job placement.


But the problem with using “observable” and “inferred” is that all student learning is inferred to some extent. If a crime is recorded on video, that’s clearly direct, observable evidence. But if a student writes a research paper or makes a presentation or takes a test, we’re only observing a sample of what they’ve learned, and maybe it’s not a good sample. Maybe the test happened to focus heavily on the concepts the student didn’t learn. Maybe the student was ill the day of the presentation. When we assess student learning, we’re trying to see into a student’s mind. It’s like looking into a black box fitted with lenses that are all somewhat blurry or distorted. We may need to look through several lenses, from several angles, to infer reasonably accurately what’s inside.


In the ASSESS listserv discussion, John Hathcoat and Jeremy Penn both suggested that direct and indirect evidence fall on a continuum. This is why. Some lenses are clearer than others. Some direct evidence is more compelling or convincing than others. If we see a nursing student intubate a patient successfully, we can be pretty confident that the student can perform this procedure correctly. But if we assess a student essay, we can’t be as confident about the student’s writing skill, because the skill level displayed can depend on factors such as the essay’s topic, the time and circumstances under which the student completes the assignment, and the clarity of the prompt (instructions).


So I define direct evidence as not only observable but sufficiently convincing that a critic would be persuaded. Imagine someone prominent in your community who thinks your college, your program, or your courses are a joke—students learn nothing worthwhile in them. Direct evidence is the kind that the critic wouldn’t challenge. Grades, student self-ratings, and surveys wouldn’t convince that critic. But rubric results, accompanied by a few samples of student work, would be harder for the critic to refute.


So should faculty be asked or required to provide direct and indirect evidence of student learning? If your accreditor requires direct and indirect evidence, obviously yes. Otherwise, the need for direct evidence depends on how it will be used. Direct evidence should be used, for example, when deciding whether students will progress or graduate or whether to fund or terminate a program. The need for direct evidence also depends on the likelihood that the evidence will be challenged. For relatively minor uses, such as evaluating a brief co-curricular experience, indirect evidence may be just as useful as direct evidence, if not even more insightful.


One last note on direct/observable evidence: learning goals for attitudes, values, and dispositions can be difficult if not impossible to observe. That’s because, as hard as it is to see into the mind (with that black box analogy), it’s even harder to see into the soul. One of the questions on the ASSESS listserv was what constitutes direct evidence that a dancer dances with confidence. Suppose you’re observing two dancers performing. One has enormous confidence and the other has none. Would you be able to tell them apart from their performances? If so, how? What would you see in one performance that you wouldn’t see in the other? If you can observe a difference, you can collect direct evidence. But if the difference is only in their soul—not observable—you’ll need to rely on indirect evidence to assess this learning goal.

Categories: Good assessment, How to assess

Post a Comment

Oops!

Oops, you forgot something.

Oops!

The words you entered did not match the given text. Please try again.

5 Comments

Reply Oliver Mark
7:08 AM on July 26, 2019 
Thank you for sharing this information with us. All of the mentioned points are written perfectly. This blog has so many new things to learn. Looking forward to read more. https://www.bookmyessay.com/
Reply MapleInc
7:13 AM on May 27, 2019 
Great Content. very useful information you have shared. I have learned lots of great things. Keep it up
Reply ★ Owner
3:28 PM on May 11, 2019 
Thank you for this, Madeleine. This is getting off the direct-indirect subject, but another good question to ask about a learning goal that's an attitude, value, or disposition is whether it's teachable. Can we teach a student to be a confident dancer? What learning activities can we give that help instill confidence? And how do we grade those learning activities? Can we ensure that, with adequate support, every student will graduate a confident dancer?

If you can teach it, you can assess it. But if you simply hope students pick it up by osmosis or inspiration, I'm not sure it's a teachable learning goal. One of my favorite quotes from Peter Ewell is "Don't hold people accountable for things they cannot do." If you can't teach it, I'm not sure it's appropriate to assess it.

Madeleine says...
Excellent points.

It comes down to something philosophical: We can only know other people's minds through their performance. The art of giving good assignments - and it's the most interesting part of teaching, for me, and the most tricky - is to come up with some performance or activity that could *only* be done competently by someone who had the skills you're trying to teach. This is much harder than it seems in many cases.

And another philosophical point, re your comment about soul: Where does reality lie? If a dancer *looks* exactly like she dances with confidence, does it matter if she doesn't really feel confident? In fact, what exactly do we mean by "confident?" Lots of successful performers get terrible nerves and suffer from impostor syndrome, but they still make a career out of their talent: so, are they more or less confident than the person who isn't unduly hard on himself as a performer, but ends up talking himself into a "safer" job?

Up to a point, these philosophical considerations feel a bit airy-fairy if you're running a CTE program training hospitality workers, or cooks, or lawyers, or barbers, or GPs. But they're absolutely relevant to the model of the general-ed, four-year liberal arts education.
Reply ★ Owner
3:18 PM on May 11, 2019 
I just realized that, while I'm usually super-practical nuts-and-bolts, this blog post doesn't give any practical examples of direct and indirect evidence of student learning. The kinds of assessments I've seen most commonly considered to yield direct evidence of student learning are tests scored using a test blueprint and any student work or performance assessed with a rubric, including field experiences and portfolios. Self-ratings, surveys, grades, overall test or assignment scores, and completion, graduation, and job placement rates are, in my experience, commonly considered to be indirect evidence of student learning.
Reply Madeleine
1:28 PM on May 10, 2019 
Excellent points.

It comes down to something philosophical: We can only know other people's minds through their performance. The art of giving good assignments - and it's the most interesting part of teaching, for me, and the most tricky - is to come up with some performance or activity that could *only* be done competently by someone who had the skills you're trying to teach. This is much harder than it seems in many cases.

And another philosophical point, re your comment about soul: Where does reality lie? If a dancer *looks* exactly like she dances with confidence, does it matter if she doesn't really feel confident? In fact, what exactly do we mean by "confident?" Lots of successful performers get terrible nerves and suffer from impostor syndrome, but they still make a career out of their talent: so, are they more or less confident than the person who isn't unduly hard on himself as a performer, but ends up talking himself into a "safer" job?

Up to a point, these philosophical considerations feel a bit airy-fairy if you're running a CTE program training hospitality workers, or cooks, or lawyers, or barbers, or GPs. But they're absolutely relevant to the model of the general-ed, four-year liberal arts education.