|Posted on February 23, 2019 at 5:55 AM||comments (4)|
Curriculum maps have become trendy in the last few years. They’ve built into some commercial assessment management systems. But to some faculty they’re simply one more pointless chore to be completed. Why bother creating a curriculum map?
First, what is a curriculum map? It’s a simple chart identifying the key learning goals addressed in each of the curriculum’s key elements or learning activities. A curriculum map for an academic program identifies the program learning goals addressed in each program requirement. A curriculum map for a course identifies the course learning goals addressed in each learning experience and assessment.
So why are we creating curriculum maps? They’re handy tools for analyzing how well a curriculum meets many of the traits of effective curricula discussed in Chapter 5 of my book Assessing Student Learning: A Common Sense Guide:
Is the curriculum designed to ensure that every student has enough opportunity to achieve each of its key learning goals? A program curriculum map will let you know if a program learning goal is addressed only in elective courses or only in one course.
Is the curriculum appropriately coherent? Is it designed so students strengthen their achievement of program learning goals as they progress through the program? Or is attention to program learning goals scattershot and disconnected?
Does the curriculum give students ample and diverse opportunities to achieve its learning goals? Many learning goals are best achieved when students experience them in diverse settings, such as courses with a variety of foci.
Does the curriculum have appropriate, progressive rigor? Do higher-numbered courses address program learning goals on a more advanced level than introductory courses? While excessive prerequisites may be a barrier to completion, do upper-level courses have appropriate prerequisites to ensure that students in them tackle program learning goals at an appropriately advanced level?
Does the curriculum conclude with a capstone experience? Not only is this an excellent opportunity for students to integrate and synthesize their learning, but it’s an opportunity for students to demonstrate their achievement of program learning goals as they approach graduation. A program curriculum map will tell you if you have a true capstone in which students synthesize their achievement of multiple program learning goals.
Is the curriculum sufficiently focused and simple? You should be able to view the curriculum map on one piece of paper or computer screen. If you can’t do this, your curriculum is probably too complicated and therefore might be a barrier to student success.
Is the curriculum responsive to the needs of students, employers, and society? Look at how many program learning goals are address in the program’s internship, field experience, or service learning requirement. If a number of learning goals aren’t addressed there, the learning goals may not be focusing sufficiently on what students most need to learn for post-graduation success.
(Oh, and, yes, curriculum maps can also be used to identify the best places to assess the curriculum’s learning goals—typically in courses or other requirements that students typically complete right before graduating. But I don’t think that should be the main purpose of a curriculum map, because you can figure that out without going to the trouble of creating a curriculum map.)
Program curriculum maps with the following traits can best help answer these questions.
Elective courses have no place in a curriculum map. Remember one of the purposes is to ensure that the curriculum is designed to ensure that every student has enough opportunity to achieve every learning goal. Electives don’t help with this analysis.
List program requirements, not program courses. If students can choose from any of four courses to fulfill a particular requirement, for example, group those four courses together and mark only the program learning outcomes that all four courses address.
Codes can help identify if the curriculum has appropriate, progressive rigor. Some assessment management systems require codes indicating whether a learning goal is introduced, developed further, or demonstrated in each course, rather than simply whether it’s addressed in the course.
Check off a course only if students are graded on their progress toward achieving the learning goal. Cast a suspicious eye at courses for which every program learning goal is checked off. How can those courses meaningfully address all those goals?
|Posted on January 31, 2019 at 7:45 AM||comments (5)|
Last year was not one of the best for higher ed assessment. A couple of very negative opinion pieces got a lot of traction among higher ed people who had been wanting to say, “See? Assessment is really as stupid and pointless as I’ve always thought it was.” At some American universities, this was a major setback on assessment progress.
The higher ed assessment community came together quickly with a response that I was proud to contribute to. But now that we’re in 2019, perhaps it would help if each of us in the assessment community reflects on why we’re here. Here’s my story, in three parts, about why assessment is my passion.
The first part is that I’m a data geek, so I find assessment fun. My first job out of grad school was in institutional research, and my favorite part of the job was getting a printout of student survey results and poring over it, trying to find the story in the numbers (to me it’s a treasure hunt), and sharing that story with others in ways that would get them excited about either feeling good about what’s going well or doing something about areas of concern.
The second part is that I love to teach. I’m not a great teacher, but I want to be the best teacher I can. I’ve always looked forward to seeing how my students do on tests and assignments. I can’t wait to tally up how they did on each test question or rubric criterion (that’s the data geek part of me). I cheer the parts they did well on and reflect on the parts where they didn’t. Why did so many miss Question 12? Can I do anything to help them do better, either during what’s left of this class or in the next one? If I can’t figure out what happened, I ask my students at the next class and, trust me, they’re happy to tell me how I screwed up!
The final reason that assessment is my passion is that I’m convinced that part of the answer to the world’s problems today is to help everyone get the best possible education. This dawned on me about 25 years ago, when I was on an accreditation team visiting a seminary. The seminary’s purpose was to educate church pastors (as opposed to, say, researchers or scholars). It was doing a thorough job educating students on doctrine, but there was very little in the curriculum on preparing students to help church members and others hear what Christians call the Good News. There was little attention to helping students develop skills to listen to and counsel church members, communicate with people of diverse backgrounds, and assess community needs, not to mention the practical skills of running a church such as budgeting and fundraising. While I’m not one to push my faith on others, I think the world might be a better place if people truly understood and truly followed the teachings of many faiths. If that’s the case, the world needs pastors well-prepared to do this. The seminary I visited had, I thought, a moral obligation to ensure—through assessment—that its graduates are prepared to be the best possible pastors, with all the skills that pastors need.
Since then, I’ve felt the same about many other colleges and many other disciplines. The world needs great teachers, nurses, lawyers, accountants, and artists. When I’ve visited U.S. service academies, I’m reminded that the U.S. needs great military officers.
Even more, the world needs people who can do all the things we promise in our gen ed curricula. The world needs people who can think critically, who recognize and avoid unethical behavior, who are open to new ideas, who can work with people from diverse backgrounds, who can evaluate the quality of evidence, arguments or claims, who are committed to serving their communities. Again I’m convinced that the world would be a far better place if everyone could do these things well.
None of us can change the world alone. But each of us can do our best with the students in our orbit, trying our best to make sure—through decent-quality assessments—that they’ve really learned what’s most important. Whenever anyone looks at the results of any assessment, be it a class quiz or a college-wide assessment, and uses those results to change what or how they teach, at least some students will get a better education as a result.
We need those better-educated students. This is what drives me. This is why I am devoting my life to helping others learn how to assess student learning. Assessment is one way each of us can help make the world a better place.
|Posted on January 16, 2019 at 7:45 AM||comments (0)|
A recent discussion on the ACCSHE listserv reminded me that setting meaningful benchmarks or standards for student learning assessments remains a real challenge. About three years ago, I wrote a blog post on setting benchmarks or standards for rubrics. Let’s revisit that and expand the concepts to assessments beyond rubrics.
The first challenge is vocabulary. I’ve seen references to goals, targets, benchmarks, standards, thresholds. Unfortunately, the assessment community doesn’t yet have a standard glossary defining these terms (although some accreditors do). I now use standard to describe what constitutes minimally acceptable student performance (such as the passing score on a test) and target to describe the proportion of students we want to meet that standard. But my vocabulary may not match yours or your accreditor's!
The second challenge is embedded in that next-to-last sentence. We’re talking about two different numbers here: the standard describing minimally acceptable performance and the target describing the proportion of students achieving that performance level. That makes things even more confusing.
So how do we establish meaningful standards? There are four basic ways. Three are:
1. External standards: Sometimes the standard is set for us by an external body, such as the passing score on a licensure exam.
2. Peers: Sometimes we want our students to do as well as or better than their peers.
3. Historical trends: Sometimes we want our students to do as well as or better than past students.
Much of the time none of these options is available to us, leaving us to set our own standard, what I call a local standard and what others call a competency-based or criterion-referenced standard. Here are the steps to setting a local standard:
Focus on what would not embarrass you. Would you be embarrassed if people found out that a student performing at this level passed your course or graduated from your program or institution? Then your standard is too low. What level do students need to reach to succeed at whatever comes next—more advanced study or a job?
Consider the relative harm in setting the standard too high or too low. A too-low standard means you’re risking passing or graduating students who aren’t ready for what comes next and that you’re not identifying problems with student learning that need attention. A too-high standard may mean you’re identifying shortcomings in student learning that may not be significant and possibly using scarce time and resources to address those relatively minor shortcomings.
When in doubt, set the standard relatively high rather than relatively low. Because every assessment is imperfect, you’re not going to get an accurate measure of student learning from any one assessment. Setting a relatively high bar increases the chance that every student is truly competent on the learning goals being assessed.
If you can, use external sources to help set standards. A business advisory board, faculty from other colleges, or a disciplinary association can all help get you out of the ivory tower and set defensible standards.
Consider the assignment being assessed. Essays completed in a 50-minute class are not going to be as polished as papers created through scaffolded steps throughout the semester.
Use samples of student work to inform your thinking. Discuss with your colleagues which seem unacceptably poor, which seem adequate though not stellar, and which seem outstanding, then discuss why.
If you are using a rubric to assess student learning, the standard you’re setting is the rubric column (performance level) that defines minimally acceptable work. This is the most important column in the rubric and, not coincidentally, the hardest one to complete. After all, you’re defining the borderline between passing and failing work. Ideally, you should complete this column first, then complete the remaining columns.
Now let’s turn from setting standards to setting targets for the proportions of students who achieve those standards. Here the challenge is that we have two kinds of learning goals. Some are essential. We want every college graduate to write a coherent, grammatically correct paragraph, for example. I don’t want my tax returns prepared by an accountant who can complete them correctly only 70% of the time, and I don’t want my prescriptions filled by a pharmacist who can fill them correctly only 70% of the time! For these essential goals, we want close to 100% of students meeting our standard.
Then there are aspirational goals, which not everyone need achieve. We may want college graduates to be good public speakers, for example, but in many cases graduates can lead successful lives even if they’re not. For these kinds of goals, a lower target may be appropriate.
Tests and rubrics often assess a combination of essential and aspirational goals, which suggests that overall test or rubric scores often aren’t very helpful in understanding student learning. Scores for each rubric trait or for each learning objective in the test blueprint are often much more useful.
Bottom line here: I have a real problem with people who say their standard or target is 70%. It’s inevitably an arbitrary number with no real rationale. Setting meaningful standards and targets is time-consuming, but I can think of few tasks that are more important, because it’s what help ensure that students truly learn what we want them to…and that’s what we’re all about.
By the way, my thinking here comes primarily from two sources: Setting Performance Standards by Cizek and a review of the literature that I did a couple of years ago for a chapter on rubric development that I contributed to the https://www.amazon.com/Handbook-Measurement-Assessment-Evaluation-Education/dp/1138892157" target="_blank">Handbook on Measurement, Assessment, and Evaluation in Higher Education. For a more thorough discussion of the ideas here, see Chapter 22 (Setting Meaningful Standards and Targets) in the new 3rd edition of my book Assessing Student Learning: A Common Sense Guide.
|Posted on December 19, 2018 at 10:55 AM||comments (1)|
One of my treats this time of year is getting the latest annual report from the National Survey of Student Engagement. I’m an enormous fan of this survey. One reason is that it’s research-based: the questions are all about practices that research has shown help students learn and succeed. Another is that, because the questions mostly ask about specific experiences rather than satisfaction, the results are “actionable”: they make clear what institutions need to do to improve student learning and success.
I’m also a fan of NSSE because of its staff and the time, energy, and thought they’ve put into validating the survey and making the results as relevant and useful as possible.
But I’m also struck by how many institutions I encounter who still aren’t using NSSE results to meaningful ways. If an institution isn’t making good use of NSSE results, what hope is there for it using its student learning assessment results?
I’ve done presentations on why assessment results aren’t used…and keep in mind that’s a different question than why assessment isn’t getting done. There are a lot of potential reasons, some of which may apply to your institution and some of which may not. Several years ago I wrote a blog post that highlighted four possible reasons:
- Change must be part of academic culture.
- Institutional leaders must commit to and support evidence-based change.
- We don’t have a clear sense of what satisfactory results are and aren’t.
- Assessment results must be shared clearly and readily.
Most of you reading this aren’t empowered to do much about #1 and #2, and I’ve written a blog post on #3. So let’s focus on #4: Assessment results must be shared clearly and readily. Here are some suggestions:
“Most information is useless. Give yourself permission to dismiss it” (Harris & Muchin, 2002). I think one of the barriers to using NSSE is the sheer volume of information it yields, not to mention the myriad opportunities to slice and dice that information. Before you share NSSE results with anyone, ask yourself, “What are the three most important things I want people to learn from these results?” Here’s an example:
- In many respects, our students are engaging in their learning more than students at peer institutions.
- Our first-year students’ study time is declining.
- Our seniors have fewer capstone experiences than their peers.
That’s plenty for people to chew on! And note that there’s a combination of good news and bad news—it’s not all doom-and-gloom.
Share only what people are willing to act upon. If your institutional community is unwilling to rethink its senior capstone experiences, for example, is it worth sharing NSSE results on those experiences?
Different people need different results. When I've shared NSSE results, I've prepared separate summaries for faculty, for student affairs staff, and for admissions staff (they got all the good news). Know what decisions each group is facing and share only results that will help inform those decisions.
Share a story with a clear point. Give every table, graph, and bulleted list a title that is a sentence that conveys the point of the table. The three points I listed above would make great titles for graphs or bulleted lists.
Consider sharing results through a live slide presentation. After too many years generating reports that no one looked at, I stopped writing reports and instead put key results on PowerPoint slides. Then I invited myself to various meetings to share those slides. This virtually ensured that the results would be at least discussed, if not used. It also forced me to keep the slides and my remarks short and very focused, because my time on the agenda was limited.
Use graphs rather than tables. You want the point to pop out at your audience. NSSE’s website has numerous examples of good visual presentations of results.
Make results easy to find and access. If you put your results on a web page, for example, you’ll need strategies to draw your audience to the web page (Jankowski et al, 2012).
For more ideas on sharing assessment results, see Chapter 25 in my new 3rd edition of Assessing Student Learning: A Common Sense Guide.
|Posted on November 13, 2018 at 6:50 AM||comments (9)|
I’m mystified by how Bloom’s taxonomy has pervaded the higher education assessment landscape. I’ve met faculty who have no idea what a rubric or a test blueprint or a curriculum map is, but it’s been burned into their brains that they must follow Bloom’s taxonomy when developing learning goals. This frustrates me no end, because I don’t think Bloom’s is the best framework for considering learning outcomes in higher education.
Bloom’s taxonomy of educational objectives is probably older than you are. It was developed by Benjamin Bloom in the 1950s. It divides learning goals into three domains: cognitive, affective (attitudinal), and psychomotor. Within the cognitive domain, it has six levels. Originally these were knowledge, comprehension, application, analysis, synthesis, and evaluation. A 2000 update renamed these levels and swapped the positions of the last two: remember, understand, apply, analyze, evaluate, and create. The last four levels are called higher order thinking skills because they require students to do more than understand.
So why don’t I like Bloom’s? One reason is that I’ve seen too many faculty erroneously view the six cognitive levels as hierarchy of prerequisites. Faculty have told me, for example, that first-year courses can only address knowledge and comprehension because students must thoroughly understand a subject before they can begin to think about it. Well, any elementary school teacher can tell you that’s bunk, but the misperception persists.
Even more important is that Bloom’s doesn’t highlight many of the skills and dispositions needed today. Teamwork, ethical judgment, professionalism, and metacognition are all examples of learning goals that don’t fit neatly into Bloom’s. That’s because they’re a combination of the cognitive and affective domains: what educators such as Costa & Kallick and Marzano and his colleagues call habits of mind.
I’m especially concerned about professionalism: coming to work or class on time, coming to work or class prepared to work, completing work on time, planning one’s time, giving work one’s best effort, self-evaluating one’s work, etc. Employers very much want these skills, but they get short shrift in Bloom’s.
So what do I recommend instead? In my workshops I suggest five categories of learning goals:
- knowledge and understanding
- career-specific thinking and performance skills
- transferrable thinking and performance skills (the kinds developed in the liberal arts)
- attitudes and values
- habits of mind
But I also like the taxonomies developed by Dee Fink and by Marzano et al.
I wouldn’t expect every course or program to have learning goals in all five of these categories, of course. But I do suggest that no more than half of a course or program’s learning goals be in the knowledge and understanding category.
For more information, see Chapter 4 (Learning Goals: Articulating What You Most Want Students to Learn) in the new 3rd edition of my book Assessing Student Learning: A Common Sense Guide.
|Posted on October 27, 2018 at 10:30 AM||comments (0)|
Collaborative learning, better known as group work, is an important way for students to learn. Some students learn better with their peers than by working alone. And employers very much want employees who bring teamwork skills.
But group work, such as a group presentation, is one of the hardest things for faculty to grade fairly. One reason is that many student groups include some slackers and some overactive eager beavers. When viewing the product of a group assignment—say they’ve been asked to work together to create a website—it can be hard to discern the quality of individual students’ achievements fairly.
Another reason is that group work is often more about performances than products—the teamwork skills each student demonstrates. As I note in Chapter 21 of Assessing Student Learning: A Common Sense Guide, performances such as working in a team or delivering a group presentation are harder to assess than products such as a paper.
In their book Collaborative Learning Techniques: A Handbook for College Faculty, Elizabeth Barkley, Claire Major, and K. Patricia Cross acknowledge that grading collaborative learning fairly and validly can be challenging. But it’s not impossible. Here are some suggestions.
Have clear learning goal(s) for the assignment. If your key learning goal is for students to develop teamwork skills, your assessment strategy will be very different than if your learning goal is for them to learn how to create a well-designed website.
Make sure your curriculum includes plenty of opportunities for students to develop and achieve your learning goal. If your key learning goal is for students to develop teamwork skills, for example, you’ll need to provide lessons, classwork, and homework that helps them learn what good and poor teamwork skills are and to practice those skills. Just putting students into a group and letting them fend for themselves won’t cut it—students will just keep using whatever bad teamwork habits they brought with them.
Deal with the slackers--and the overactive eager beavers--proactively. Barkley, Major and Cross suggest several ways to do this. Design a group assignment in which each group member must make a discrete contribution for which they’re held accountable. Make these contributions equitable, so all students must participate evenly. Make clear to students that they’ll be graded for their own contribution as well as for the overall group performance or product. And check in with each group periodically and, if necessary, speak individually with any slackers and also those eager beavers who try to do everything themselves.
Consider observing student groups working together. This isn’t always practical, of course—your presence may stifle the group’s interactions—but it’s one way to assess each student’s teamwork skills. Use a rubric to record what you see. Since you’re observing several students simultaneously, keep the rubric simple enough to be manageable—maybe a rating scale rubric or a structured observation guide, both of which are discussed in the rubrics chapter of Assessing Student Learning.
Consider asking students to rate each other. Exhibit 21.1 in Assessing Student Learning is a rating scale rubric I’ve used for this purpose. I tell students that their groupmates’ ratings of them will be averaged and be 5% of their final grade. I weight peer ratings very low because I don’t want students’ grades to be advantaged or disadvantaged by any biases of their peers.
Give each student two grades: one grade for the group product or performance and one for his or her individual contribution to it. This only works when it’s easy to discern each student’s contribution. You can weight the two grades however you like—perhaps equally, or perhaps weighting the group product or performance more heavily than individual contributions, or vice versa.
Give the group a total number of points, and let them decide how to divide those points among group members. Some faculty have told me they’ve used this approach and it works well.
Barkley, Major and Cross point out that there’s a natural tension between promoting collaborative learning and teamwork and assigning individual grades. Whatever approach you choose, try to minimize this tension as much as you can.
|Posted on September 23, 2018 at 10:35 AM||comments (0)|
A recent Inside Higher Ed piece, “The Contamination of Student Assessment” by Jay Sterling Silver, argued that behaviors such as class attendance and class participation shouldn’t be factored into grades because grades should be “unadulterated measurements of knowledge and skills that we represent them to be—and that employers and graduate admissions committees rely on them to be.” In other words, these behaviors are unrelated to key learning goals.
He’s got a point; a grade should reflect achievement of key learning goals. (That’s what competency-based education tries to achieve, as I discussed in a blog several years ago.) But I behaviors like coming to class, submitting work on time, giving assignments one’s best effort, and participating in class discussions are important. They fall under what I call professionalism: traits that include coming to work on time and prepared to work, dependably completing assigned work thoroughly and on time, giving one’s work one’s best effort, and managing one’s time.
Surveys of employers confirm that these are important traits in the people they hire. Every few years, for example, Hart Research Associates conducts a survey for AAC&U on how well employers think college graduates are prepared on a number of key learning outcomes. The 2018 survey added two learning outcomes that weren’t in previous surveys:
- Self-motivated: ability to take initiative and be proactive
- Work independently: set priorities, manage time and deadlines
Of the 15 learning outcomes in the 2018 survey, these were tied for #4 in importance by hiring managers.
So I think the answer is to add professionalism as an additional learning goal. Of course “professionalism” isn’t a well-stated learning goal; it’s a category. I leave it up to college and program faculty to decide how best to articulate what forms of professionalism are most important to their students and prospective employers.
Then an assignment like a library research paper might have three learning goals—information literacy, writing, and professionalism—and be graded on all three. Professionalism might be demonstrated by how well students followed the directions, whether the assignment was turned in on time, and whether the student went above and beyond the bare minimum requirements for the assignment.
Professionalism, by the way, isn’t just a skill and isn’t just an attitude. It’s a combination of both, similar to what Arthur Costa and Bea Kallick call habits of mind, which include things like persisting, managing impulsivity, taking responsible risks, and striving for accuracy. One of the reasons I’m not a fan of Bloom’s taxonomy is that it doesn’t really address habits of mind, which—as evidenced by Hart’s new survey—are becoming increasingly important learning goals of a college education.
|Posted on September 2, 2018 at 8:25 AM||comments (2)|
In a recent guest post in Inside Higher Ed, “What Students See in Rubrics,” Denise Krane explained her dissatisfaction with rubrics, which can be boiled down to this statement toward the end of her post, “Ideally, rubrics are assignment specific.”
I don’t know where Denise got this idea, but it’s flat-out wrong. As I’ve mentioned in previous blog posts on rubrics, a couple of years ago I conducted a literature review for a chapter on rubric development that I wrote for the second edition of the Handbook of Measurement, Assessment, and Evaluation in Higher Education. The rubric experts I found (for example, Brookhart; Lane; Linn, Baker & Dunbar; and Messick) are unanimous in advocating what they call general rubrics over what they call task-specific rubrics: rubrics that assess achievement of the assignment’s learning outcomes rather than achievement of the task at hand.
Their reason is exactly what Denise advocates: we want students to focus on long-term, deep learning—in the case of writing, to develop the tools to, as Denise says, grapple with writing in general. Indeed, some experts such as Lane posit that one of the criteria of a valid rubric is its generalizability: it should tell you how well students can write (or think, or solve problems) across a range of tasks, not just the one being assessed. If you use a task-specific rubric, students will learn how to do that one task but not much more. If you use a general rubric, students will learn skills they can use in whole families of tasks.
To be fair, the experts also caution against general rubrics that are too general, such as one writing rubric used to assess student work in courses and programs across an entire college. Many experts (for example, Cooper, Freedman, Lane, and Lloyd-Jones) suggest developing rubrics for families of related assignments—perhaps one for academic writing in the humanities and another for business writing. This lets the rubric include discipline-specific nuances. For example, academic writing in the humanities is often expansive, while business writing must be succinct.
How do you move from a task-specific rubric to a general rubric? It’s all about the traits being assessed—those things listed on the left side of the rubric. Those things should be traits of the learning outcomes being assessed, not the assignment. So instead of listing each element of the assignment (I’ve seen rubrics that literally list “opening paragraph,” “second paragraph,” and so on ), list each key trait of the learning goals. When I taught writing, for example, my rubric included traits like focus, organization, and sentence structure.
Over the last few months I’ve worked with a lot of faculty on creating rubrics, and I’ve seen that moving from a task-specific to a general rubric can be remarkably difficult. One reason is that faculty want students to complete the assignment correctly: Did they provide three examples? Did they cite five sources? If this is important, I suggest making “Following directions” one of the learning outcomes of the assignment and including it as a trait assessed by the rubric. Then create a separate checklist of all the components of the assignment. Ask students to complete the checklist themselves before submitting the assignment. Also consider asking students to pair up and complete checklists for each other’s assignments.
To identify the other traits assessed by the rubric, ask yourself, “What does good writing/problem solving/critical thinking/presenting look like? Focus not on this assignment but on why you’re giving students the assignment. What you want them to learn from this assignment that they can use in subsequent courses or after they graduate?
Denise mentioned two other things about rubrics that I’d also like to address. She surveyed her students about their perceptions of rubrics, and one complaint was that faculty expectations vary from one professor to another. The problem here is lack of collaboration. Faculty teaching sections of the same course--or related courses--should collaborate on a common rubric that they all use to grade student work. This lets students work on the same important skill over and over again in varying course contexts and see connections in their learning. If one professor wants to emphasize something above and beyond the common rubric, fine. The common elements can be the top half of the rubric, and the professor-specific elements can be the bottom half.
Denise also mentioned that her rubric ran three pages, and she hated. I would too! Long rubrics focus on the trees rather than the forest of what we’re trying to help students learn. A shorter rubric (I recommend that rubrics fit on one page) focuses students on the most important things they’re supposed to be learning. If it frustrates you that your rubric doesn’t include everything you want to assess, keep in mind that no assessment can assess everything. Even a comprehensive final exam can’t ask every conceivable question. Just make sure that your rubric, like your exam, focuses on the most important things you want students to learn.
If you’re interested in a deeper dive into what I learned about rubrics, here are some of my past blog posts. My book chapter in the Handbook has the full citations of the authors I've mentioned here.
|Posted on August 14, 2018 at 8:50 AM||comments (1)|
A while back, a faculty member teaching in a community college career program told me, “I don’t need to assess. I know what my students are having problems with—math.”
Well, maybe so, but I’ve found that my perceptions often don’t match reality, and systematic evidence gives me better insight. Let me give you a couple of examples.
Example #1: you may have noticed that my website blog page now has an index of sorts on the right side. I created it a few months ago, and what I found really surprised me. I aim for practical advice on the kinds of assessment issues that people commonly face. Beforehand I’d been feeling pretty good about the range and relevance of assessment topics that I’d covered. The index showed that, yes, I’d done lots of posts on how to assess and specifically on rubrics, a pet interest of mine. I was pleasantly surprised by the number of posts I’d done on sharing and using results.
But what shocked me was how little I’d written on assessment culture: only four posts in five years! Compare that with seventeen posts on curriculum design and teaching. Assessment culture is an enormous issue for assessment practitioners. Now knowing the short shrift I’d been giving it, I’ve written several more blog posts related to assessment culture, bring the total to ten (including this post).
(By the way, if there’s anything you’d like to see a blog post on, let me know!)
Example #2: Earlier this summer I noticed that some of the flowering plants in my backyard weren’t blooming much. I did a shade study: one sunny day when I was home all day, every hour I made notes on which plants were in sun and which were in shade. I’d done this about five years ago but, as with the blog index, the results shocked me; some trees and shrubs had grown a lot bigger in five years and consequently some spots in my yard were now almost entirely in shade. No wonder those flowers didn’t bloom! I’ll be moving around a lot of perennials this fall to get them into sunnier spots.
So, yes, I’m a big fan of using systematic evidence to inform decisions. I’ve seen too often that our perceptions may not match reality.
But let’s go back to that professor whose students were having problems with math and give him the benefit of the doubt—maybe he’s right. My question to him was, “What are you doing about it?” The response was a shoulder shrug. His was one of many institutions with an assessment office but no faculty teaching-learning center. In other words, they’re investing more in assessment than in teaching. He had nowhere to turn for help.
My point here is that assessment is worthwhile only if the results are used to make meaningful improvements to curricula and teaching methods. Furthermore, assessment work is worthwhile only if the impact is in proportion to the time and effort spent on the assessment. I recently worked with an institution that undertook an elaborate assessment of three general education learning outcomes, in which student artifacts were sampled from a variety of courses and scored by a committee of trained reviewers. The results were pretty dismal—on average only about two thirds of students were deemed “proficient” on the competencies’ traits. But the institutional community is apparently unwilling to engage with this evidence, so nothing will be done beyond repeating the assessment in a couple of years. Such an assessment is far from worthwhile; it’s a waste of everyone’s time.
This institution is hardly alone. When I was working on the new 3rd edition of my book Assessing Student Learning: A Common Sense Guide, I searched far and wide for examples of assessments whose results led to broad-based change and found only a handful. Overwhelmingly, the changes I see are what I call minor tweaks, such as rewriting an assignment or adding more homework. These changes can be good—collectively they can add up to a sizable impact. But the assessments leading to these kinds of changes are worthwhile only if they’re very simple, quick assessments in proportion to the minor tweaks they bring about.
So is assessment worth it? It’s a mixed bag. On one hand, the time and effort devoted to some assessments aren’t worth it—the findings don’t have much impact. On the other hand, however, I remain convinced of the value of using systematic evidence to inform decisions affecting student learning. Assessment has enormous potential to move us from providing a good education to providing a truly great education. The keys to achieving this are commitments to (1) making that good-to-great transformation, (2) using systematic evidence to inform decisions large and small, and (3) doing only assessments whose impact is likely to be in proportion to the time, effort, and resources spent on them.
|Posted on July 30, 2018 at 8:20 AM||comments (2)|
I often hear questions about how long an “assessment cycle” should be. Fair warning: I don’t think you’re going to like my answer.
The underlying premise of the concept of an assessment cycle is that assessment of key program, general education, or institutional learning goals is too burdensome to be completed in its entirety every year, so it’s okay for assessments to be staggered across two or more years. Let’s unpack that premise a bit.
First, know that if an accreditor finds an institution or program out of compliance with even one of its standards—including assessment—Federal regulations mandate that the accreditor can give the institution no more than two years to come into compliance. (Yes, the accreditor can extend those two years for “good cause,” but let’s not count on that.) So an institution that has done nothing with assessment has a maximum of two years to come into compliance, which often means not just planning assessments but conducting them, analyzing the results, and using the results to inform decisions. I’ve worked with institutions in this situation and, yes, it can be done. So an assessment cycle, if there is one, should generally run no longer than two years.
Now consider the possibility that you’ve assessed an important learning goal, and the results are terrible. Perhaps you learn that many students can’t write coherently, or they can’t analyze information or make a coherent argument. Do you really want to wait two, three, or five years to see if subsequent students are doing better? I’d hope not! I’d like to see learning goals with poor results put on red alert, with prompt actions so students quickly start doing better and prompt re-assessments to confirm that.
Now let’s consider the premise that assessments are too burdensome for them all to be conducted annually. If your learning goals are truly important, faculty should be teaching them in every course that addresses them. They should be giving students learning activities and assignments on those goals; they should be grading students on those goals; they should be reviewing the results of their tests and rubrics; and they should be using the results of their review to understand and improve student learning in their courses. So, once things are up and running, there really shouldn’t be much extra burden in assessing important learning goals. The burdens are cranking out those dreaded assessment reports and finding time to get together with colleagues to review and discuss the results collaboratively. Those burdens are best addressed by minimizing the work of preparing those reports and by helping faculty carve out time to talk.
Now let’s consider the idea that an assessment cycle should stagger the goals being assessed. That implies that every learning goal is discrete and that it needs its own, separate assessment. In reality, learning goals are interrelated; how can one learn to write without also learning to think critically? And we know that capstone assignments—in which students work on several learning goals at once—are not only great opportunities for students to integrate and synthesize their learning but also great assessment opportunities, because we can look at student achievement of several learning goals all at once.
Then there’s the message we send when we tell faculty they need to conduct a particular assessment only once every three, four, or five years: assessment is a burdensome add-on, not part of our normal everyday work. In reality, assessment is (or should be) part of the normal teaching-learning process.
And then there are the practicalities of conducting an assessment only once every few years. Chances are that the work done a few years ago will have vanished or at least collective memory will have evaporated (why on earth did we do that assessment?). Assessment wheels must be reinvented, which can be more work than tweaking last year’s process.
So should assessments be conducted on a fixed cycle? In my opinion, no. Instead:
- Use capstone assignments to look at multiple goals simultaneously.
- If you’re getting started with assessment, assess everything, now. You’ve been dragging your feet too long already, and you’re risking an accreditation action. Remember you must not only have results but be using them within two years.
- If you’ve got disappointing results, move additional assessments of those learning goals to a front burner, assessing them frequently until you get results where you want them.
- If you’ve got terrific results, consider moving assessments of those learning goals to a back burner, perhaps every two years or so, just to make sure results aren’t slipping. This frees up time to focus on the learning goals that need time and attention.
- If assessment work is widely viewed as burdensome, it’s because its cost-benefit is out of whack. Perhaps assessment processes are too complicated, or people view the learning goals being assessed as relatively unimportant, or the results aren’t adding useful insight. Do all you can to simplify assessment work, especially reporting. If people don't find a particular assessment useful, stop doing it and do something else instead.
- If assessment work must be staggered, stagger some of your indirect assessment tools, not the learning goals or major direct assessments. An alumni survey or student survey might be conducted every three years, for example.
- For programs that “get” assessment and are conducting it routinely, ask for less frequent reports, perhaps every two or three years instead of annually. It’s a win-win reward: less work for them and less work for those charged with reviewing and offering feedback on assessment reports.
|Posted on July 15, 2018 at 7:45 AM||comments (7)|
The word “demonstrate” in learning goals raises a red flag for me. Consider these (real) learning goals:
- Demonstrate fundamental business and entrepreneurship skills
- Demonstrate critical and creative thinking.
- Demonstrate information literacy skills.
- Demonstrate teamwork and collaboration.
- Demonstrate ethical self-awareness.
- Demonstrate personal responsibility.
Clearly the people who wrote these learning goals were told that they had to start with an action word. So they plopped the word “demonstrate” in front of a fuzzy goal. But adding “demonstrate” doesn’t make the goal any less fuzzy. What are “fundamental business and entrepreneurship skills”? What is “personal responsibility”? Until these concepts are stated more clearly, these learning goals remain fuzzy and therefore difficult to assess meaningfully.
Now consider these (real) learning goals:
- Demonstrate proficiency in analyzing work-related scenarios, taking appropriate action and evaluating results of the action.
- Demonstrate proficiency in the use of technology for collecting and analyzing information
- Demonstrate the ability to work cooperatively with others
- Demonstrate enhanced competencies in time management
Here the phrase “demonstrate proficiency/ability/competencies” are simply superfluous, making the learning goal unnecessarily wordy. Consider these restatements:
- Analyze work-related scenarios, take appropriate action, and evaluate the results of the action.
- Use technology to collect and analyze information.
- Work cooperatively with others.
- Manage time effectively.
Not only are they clearer but, because they’re shorter, they pack a punch; they have a better chance of engaging students and getting them enthused about their learning.
So should we abolish the word “demonstrate” from our assessment lexicon? Well, consider this (real) learning outcome:
- Demonstrate appropriate pitch, tone and demeanor in professional settings.
If we make clear what we want students to demonstrate, using observable terms, “demonstrate” may be fine.
Now consider these (real) learning outcomes:
- Demonstrate appropriate, professional conduct. (if you define it)
- Demonstrate professionalism and cultural sensitivity while interacting and communicating with others.
It could be argued that these learning outcomes are a bit fuzzy. What is appropriate, professional conduct, after all? What is cultural sensitivity? But if we clarified these terms in the learning outcome, we’d come up with a pretty long list of traits—so many that the learning outcome would be too cumbersome to be effective. In these cases, I’m okay with leaving these learning outcomes as is, provided that the rubrics used to assess them explicate these terms into traits with clear, concrete language that students easily understand.
So, no, I don't think we should abolish the word "demonstrate" altogether, but think twice--or even three times--before using it.
|Posted on June 24, 2018 at 4:30 PM||comments (1)|
A recent paper co-sponsored by AALHE and Watermark identified some key professional development needs of assessment practitioners.
While a book is no substitute for a rich, interactive professional development experience, some of the things that assessment practitioners want to learn about are discussed in my books Assessing Student Learning: A Common Sense Guide (new 3rd edition) and Five Dimensions of Quality: A Common Sense Guide to Accreditation and Accountability. Perhaps they’re a good place to kick off your professional development.
Analyzing and Interpreting Assessment Data
See Chapter 24 (Analyzing Evidence of Student Learning) of Assessing Student Learning (3rd ed.).
Analyzing and Interpreting Qualitative Results
See “Summarizing qualitative evidence” on pages 313-316 of Chapter 23 (Summarizing and Storing Evidence of Student Learning) of Assessing Student Learning (3rd ed.)
Reporting Assessment Results
See Chapter 25 (Sharing Evidence of Student Learning) of Assessing Student Learning (3rd ed.) and Chapter 16 (Transparency: Sharing Evidence Clearly and Readily) of Five Dimensions of Quality.
This is such a big issue that the 3rd edition of Assessing Student Learning devotes six chapters to it. See Part 3, which includes the following chapters:
Chapter 9 (Guiding and Coordinating Assessment Efforts)
Chapter 10 (Helping Everyone Learn What to Do)
Chapter 11 (Supporting Assessment Efforts)
Chapter 12 (Keeping Assessment Cost-Effective)
Chapter 13 (Collaborating on Assessment)
Chapter 14 (Valuing Assessment and the People Who Contribute)
A good place to start is Chapter 14, because it begins with a section titled, “Why is this so hard?” Even better, see the chapter that section summarizes: Chapter 4 (Why Is This So Hard?) of Five Dimensions of Quality.
Also see Chapter 17 (Using Evidence to Ensure and Advance Quality and Effectiveness) in Five Dimensions of Quality.
Culture of Change
See Chapter 18 (Sustaining a Culture of Betterment) of Five Dimensions of Quality along with the aforementioned Chapter 4 (Why Is This So Hard?) in the same book. For a briefer discussion, see “Value innovation, especially in improving teaching” on pages 180-181 of Chapter 14 (Valuing Assessment and the People Who Contribute) of Assessing Student Learning (3rd ed.).
Effective/Meaningful/Best Assessment Practices
See Chapter 3 (What Are Effective Assessment Practices?) of Assessing Student Learning (3rd ed.) and Chapter 14 (Good Evidence Is Useful) of Five Dimensions of Quality.
Co-Curricular Learning Outcomes and Assessment
Information on co-curricula is scattered throughout the new 3rd edition of Assessing Student Learning. See the following:
“Learning goals for co-curricular experiences” on pages 57-58 of Chapter 4 (Learning Goals: Articulating What You Most Want Students to Learn)
“Planning assessments in co-curricula” on pages 110-112 of Chapter 8 (Planning Assessments in Other Settings)
Chapter 20 (Other Assessment Tools)
Chapter 21 (Assessing the Hard-to-Assess)
See Chapter 15 (Designing Rubrics to Plan and Assess Assignments) of Assessing Student Learning (3rd ed.).
See Chapter 22 (Setting Meaningful Standards and Targets) of Assessing Student Learning (3rd ed.) and Chapter 15 (Setting and Justifying Targets for Success) of Five Dimensions of Quality.
See Chapter 20 (Program Reviews: Drilling Down into Programs and Services) of Five Dimensions of Quality.
|Posted on June 10, 2018 at 8:45 AM||comments (2)|
Architecture critic Kate Wagner recently said, “All buildings are interesting. There is not a single building that isn’t interesting in some way.” I think we can say the same thing about assessment: All assessment is interesting. There is not a single assessment that isn’t interesting in some way.
Kate points out that what makes seemingly humdrum buildings interesting are the questions we can ask about them—in other words, how we analyze them. She suggests a number of questions that can be easily adapted to assessment:
- How do these results compare to other assessment results? We can compare results against results for other students (at our institution or elsewhere), against results for other learning goals, against how students did when they entered (value-added), against past cohorts of students, or against an established standard. Each of these comparisons can be interesting. (See Chapter 22 of my book Assessing Student Learning for more information on perspectives for comparing results.)
- Are we satisfied with the results? Why or why not?
- What do these results say about our students at this time? Students, curricula, and teaching methods are rapidly changing, which makes them--and assessment--interesting. Assessment results are a piece of history: what students learned (and didn’t learn) at this time, in this setting.
- What does this assessment say about what we and our institution value? What does it say about the world in which we live?
Why do so many faculty and staff fail to find assessment interesting? I’ve alluded to a number of possible reasons in past blog posts (such as here and here), but let me throw out a few that I think are particularly relevant.
1. Sometimes assessment simply isn’t presented as something that’s supposed to be interesting. It’s a chore to get through accreditation, nothing more. Just as Kate felt obliged to point out that even humdrum buildings are interesting, sometimes faculty and staff need to be reminded that assessment should be designed to yield interesting results.
2. Sometimes faculty and staff aren’t particularly interested in the learning goal being assessed. If a faculty member focuses on basic conceptual understanding in her course, she’s not going to be particularly interested in the assessment of critical thinking that she's obliged to do. Rethinking key learning goals and helping faculty and staff rethink their curricula can go a long way toward generating assessment results that faculty and staff find interesting.
3. Some faculty and staff find results mildly interesting, but not interesting enough to be worth all the time and effort that’s gone into generating them. A complex, time-consuming assessment whose results show that students are generally doing fine and are not all that different from past years is interesting but not terribly interesting. The cost-benefit isn’t there. Here the key is to scale back less-interesting assessments—maybe repeat the assessment every two or three years just to make sure results aren’t slipping—and focus on assessments that faculty and staff will find more interesting and useful.
4. Some faculty and staff aren’t really that interested in teaching—they’re far more engaged with their research agenda. And some faculty and staff aren’t really that interested in improving their teaching. Institutional leaders can help here by rethinking incentives and rewards to encourage faculty and staff to try to improve their teaching.
Kate says, “All of us have the potential to be nimble interpreters of the world around us. All we need to do is look around.” Similarly, all of us have the potential to be nimble interpreters of evidence of student learning. All we need to do is use the analytical skills we learned in college and teach to our students to find what's interesting.
|Posted on May 27, 2018 at 7:40 AM||comments (0)|
When I help faculty and co-curricular staff move ahead with their assessment efforts, I probably spend half our time on helping them articulate their learning goals. As the years have gone by, I’ve become ever more convinced that learning goals are the foundation of an assessment structure…and without a solid foundation, a structure can’t be well-constructed.
So what are well-stated learning goals? They have the following characteristics:
They are outcomes: what students will be able to do after they successfully complete the learning experience, not what they will do or learn during the learning experience. Example: Prepare effective, compelling visual summaries of research.
They are clear, written in simple, jargon-free terms that everyone understands, including students, employers, and colleagues in other disciplines. Example: Work collaboratively with others.
They are observable, written using action verbs, because if you can see it, you can assess it. Example: Identify and analyze ethical issues in the discipline.
They focus on skills more than knowledge, conceptual understanding, or attitudes and values, because thinking and performance skills are what employers seek in new hires. I usually suggest that at least half the learning goals of any learning experience focus on skills. Example: Integrate and properly cite scientific literature.
They are significant and aspirational: things that will take some time and effort for students to learn and that will make a real difference in their lives. Example: Identify, articulate, and solve problems in [the discipline or career field].
They are relevant, meeting the needs of students, employers, and society. They focus more on what students need to learn than what faculty want to teach. Example: Interpret numbers, data, statistics, and visual representations of them appropriately.
They are short and therefore powerful. Long, qualified or compound statements get everyone lost in the weeds. Example: Treat others with respect.
They fit the scope of the learning activity. Short co-curricular learning experiences have narrower learning goals than an entire academic program, for example.
They are limited in number. I usually suggest no more than six learning goals per learning experience. If you have 10, 15, or 20 learning goals—or more—everyone focuses on trees rather than the forest of the most important things you want students to learn.
They help students achieve bigger, broader learning goals. Course learning goals help students achieve program and/or general education learning goals; co-curricular learning goals help students achieve institutional learning goals; program learning goals help students achieve institutional learning goals.
For more information on articulating well-stated learning goals, see Chapter 4 of the new 3rd edition of my book Assessing Student Learning: A Common Sense Guide.
|Posted on May 2, 2018 at 6:55 AM||comments (0)|
I look on learning goals as promises that we make to students, employers, and society: If a student passes a course or graduates, he or she WILL be able to do the things we promise in our learning goals.
But there are some things we hope to instill in students that we can’t guarantee. We can’t guarantee, for example, that every graduate will be a passionate lifelong learner, appreciate artistic expressions, or make ethical decisions. I think these kinds of statements are important aims that might be expressed in a statement of values, but they’re not really learning goals, because they’re something we hope for, not something we can promise. Because they’re not really learning goals, they’re very difficult if not impossible statements to assess meaningfully.
How can you tell if a learning goal is true learning goal—an assessable promise that we try to keep? Ask yourself the following questions.
Is the learning goal stated clearly, using observable action verbs? Appreciate diversity is a promise we may not be able to keep, but Communicate effectively with people from diverse backgrounds is an achievable, assessable learning goal.
How have others assessed this learning goal? If someone else has assessed it meaningfully and usefully, don’t waste time reinventing the wheel.
How would you recognize people who have achieved this learning goal? Imagine that you run into two alumni of your college. As you talk with them, it becomes clear that one appreciates artistic expressions and the other doesn’t. What might they say about their experiences and views that would lead you to that conclusion? This might give you ideas on ways to express the learning goal in more concrete, observable terms, which makes it easier to figure out how to assess it.
Is the learning goal teachable? Ask faculty who aim to instill this learning goal to share how they help students achieve it. If they can name specific learning activities, the goal is teachable—and assessable, because they can grade the completed learning activities. But if the best they can say is something like, “I try to model it” or “I think they pick it up by osmosis,” the goal may not be teachable—or assessable. Don’t try to assess what can’t be taught.
What knowledge and skills are part of this learning goal? We can’t guarantee, for example, that all graduates will make ethical decisions, but we can make sure that they recognize ethical and unethical decisions, and we can assess their ability to do so.
How important is this learning goal? Most faculty and colleges I work with have too many learning goals—too many to assess well and, more important, too many to help students achieve well in the time we have with them. Ask yourself, “Can our students lead happy and fulfilling lives if they graduate without having achieved this particular learning goal?”
But just because a learning goal is a promise we can’t keep doesn’t mean it isn’t important. A world in which people fail to appreciate artistic expressions or have compassion for others would be a dismal place. So continue to acknowledge and value hard-to-assess learning goals even if you’re not assessing them.
For more information on assessing the hard-to-assess, see Chapter 21 of the new 3rd edition of Assessing Student Learning: A Common Sense Guide.
|Posted on March 28, 2018 at 6:25 AM||comments (1)|
In my February 28 blog post, I noted that many faculty express frustration with assessment along the following lines:
- What I most want students to learn is not what’s being assessed.
- I’m being told what and how to assess, without any input from me.
- I’m being told what to teach, without any input from me.
- I’m being told to assess skills that employers want, but I teach other things that I think are more important.
- A committee is doing a second review of my students’ work. I’m not trusted to assess student work fairly and accurately through my grading processes.
- I’m being asked to quantify student learning, but I don’t think that’s appropriate for what I’m teaching.
- I’m being asked to do this on top of everything else I’m already doing.
- Assessment treats learning as a scientific process, when it’s a human endeavor; every student and teacher is different.
The underlying theme here is that these faculty don’t feel that they and their views are valued and respected. When we value and respect people:
- We design assessment processes so the results are clearly useful in helping to make important decisions, not paper-pushing exercises designed solely to get through accreditation.
- We make assessment work worthwhile by using results to make important decisions, such as on resource allocations, as discussed in my March 13 blog post.
- We truly value great teaching and actively encourage the scholarship of teaching as a form of scholarship.
- We truly value innovation, especially in improving one’s teaching because, if no one wants to change anything, there’s no point in assessing.
- We take the time to give faculty and staff clear guidance and coordination, so they understand what they are to do and why.
- We invest in helping them learn what to do: how to use research-informed teaching strategies as well as how to assess.
- We support their work with appropriate resources.
- We help them find time to work on assessment and to keep assessment work cost-effective, because we respect how busy they are.
- We take a flexible approach to assessment, recognizing that one size does not fit all. We do not mandate a single institution-wide assessment approach but instead encourage a variety of assessment strategies, both quantitative and qualitative. The more choices we give faculty, the more they feel empowered.
- We design assessment processes so faculty are leaders rather than providers of assessment. We help them work collaboratively rather than in silos, inviting them to contribute to decisions on what, why, and how we assess. We try to assess those learning outcomes that the institutional community most values. More than anything else, we spend more time listening than telling.
- We recognize and honor assessment work in tangible ways, perhaps through a celebratory event, public commendations, or consideration in promotion, tenure, and merit pay applications.
For more information on these and other strategies to value and respect people who work on assessment, see Chapter 14, “Valuing Assessment and the People Who Contribute,” in the new third edition of my book Assessing Student Learning: A Common Sense Guide.
|Posted on March 13, 2018 at 9:50 AM||comments (9)|
In my February 28 blog post, I noted that many faculty have been expressing frustration that assessment is a waste of an enormous amount of time and resources that could be better spent on teaching. Here are some strategies to help make sure your assessment activities are meaningful and cost-effective, all drawn from the new third edition of Assessing Student Learning: A Common Sense Guide.
Don’t approach assessment as an accreditation requirement. Sure, you’re doing assessment because your accreditor requires it, but cranking out something only to keep an accreditor happy is sure to be viewed as a waste of time. Instead approach assessment as an opportunity to collect information on things you and your colleagues care about and that you want to make better decisions about. Then what you’re doing for the accreditor is summarizing and analyzing what you’ve been doing for yourselves. While a few accreditors have picky requirements that you must comply with whether you like them or not, most want you to use their standards as an opportunity to do something genuinely useful.
Keep it useful. If an assessment hasn’t yielded useful information, stop doing it and do something else. If no one’s interested in assessment results for a particular learning goal, you’ve got a clue that you’ve been assessing the wrong goal.
Make sure it’s used in helpful ways. Design processes to make sure that assessment results inform things like professional development programming, resource allocations for instructional equipment and technologies, and curriculum revisions. Make sure faculty are informed about how assessment results are used so they see its value.
Monitor your investment in assessment. Keep tabs on how much time and money each assessment is consuming…and whether what’s learned is useful enough to make that investment worthwhile. If it isn’t, change your assessment to something more cost-effective.
Be flexible. A mandate to use an assessment tool or strategy that’s inappropriate for a particular learning goal or discipline is sure to be viewed as a waste of everyone’s time. In assessment, one size definitely does not fit all.
Question anything that doesn’t make sense. If no one can give a good explanation for doing something that doesn’t make sense, stop doing it and do something more appropriate.
Start with what you have. Your college has plenty of direct and indirect evidence of student learning already on hand, from grading processes, surveys, and other sources. Squeeze information out of those sources before adding new assessments.
Think twice about blind-scoring and double-scoring student work. The costs in terms of both time and morale can be pretty steep (“I’m a professional! Why can’t they trust me to assess my own students’ work?” ). Start by asking faculty to submit their own rubric ratings of their own students’ work. Only move to blind- and double-scoring if you see a big problem in their scores of a major assessment.
Start at the end and work backwards. If your program has a capstone requirement, students should be demonstrating achievement in many key program learning goals in it. Start assessment there. If students show satisfactory achievement of the learning goals, you’re done! If you’re not satisfied with their achievement of a particular learning goal, you can drill down to other places in the curriculum that address that goal.
Help everyone learn what to do. Nothing galls me more than finding out what I did wasn’t what was wanted and has to be redone. While we all learn from experience and do things better the second time, help everyone learn what to do so, their first assessment is a useful one.
Minimize paperwork and bureaucratic layers. Faculty are already routinely assessing student learning through the grading process. What some resent is not the work of grading but the added workload of compiling, analyzing, and reporting assessment evidence from the grading process. Make this process as simple, intuitive, and useful as possible. Cull from your assessment report template anything that’s “nice to know” versus absolutely essential.
Make assessment technologies an optional tool, not a mandate. Only a tiny number of accreditors require using a particular assessment information management system. For everyone else, assessment information systems should be chosen and implemented to make everyone’s lives easier, not for the convenience of a few people like an assessment committee or a visiting accreditation team. If a system is hard to learn, creates more work, or is expensive, it will create resentment and make things worse rather than better. I recently encountered one system for which faculty had to tally and analyze their results, then enter the tallied results into the system. Um, shouldn’t an assessment system do the work of tallying and analysis for the faculty?
Be sensible about staggering assessments. If students are not achieving a key learning goal well, you’ll want to assess it frequently to see if they’re improving. But if students are achieving another learning goal really well, put it on a back burner, asking for assessment reports on it only every few years, to make sure things aren’t slipping.
Help everyone find time to talk. Lots of faculty have told me that they “get” assessment but simply can’t find time to discuss with their colleagues what and how to assess and how best to use the results. Help them carve out time on their calendars for these important conversations.
Link your assessment coordinator with your faculty teaching/learning center, not an accreditation or institutional effectiveness office. This makes clear that assessment is about understanding and improving student learning, not just a hoop to jump through to address some administrative or accreditation mandate.
|Posted on March 4, 2018 at 8:05 AM||comments (1)|
The vitriol in some recent op-ed pieces and the comments that followed them might leave the impression that faculty hate assessment. Well, some faculty clearly do, but a national survey suggests that they’re in the minority.
The Faculty Survey of Assessment Culture, directed by Dr. Matthew Fuller at Sam Houston State University, can give us some insight. Its key drawback is, because it’s still a relatively nascent survey, it has only about 1155 responses from its last reported administration in 2014. So the survey may not represent what faculty throughout the U.S. really think, but I nonetheless think it’s worth a look.
Most of the survey is a series of statements to which faculty respond by choosing Strongly Agree, Agree, Only Slightly Agree, Only Slightly Disagree, Disagree, or Strongly Disagree.
Here are the percentages who agreed or strongly agreed with each statement. Statements that are positive about assessment are in green; those that are negative about assessment are in red.
80% The majority of administrators are supportive of assessment.
77% Faculty leadership is necessary for my institution’s assessment efforts.
76% Assessment is a good thing for my institution to do.
70% I am highly interested in my institution’s assessment efforts.
70% Assessment is vital to my institution’s future.
67% In general I am eager to work with administrators.
67% Assessment is a good thing for me to do.
64% I am actively engaged in my institution’s assessment efforts.
63% Assessments of programs are typically connected back to student learning
62% My academic department or college truly values faculty involvement in assessment.
61% I engage in institutional assessment efforts because it is the right thing to do for our students.
60% Assessment is vital to my institution’s way of operating.
57% Discussions about student learning are at the heart of my institution.
57% In general a recommended change is more likely to be enacted by administrators if it is supported by assessment data.
53% I clearly understand assessment processes at my institution.
52% Assessment supports student learning at my institution.
51% Assessment is primarily the responsibility of faculty members.
51% Change occurs more readily when supported by assessment results.
50% It is clear who is ultimately in charge of assessment.
50% I am familiar with the office that leads student assessment efforts for accreditation purposes.
50% Assessment for accreditation purposes is prioritized above other assessment efforts.
49% Assessment results are used for improvement.
49% The majority of administrators primarily emphasize assessment for the improvement of student learning.
49% I engage in institutional assessment because doing so makes a difference to student learning at my institution.
48% Assessment processes yield evidence of my institution’s effectiveness.
48% I have a generally positive attitude toward my institution’s culture of assessment.
47% Senior leaders, i.e., President or Provost, have made clear their expectations regarding assessment.
47% Administrators are supportive of making changes.
46% I am familiar with the office that leads student assessment efforts for student learning.
45% Assessment data are used to identify the extent to which student learning outcomes are met.
44% My institution is structured in a way that facilitates assessment practices focused on improved student learning.
44% The majority of administrators only focus on assessment in response to compliance requirements.
43% Student assessment results are shared regularly with faculty members.
41% I support the ways in which administrators have used assessment on my campus.
40% Assessment is an organized coherent effort at my institution.
40% Assessment results are available to faculty by request.
38% Assessment data are available to faculty by request.
37% Assessment results are shared regularly throughout my institution.
35% Faculty are in charge of assessment at my institution.
33% Engaging in assessment also benefits my research/scholarship agenda.
32% Budgets can be negatively impacted by assessment results.
32% Administrators share assessment data with faculty members using a variety of communication strategies (i.e., meetings, web, written correspondence, presentations).
31% Assessment data are regularly used in official institutional communications.
30% There are sufficient financial resources to make changes at my institution.
29% Assessment is a necessary evil in higher education.
28% Communication of assessment results has been effective.
28% Assessment results are criticized for going nowhere (i.e., not leading to change).
27% Assessment results in a fair depiction of what I do as a faculty member.
27% Administrators use assessment as a form of control (i.e., to regulate institutional processes).
26% Assessment efforts do not have a clear focus.
26% I enjoy engaging in institutional assessment efforts.
24% Assessment success stories are formally shared throughout my institution.
23% Assessment results in an accurate depiction of what I do as a faculty member.
22% Assessment is conducted based on the whims of the people in charge.
21% If assessment was not required I would not be doing it.
21% Assessment is primarily the responsibility of administrators.
21% I am aware of several assessment success stories (i.e. instances of assessment resulting in important changes).
20% I do not have time to engage in assessment efforts.
19% Assessment results have no impact on resource allocations.
18% Assessment results are used to scare faculty into compliance with what the administration wants.
18% There is pressure to reveal only positive results from assessment efforts.
17% I avoid doing institutional assessment activities if I can.
17% I engage in assessment because I am afraid of what will happen if I do not.
14% I perceive assessment as a threat to academic freedom.
10% Assessment results are used to punish faculty members (i.e., not rewarding innovation or effective teaching, research, or service).
4% Assessment is someone else’s problem, not mine.
Overall, there’s good news here. Most faculty agreed with most positive statements about assessment, and most disagreed with most negative statements. I was particularly heartened that about three-quarters of respondents agreed that “assessment is a good thing for my institution to do,” about 70% agreed that “assessment is vital to my institution’s future,” and about two-thirds agreed that “assessment is a good thing for me to do.”
But there’s also plenty to be concerned about here. Only 35% agree that faculty are in charge of assessment and, by several measures, only a minority see assessment results shared and used. Almost 30% view assessment as a necessary evil.
Survey researchers know that people are more apt to agree than disagree with a statement, so I also looked at the percentages of faculty who disagreed or strongly disagreed with each statement. These responses do not mirror the agreed/strongly agreed results above, because on some items a larger proportion of faculty marked Only Slightly Agree or Only Slightly Disagree. Again, the positive statements are in green and the negative ones in red.
3% The majority of administrators are supportive of assessment.
6% Faculty leadership is necessary for my institution’s assessment efforts.
6% Assessment is a good thing for my institution to do.
7% Assessment is vital to my institution’s future.
8% I am highly interested in my institution’s assessment efforts.
8% In general a recommended change is more likely to be enacted by administrators if it is supported by assessment data.
9% I am actively engaged in my institution’s assessment efforts.
9% In general I am eager to work with administrators.
9% My academic department or college truly values faculty involvement in assessment.
10% Change occurs more readily when supported by assessment results.
10% Assessment is a good thing for me to do.
12% Assessment results are available to faculty by request.
13% Assessment is vital to my institution’s way of operating.
13% Assessment data are available to faculty by request.
13% The majority of administrators primarily emphasize assessment for the improvement of student learning.
13% I engage in institutional assessment efforts because it is the right thing to do for our students.
14% Discussions about student learning are at the heart of my institution.
14% I clearly understand assessment processes at my institution.
14% Assessment data are used to identify the extent to which student learning outcomes are met.
15% Assessments of programs are typically connected back to student learning.
15% Assessment results are used for improvement.
16% Assessment is primarily the responsibility of faculty members.
16% Administrators are supportive of making changes.
17% Assessment supports student learning at my institution.
18% Assessment processes yield evidence of my institution’s effectiveness.
18% I support the ways in which administrators have used assessment on my campus.
19% It is clear who is ultimately in charge of assessment.
19% Assessment is an organized coherent effort at my institution.
19% I have a generally positive attitude toward my institution’s culture of assessment.
20% Senior leaders, i.e., President or Provost, have made clear their expectations regarding assessment.
20% My institution is structured in a way that facilitates assessment practices focused on improved student learning.
20% I engage in institutional assessment because doing so makes a difference to student learning at my institution.
21% I am familiar with the office that leads student assessment efforts for accreditation purposes.
21% Budgets can be negatively impacted by assessment results.
22% The majority of administrators only focus on assessment in response to compliance requirements.
23% Student assessment results are regularly shared with faculty members.
24% I am familiar with the office that leads student assessment efforts for student learning.
24% Assessment for accreditation purposes is prioritized above other assessment efforts.
24% Assessment data are regularly used in official institutional communications.
28% Faculty are in charge of assessment at my institution.
29% Assessment results have no impact on resource allocations.
29% Assessment results are regularly shared throughout my institution.
29% I enjoy engaging in institutional assessment efforts.
31% Administrators share assessment data with faculty members using a variety of communication strategies (i.e., meetings, web, written correspondence, presentations).
31% Communication of assessment results has been effective.
31% Administrators use assessment as a form of control (i.e., to regulate institutional processes).
32% Assessment results are criticized for going nowhere (i.e., not leading to change).
32% Assessment results in a fair depiction of what I do as a faculty member.
33% There are sufficient financial resources to make changes at my institution.
34% Assessment success stories are formally shared throughout my institution.
34% Assessment results in an accurate depiction of what I do as a faculty member.
35% Assessment is primarily the responsibility of administrators.
36% I am aware of several assessment success stories (i.e., instances of assessment resulting in important changes).
36% Engaging in assessment also benefits my research/scholarship agenda.
41% Assessment efforts do not have a clear focus.
41% I do not have time to engage in assessment efforts.
42% Assessment is a necessary evil in higher education.
50% Assessment is conducted based on the whims of the people in charge.
50% There is pressure to reveal only positive results from assessment efforts.
53% Assessment results are used to scare faculty into compliance with what the administration wants.
55% I avoid doing institutional assessment activities if I can.
56% If assessment was not required I would not be doing it.
56% I engage in assessment because I am afraid of what will happen if I do not.
60% Assessment results are used to punish faculty members (i.e., not rewarding innovation or effective teaching, research, or service).
62% I perceive assessment as a threat to academic freedom.
78% Assessment is someone else’s problem, not mine.
Here there’s more good news. We want small proportions of faculty to disagree with the positive statements about assessment, and for the most part they do. About a third disagree that assessment results and success stories are shared, but that matches what we saw with the agree-strongly agree results.
But there are also areas of concern here. We want large proportions of faculty to disagree with the negative statements about assessment, and that doesn’t always happen. Less than a quarter disagree that budgets can be negatively impacted by assessment results and that administrators look at assessment only through a compliance lens. Less than a third disagreed that assessment results don’t lead to change or resource allocations. The results that concerned me most? Only 42% disagreed that assessment is a necessary evil; only half disagreed that there is pressure to reveal only positive assessment results; and only a bit over half disagreed that “If assessment was not required I would not be doing it.”
So, while most faculty “get” assessment, there are sizable numbers who don’t yet see value in it. We've come a long way, but there's still plenty of work to do!
(Some notes on the presentation of these results: Note that I sorted results from highest to lowest, rounded percentages to the nearest whole percent, and color-coded "good" and "bad" statements. Those all help the key points of a very lengthy survey pop out at the reader.)
|Posted on February 28, 2018 at 10:25 AM||comments (14)|
Two recent op-ed pieces in the Chronicle of Higher Education and the New York Times –and the hundreds of online comments regarding them—make clear that, 25 years into the assessment movement, a lot of faculty really hate assessment.
It’s tempting for assessment people to spring into a defensive posture and dismiss what these people are saying. (They’re misinformed! The world has changed!) But if that’s our response, aren’t we modeling the fractures deeply dividing the US today, with people existing in their own echo chambers and talking past each other rather than really listening and trying to find common ground on which to build? And shouldn’t we be practicing what we preach, using systematic evidence to inform what we say and do?
So I took a deeper dive into those comments. I did a content analysis of the articles and many of the comments that followed. (The New York Times article had over 500 comments—too many for me to handle—so I looked only at NYT comments with at least 12 recommendations.)
If you’re not familiar with content analysis, it’s looking through text to identify the frequency of ideas or themes. For example, I counted how many comments mentioned that assessment is expensive. I do content analysis by listing all the comments as bullets in a Word document, then cutting and pasting the bulleted comments to group similar comments together under headings. I then cut and paste the groups so the most frequently mentioned themes are at the top of the document. There is qualitative analysis software that can help if you don’t want to do this manually.
A caveat: Comments don’t always fall into neat, discrete categories; judgement is needed to decide where to place some. I did this analysis quickly, and it’s entirely possible that, if you’d done this instead of me, you might have come up with somewhat different results. But assessment is not rigorous research; we just need information good enough to help inform our thinking, and I think my analysis is fine for the purpose of figuring out how we might deal with this.
Why take the time to do a content analysis instead of just reading through the comments? Because, when we process a list of comments, there’s a good chance we won’t identify the most frequently mentioned ideas accurately. As I was doing my content analysis, I was struck by how many faculty complained that assessment is (I’m being snarky here) either a vast right-wing conspiracy or a vast left-wing conspiracy, simply because I’d never heard that before. It turned out, however, that there were other themes that emerged far more frequently. This is a good lesson for faculty who think they don’t need to formally assess because they “know” what their students are struggling with. Maybe they do…but maybe not.
So what did I find? As I’d expected, there are many reasons why faculty may hate assessment. I found that most of their complaints fall into just four broad categories:
It’s a waste of an enormous amount of time and resources that could be better spent on teaching. Almost 40% of the comments fell into this category. Some examples:
- We faculty are angry over the time and dollars wasted.
- The assessment craze is not only of little value, but it saps the meager resources of time and money available for classroom instruction.
- Faced with outrage over the high cost of higher education, universities responded by encouraging expensive administrative bloat.
- It is not that the faculty are not trying, but the data and methods in general use are very poor at measuring learning.
- Our “assessment expert” told us to just put down as a goal the % of students we wanted to rate us as very good or good on a self-report survey. Which we all know is junk.
I and what I think is important is not valued or respected. Over 30% of the comments fell into this category. Some examples:
- Assessment of student learning outcomes is an add-on activity that says your standard examination and grading scheme isn’t enough so you need to do a second layer of grading in a particular numerical format.
- The fundamental, flawed premise of most of modern education is that teaching is a science.
- Bureaucratic jargon subtly shapes the expectations of students and teachers alike.
- When the effort to reduce learning to a list of job-ready skills goes too far, it misses the point of a university education.
- Learning outcomes have disempowered faculty.
- The only learning outcomes I value: students complete their formal education with a desire to learn more
- Assessment reflects a misguided belief that learning is quantifiable.
External and economic forces are behind this. About 15% of comments fell into this category, including those right-wing/left-wing conspiracy comments. Some examples:
- There’s a whole industry out there that’s invested in outcomes assessment.
- The assessment boom coincided with the decision of state legislatures to reduce spending on public universities.
- Educational institutions have been forced to operate out of a business model.
- It is the rise of adjuncts and online classes that has led to the assessment push.
I’m unfairly held responsible for student learning. About 10% of comments fell into this category. Some examples:
- Students, not faculty, are responsible for student learning.
- It is much more profitable to skim money from institutions of higher learning than fixing the underlying causes of the poverty and lack of focus that harm students.
- The root cause is lack of a solid foundation built in K-12.
Two things struck me about these four broad categories. The first one was that they don’t quite align with what I’ve heard as I’ve worked with literally thousands of faculty at hundreds of colleges over the last two decades. Yes, I’ve heard plenty about assessment being useless, and I’ve written about faculty feeling devalued and disrespected by assessment, but I’d never heard the external-forces or blame-game reasons before. And I’ve heard plenty about other reasons that weren’t mentioned in these comments, especially finding time to work on assessment, not understanding how to assess (or how to teach), and moving from a culture of silos to one of collaboration. I think the reason for the disconnect between what I’ve heard and what was expressed here is that these comments reflect the angriest faculty, not all faculty. But their anger is legitimate and something we should all work to address.
[UPDATED 2/28/2018 4:36 PM EST] So what should we do? First, we clearly need better information on faculty experiences and views regarding assessment so we can understand which issues are most pervasive and address them. The Surveys of Assessment Culture developed by Matt Fuller at Sam Houston State University is an important start.
In the meanwhile, the good news is the comments in and accompanying these two pieces all represent solvable problems. (No, we can’t solve all of society’s ills, but we can help faculty deal with them.) I’ll share some ideas in upcoming blog posts. If you don’t want to wait, you’ll find plenty of practical suggestions in the new 3rd edition of my book Assessing Student Learning: A Common Sense Guide.
|Posted on February 22, 2018 at 7:00 PM||comments (0)|
I was intrigued by an article in the September 23, 2016, issue of Inside Higher Ed titled “When a C Isn’t Good Enough.” The University of Arizona found that students who earned an A or B in their first-year writing classes had a 67% chance of graduating, but those earning a C had only a 48% chance. The university is now exploring a variety of ways to improve the success of students earning a C, including requiring C students to take a writing competency test, providing resources to C students, and/or requiring C students to repeat the course.
I know nothing about the University of Arizona beyond what’s in the article. But if I were working with the folks there, I’d offer the following ideas to them, if they haven’t considered them already.
1. I’d like to see more information on why the C students earned a C. Which writing skills did they struggle most with: basic grammar, sentence structure, organization, supporting arguments with evidence, etc.? Or was there another problem? For example, maybe C students were more likely to hand in assignments late (or not at all).
2. I’d also like to see more research on why those C students were less likely to graduate. How did their GPAs compare to A and B students? If their grades were worse, what kinds of courses seemed to be the biggest challenge for them? Within those courses, what kinds of assignments were hardest for them? Why did they earn a poor grade on them? What writing skills did they struggle most with: basic grammar, organization, supporting arguments with evidence, etc.? Or, again, maybe there was another problem, such as poor self-discipline in getting work handed in on time.
And if their GPAs were not that different from those of A and B students (or even if they were), what else was going on that might have led them to leave? The problem might not be their writing skills per se. Perhaps, for example, that students with work or family obligations found it harder to devote the study time necessary to get good grades. Providing support for that issue might help more than helping them with their writing skills.
3. I’d also like to see the faculty responsible for first-year writing articulate a clear, appropriate, and appropriately rigorous standard for earning a C. In other words, they could use the above information on the kinds and levels of writing skills that students need to succeed in subsequent courses to articulate the minimum performance levels required to earn a C. When I taught first-year writing at a public university in Maryland, the state system had just such a statement, the “Maryland C Standard.”
4. I’d like to see the faculty adopt a policy that, in order to pass first-year writing, students must meet the minimum standard of every writing criterion. Thus, if student work is graded using a rubric, the grade isn’t determined by averaging the scores on various rubric criteria—that lets a student with A arguments but F grammar earn a C with failing grammar. Instead, students must earn at least a C on every rubric criterion in order to pass the assignment. Then the As, Bs, and Cs can be averaged into an overall grade for the assignment.
(If this sounds vaguely familiar to you, what I’m suggesting is the essence of competency-based education: students need to demonstrate competence on all learning goals and objectives in order to pass a course or graduate. Failure to achieve one goal or objective can’t be offset by strong performance on another.)
5. If they haven’t done so already, I’d also like to see the faculty responsible for first-year writing adopt a common rubric, articulating the criteria they’ve identified, that would be used to assess and grade the final assignment in every section, no matter who teaches it. This would make it easy to study student performance across all sections of the course and identify pervasive strengths and weaknesses in their writing. If some faculty members or TAs have additional grading criteria, they could simply add those to the common rubric. For example, I graded my students on their use of citation conventions, even though that was not part of the Maryland C Standard. I added that to the bottom of my rubric.
6. Because work habits are essential to success in college, I’d also suggest making this a separate learning outcome for first-year writing courses. This means grading students separately on whether they turn in work on time, put in sufficient effort, etc. This would help everyone understand why some students fail to graduate—is it because of poor writing skills, poor work habits, or both?
These ideas all move responsibility for addressing the problem from administrators to the faculty. That responsibility can’t be fulfilled unless the faculty commit to collaborating on identifying and implementing a shared strategy so that every student, no matter which section of writing they enroll in, passes the course with the skills needed for subsequent success.