Linda Suskie

  A Common Sense Approach to Assessment & Accreditation

Blog

view:  full / summary

A New Paradigm for Assessment

Posted on May 21, 2017 at 6:10 AM Comments comments (4)

I was impressed with—and found myself in agreement with—Douglas Roscoe’s analysis of the state of assessment in higher education in “Toward an Improvement Paradigm for Academic Quality” in the Winter 2017 issue of Liberal Education. Like Douglas, I think the assessment movement has lost its way, and it’s time for a new paradigm. And Douglas’s improvement paradigm—which focuses on creating spaces for conversations on improving teaching and curricula, making assessment more purposeful and useful, and bringing other important information and ideas into the conversation—makes sense. Much of what he proposes is in fact echoed in Using Evidence of Student Learning to Improve Higher Education by George Kuh, Stanley Ikenberry, Natasha Jankowski, Timothy Cain, Peter Ewell, Pat Hutchings, and Jillian Kinzie.


But I don’t think his improvement paradigm goes far enough, so I propose a second, concurrent paradigm shift.


I’ve always felt that the assessment movement tried to do too much, too quickly. The assessment movement emerged from three concurrent forces. One was the U.S. federal government, which through a series of Higher Education Acts required Title IV gatekeeper accreditors to require the institutions they accredit to demonstrate that they were achieving their missions. Because the fundamental mission of an institution of higher education is, well, education, this was essentially a requirement that institutions demonstrate that its intended student learning outcomes were being achieved by its students.


The Higher Education Acts also required Title IV gatekeeper accreditors to require the institutions they accredit to demonstrate “success with respect to student achievement in relation to the institution’s mission, including, as appropriate, consideration of course completion, state licensing examinations, and job placement rates” (1998 Amendments to the Higher Education Act of 1965, Title IV, Part H, Sect. 492(b)(4)(E)). The examples in this statement imply that the federal government defines student achievement as a combination of student learning, course and degree completion, and job placement.


A second concurrent force was the movement from a teaching-centered to learning-centered approach to higher education, encapsulated in Robert Barr and John Tagg’s 1995 landmark article in Change, “From Teaching to Learning: A New Paradigm for Undergraduate Education.” The learning-centered paradigm advocates, among other things, making undergraduate education an integrated learning experience—more than a collection of courses—that focuses on the development of lasting, transferrable thinking skills rather than just basic conceptual understanding.


The third concurrent force was the growing body of research on practices that help students learn, persist, and succeed in higher education. Among these practices: students learn more effectively when they integrate and see coherence in their learning, when they participate in out-of-class activities that build on what they’re learning in the classroom, and when new learning is connected to prior experiences.


These three forces led to calls for a lot of concurrent, dramatic changes in U.S. higher education:

  • Defining quality by impact rather than effort—outcomes rather than processes and intent
  • Looking on undergraduate majors and general education curricula as integrated learning experiences rather than collections of courses
  • Adopting new research-informed teaching methods that are a 180-degree shift from lectures
  • Developing curricula, learning activities, and assessments that focus explicitly on important learning outcomes
  • Identifying learning outcomes not just for courses but for for entire programs, general education curricula, and even across entire institutions
  • Framing what we used to call extracurricular activities as co-curricular activities, connected purposefully to academic programs
  • Using rubrics rather than multiple choice tests to evaluate student learning
  • Working collaboratively, including across disciplinary and organizational lines, rather than independently


These are well-founded and important aims, but they are all things that many in higher education had never considered before. Now everyone was being asked to accept the need for all these changes, learn how to make these changes, and implement all these changes—and all at the same time. No wonder there’s been so much foot-dragging on assessment! And no wonder that, a generation into the assessment movement and unrelenting accreditation pressure, there are still great swaths of the higher education community who have not yet done much of this and who indeed remain oblivious to much of this.


What particularly troubles me is that we’ve spent too much time and effort on trying to create—and assess—integrated, coherent student learning experiences and, in doing so, left the grading process in the dust. Requiring everything to be part of an integrated, coherent learning experience can lead to pushing square pegs into round holes. Consider:

  • The transfer associate degrees offered by many community colleges, for example, aren’t really programs—they’re a collection of general education and cognate requirements that students complete so they’re prepared to start a major after they transfer. So identifying—or assessing—program learning outcomes for them frankly doesn’t make much sense.
  • The courses available to fulfill some general education requirements don’t really have much in common, so their shared general education outcomes become so broad as to be almost meaningless.
  • Some large universities are divided into separate colleges and schools, each with their own distinct missions and learning outcomes. Forcing these universities to identify institutional learning outcomes applicable to every program makes no sense—again, the outcomes must be so broad as to be almost meaningless.
  • The growing numbers of students who swirl through multiple colleges before earning a degree aren’t going to have a really integrated, coherent learning experience no matter how hard any of us tries.


At the same time, we have given short shrift to helping faculty learn how to develop and use good assessments in their own classes and how to use grading information to understand and improve their own teaching. In the hundreds of workshops and presentations I’ve done across the country, I often ask for a show of hands from faculty who routinely count how many students earned each score on each rubric criterion of a class assignment, so they can understand what students learned well and what they didn’t learn well. Invariably a tiny proportion raises their hands. When I work with faculty who use multiple choice tests, I ask how many use a test blueprint to plan their tests so they align with key course objectives, and it’s consistently a foreign concept to them.


In short, we’ve left a vital part of the higher education experience—the grading process—in the dust. We invest more time in calibrating rubrics for assessing institutional learning outcomes, for example, than we do in calibrating grades. And grades have far more serious consequences to our students, employers, and society than assessments of program, general education, co-curricular, or institutional learning outcomes. Grades decide whether students progress to the next course in a sequence, whether they can transfer to another college, whether they graduate, whether they can pursue a more advanced degree, and in some cases whether they can find employment in their discipline.


So where we should go? My paradigm springs from visits to two Canadian institutions a few years ago. At that time Canadian quality assurance agencies did not have any requirements for assessing student learning, so my workshops focused solely on assessing learning more effectively in the classroom. The workshops were well received because they offered very practical help that faculty wanted and needed. And at the end of the workshops, faculty began suggesting that perhaps they should collaborate to talk about shared learning outcomes and how to teach and assess them. In other words, discussion of classroom learning outcomes began to flow into discussion of program learning outcomes. It’s a naturalistic approach that I wish we in the United States had adopted decades ago.


What I now propose is moving to a focus on applying everything we’ve learned about curriculum design and assessment to the grading process in the classroom. In other words, my paradigm agrees with Roscoe’s that “assessment should be about changing what happens in the classroom—what students actually experience as they progress through their courses—so that learning is deeper and more consequential.” My paradigm emphasizes the following.

  1. Assessing program, general education, and institutional learning outcomes remain an assessment best practice. Those who have found value in these assessments would be encouraged to continue to engage in them and honored through mechanisms such as NILOA’s Excellence in Assessment designation.
  2. Teaching excellence is defined in significant part by four criteria: (1) the use of research-informed teaching and curricular strategies, (2) the alignment of learning activities and grading criteria to stated course objectives, (3) the use of good quality evidence, including but not limited to assessment results from the grading process, to inform changes to one’s teaching, and (4) active participation in and application of professional development opportunities on teaching including assessment.
  3. Investments in professional development on research-informed teaching practices exceed investments in assessment.
  4. Assessment work is coordinated and supported by faculty professional development centers (teaching-learning centers) rather than offices of institutional effectiveness or accreditation, sending a powerful message that assessment is about improving teaching and learning, not fulfilling an external mandate.
  5. We aim to move from a paradigm of assessment, not just to one of improvement as Roscoe proposes, but to one of evidence-informed improvement—a culture in which the use of good quality evidence to inform discussions and decisions is expected and valued.
  6. If assessment is done well, it’s a natural part of the teaching-learning process, not a burdensome add-on responsibility. The extra work is in reporting it to accreditors. This extra work can’t be eliminated, but it can be minimized and made more meaningful by establishing the expectation that reports address only key learning outcomes in key courses (including program capstones), on a rotating schedule, and that course assessments are aggregated and analyzed within the program review process.


Under this paradigm, I think we have a much better shot at achieving what’s most important: giving every student the best possible education.

How hard should a multiple choice test be?

Posted on March 18, 2017 at 8:25 AM Comments comments (0)

My last blog post on analyzing multiple choice test results generated a good bit of feedback, mostly on the ASSESS listserv. Joan Hawthorne and a couple of other colleagues thoughtfully challenged my “50% rule”—that any questions that more than 50% of your students get wrong may suggest something wrong and should be reviewed carefully.

 

Joan pointed out that my 50% rule shouldn’t be used with tests that are so important that students should earn close to 100%. She’s absolutely right. Some things we teach—healthcare, safety—are so important that if students don’t learn them well, people could die. If you’re teaching and assessing must-know skills and concepts, you might want to look twice at any test items that more than 10% or 15% of students got wrong.

 

With other tests, how hard the test should be depends on its purpose. I was taught in grad school that the purpose of some tests is to separate the top students from the bottom—distinguish which students should earn an A, B, C, D, or F. If you want to maximize the spread of test scores, an average item difficulty of 50% is your best bet—in theory, you should get test scores ranging all the way from 0 to 100%. If you want each test item to do the best possible job discriminating between top and bottom students, again you’d want to aim for a 50% difficulty.

 

But in the real world I’ve never seen a good test with an overall 50% difficulty for several good reasons.

 

1. Difficult test questions are incredibly hard to write. Most college students want to get a good grade and will at least try to study for your test. It’s very hard to come up with a test question that assesses an important objective but that half of them will get wrong. Most difficult items I’ve seen are either on minutiae, “trick” questions on some nuanced point, or questions that are more tests of logical reasoning skill than course learning objectives. In my whole life I’ve written maybe two or three difficult multiple choice questions that I’ve been proud of: that truly focused on important learning outcomes and didn’t require a careful nuanced reading or logical reasoning skills. In my consulting work, I’ve seen no more than half a dozen difficult but effective items written by others. This experience has led me to suggest that “50% rule.”

 

2. Difficult tests are demoralizing to students, even if you “curve” the scores and even if they know in advance that the test will be difficult.

 

3. Difficult tests are rarely appropriate, because it’s rare for the sole or major purpose of a test to be to maximize the spread of scores. Many tests have dual purposes. There are certain fundamental learning objectives we want to make sure (almost) every student has learned, or they’re going to run into problems later on. Then there are some learning objectives that are more challenging—that only the A or maybe B students will achieve—and those test items will separate the A from B students and so on.

 

So, while I have great respect for those who disagree with me, I stand by my suggestion in my last blog post. Compare each item’s actual difficulty (the percent of students who answered incorrectly) against how difficult you wanted that item to be, and carefully evaluate any items that more than 50% of your students got wrong.

 

 

 

What to look for in multiple choice test reports

Posted on February 28, 2017 at 8:15 AM Comments comments (2)

Next month I’m doing a faculty professional development workshop on interpreting the reports generated for multiple choice tests. Whenever I do one of these workshops, I ask the sponsoring institution to send me some sample reports. I’m always struck by how user-unfriendly they are!

 

The most important thing to look at in a test report is the difficulty of each item—the percent of students who answered each item correctly. Fortunately these numbers are usually easy to find. The main thing to think about is whether each item was as hard as you intended it to be. Most tests have some items on essential course objectives that every student who passes the course should know or be able to do. We want virtually every student to answer those items correctly, so check those items and see if most students did indeed get them right.

 

Then take a hard look at any test items that a lot of students got wrong. Many tests purposefully include a few very challenging items, requiring students to, say, synthesize their learning and apply it to a new problem they haven’t seen in class. These are the items that separate the A students from the B and C students. If these are the items that a lot of students got wrong, great! But take a hard look at any other questions that a lot of students got wrong. My personal benchmark is what I call the 50 percent rule: if more than half my students get a question wrong, I give the question a hard look.

 

Now comes the hard part: figuring out why more students got a question wrong than we expected. There are several possible reasons including the following:

 

  • The question or one or more of its options is worded poorly, and students misinterpret them.
  • We might have taught the question’s learning outcome poorly, so students didn’t learn it well. Perhaps students didn’t get enough opportunities, through classwork or homework, to practice the outcome.
  • The question might be on a trivial point that few students took the time to learn, rather than a key course learning outcome. (I recently saw a question on an economics test that asked how many U.S. jobs were added in the last quarter. Good heavens, why do students need to memorize that? Is that the kind of lasting learning we want our students to take with them?)

 

 

If you’re not sure why students did poorly on a particular test question, ask them! Trust me, they’ll be happy to tell you what you did wrong!

 

Test reports provide two other kinds of information: the discrimination of each item and how many students chose each option. These are the parts that are usually user-unfriendly and, frankly, can take more time to decipher than they’re worth.

 

The only thing I’d look for here is any items with negative discrimination. The underlying theory of item discrimination is that students who get an A on your test should be more likely to get any one question right than students who fail it. In other words, each test item should discriminate between top and bottom students. Imagine a test question that all your A students get wrong but all your failing students answer correctly. That’s an item with negative discrimination. Obviously there’s something wrong with the question’s wording—your A students interpreted it incorrectly—and it should be thrown out. Fortunately, items with negative discrimination are relatively rare and usually easy to identify in the report.

What does a new CAO survey tell us about the state of assessment?

Posted on January 26, 2017 at 8:40 AM Comments comments (6)

A new survey of chief academic officers (CAOs) conducted by Gallup and Inside Higher Education led me to the sobering conclusion that, after a generation of work on assessment, we in U.S. higher education remain very, very far from pervasively conducting truly meaningful and worthwhile assessment.


Because we've been working on this so long, as I reviewed the results of this survey, I was deliberately tough. The survey asked CAOs to rate the effectiveness of their institutions on a variety of criteria using a scale of very effective, somewhat effective, not too effective, and not effective at all. The survey also asked CAOs to indicate their agreement with a variety of statements on a five-point scale, where 5 = strongly agree, 1 = strongly disagree, and the other points are undefined. At this point I would have liked to see most CAOs rate their institutions at the top of the scale: either “very effective” or “strongly agree.” So these are the results I focused on and, boy, are they depressing.


Quality of Assessment Work

Less than a third (30%) of CAOs say their institution is very effective in identifying and assessing student outcomes. ‘Nuff said on that! :(


Value of Assessment Work

Here the numbers are really dismal. Less than 10% (yes, ten percent, folks!) of CAOs strongly agree that:

  • Faculty members value assessment efforts at their college (4%).
  • The growth of assessment systems has improved the quality of teaching and learning at their college (7%).
  • Assessment has led to better use of technology in teaching and learning (6%). (Parenthetically, that struck me as an odd survey question; I had no idea that one of the purposes of assessment was to improve the use of technology in T&L!)


And just 12% strongly disagree that their college’s use of assessment is more about keeping accreditors and politicians happy than it is about teaching and learning.

 

And only 6% of CAOs strongly disagree that faculty at their college view assessment as requiring a lot of work on their parts. Here I’m reading something into the question that might not be there. If the survey asked if faculty view teaching as requiring a lot of work on their parts, I suspect that a much higher proportion of CAOs would disagree because, while teaching does require a lot of work, it’s what faculty generally find to be valuable work--it's what they are expected to do, after all. So I suspect that, if faculty saw value in their assessment work commensurate with the time they put into it, this number would be a lot higher.

 

Using Evidence to Inform Decisions

Here’s a conundrum:

  • Over two thirds (71%) of CAOs say their college makes effective use of data used to measure student outcomes,
  • But only about a quarter (26%) said their college is very effective in using data to aid and inform decision making.
  • And only 13% strongly agree that their college regularly makes changes in the curriculum, teaching practices, or student services based on what it finds through assessment.


 So I’m wondering what CAOs consider effective uses of assessment data!


 Furthermore,

  • About two thirds (67%) of CAOs say their college is very effective in providing a quality undergraduate education.
  • But less than half (48%) say it’s very effective in preparing students for the world of work,
  • And only about a quarter (27%) say it’s very effective in preparing students for engaged citizens.
  • And (as I've already noted) only 30% say it’s very effective in identifying and assessing student outcomes.


How can CAOs who admit their colleges are not very effective in preparing students for work or citizenship engagement or assessing student learning nonetheless think their college is very effective in providing a quality undergraduate education? What evidence are they using to draw that conclusion?


And,

  • While less than half of CAOs saying their colleges are very effective in preparing students for work,
  • Only about a third (32%) strongly agree that their institution is increasing attention to the ability of its degree programs to help students get a good job.


My Conclusions

After a quarter century of work to get everyone to do assessment well:

  • Assessment remains spotty; it is the very rare institution that is doing assessment pervasively and consistently well.
  • A lot of assessment work either isn’t very useful or takes more time than it’s worth.
  • We have not yet transformed American higher education into an enterprise that habitually uses evidence to inform decisions.

What are the characteristics of effective curricula?

Posted on January 6, 2017 at 8:20 PM Comments comments (9)

I'm working on a book chapter on curriculum design, and I've come up with eight characteristics of effective curricula, whether for a course, program, general education, or co-curricular experience:

• They treat a learning goal as a promise.

• They are responsive to the needs of students, employers, and society.

• They are greater than the sum of their parts.

• They give students ample and diverse opportunities to achieve key learning goals.

• They have appropriate, progressive rigor.

• They conclude with an integrative, synthesizing capstone experience.

• They are focused and simple.

• They use research-informed strategies to help students learn and succeed, including high-impact practices.

 

What do you think? Do these make sense? Have I missed anything?

And...do the curricula you work with have these characteristics?

Making a habit of using classroom assessment information to inform our own teaching

Posted on December 20, 2016 at 10:50 AM Comments comments (1)

Given my passion for assessment, you might not be surprised to learn that, whenever I teach, the most fun part for me is analyzing how my students have done on the tests and assignments I’ve given them. Once tests or papers are graded, I can’t wait to count up how many students got each test question right or how many earned each possible score on each rubric criterion. When I teach workshops, I rely heavily on minute papers, and I can’t wait to type up all the comments and do a qualitative analysis of them. I love to teach, and I really want to be as good a teacher as I can. And, for me, an analysis of what students have and haven’t learned is the best possible feedback on how well I’m teaching, much more meaningful and useful than student evaluations of teaching.

 

I always celebrate the test questions or rubric criteria that all my students did well on. I make a point of telling the class and, no matter how jaded they are, you should see their faces light up!

 

And I always reflect on the test questions or rubric criteria for which my students did poorly. Often I can figure out on my own what happened. Often it’s simply a poorly written question or assignment, but sometimes I have to admit to myself that I didn’t teach that concept or skill particularly well. If I can’t figure out what happened, I ask the class and, trust me, they’re happy to tell me how I screwed up! If it’s a really vital concept or skill and we’re not at the end of the course, I’ll often tell them, “I screwed up, but I can’t let you out of here not knowing how to do this. We’re going to go over it again, you’re going to get more homework on it, and you’ll submit another assignment (or have more test questions) on this.” If it's the end of the course, I make notes to myself on what I'll do differently next time.

 

I often share this story at the faculty workshops I facilitate. I then ask for a show of hands of how many participants do this kind of analysis in their own classes. The number of hands raised varies—sometimes there will be maybe half a dozen hands in a room of 80, sometimes more—but rarely do more than a third or half of those present raise their hands. This is a real issue, because if faculty aren’t in the habit of analyzing and reflecting on assessment results in their own classes, how can we expect them to do so collaboratively on broader learning outcomes? In short, it’s a troubling sign that the institutional community is not yet in the habit of using systematic evidence to understand and improve student learning, which is what all accreditors want.

 

Here, then, is my suggestion for a New Year’s resolution for all of you who teach or in any way help students learn: Start doing this! You don’t have to do this for every assignment in every course you teach, but pick at least one key test or assignment in one course whose scores aren’t where you’d like them. Your analysis and reflection on that one test or assignment will lead you into the habit of using the assessment evidence in front of you more regularly, and it will make you an even better teacher than you are today.

 

Lessons from the Election for Higher Education

Posted on November 28, 2016 at 7:25 AM Comments comments (0)

If you share my devastation at the results of the U.S. presidential election and its implications for our country and our world, and if you are struggling to understand what has happened and wondering what you can do as a member of the higher education community, this blog post is for you. I don’t have answers, of course, but I have some ideas.

 

Why did Trump get so many votes? While the reasons are complex, and people will be debating them for years, there seem to be two fundamental factors. One can be summed up in that famous line from Bill Clinton’s campaign: It’s the economy, stupid.  Jed Kolko at fivethirtyeight.com found that people who voted for Trump were more likely to feel under economic threat, worried about the future of their jobs.

 

The other reason is education. Nate Silver at fivethirtyeight.com has tweeted that Clinton won all 18 states where an above average share of the population has advanced degrees, but she lost 29 of the other 32.  Education and salary are highly correlated, but Nate Silver has found signs that education appears to be a stronger predictor of who voted for Trump than salary.

 

Why is education such a strong predictor of how people voted? Here’s where we need more research, but I’m comfortable speculating that reasons might include any of the following:

  • People without a college education have relatively few prospects for economic security. In my book Five Dimensions of Quality I noted that the Council of Foreign Relations found that, “going back to the 1970s, all net job growth has been in jobs that require at least a bachelor’s degree.” I also noted a statistic from Anthony Carnevale and his colleagues: “By 2020, 65 percent of all jobs will require postsecondary education and training, up from 28 percent in 1973.”
  • Colleges do help students learn to think critically: to distinguish credible evidence from what I call “incredible” evidence, to weigh evidence carefully when making difficult decisions, and to make decisions based more on good quality evidence than on emotional response.
  • College-educated citizens are more likely to have attended quality good schools from kindergarten on, learning to think critically not just in college but throughout their schooling.
  • College-educated citizens are more optimistic because their liberal arts studies give them the open-mindedness and flexibility to handle changing times, including changing careers.

We do have a tremendous divide in this country—an education divide—and it is growing. While college degree holders have always earned more than those without a college degree, the income disparity has grown; college graduates now earn 80% more than high school graduates, up from 40% in the 1970s.

 

If we want a country that offers economic security, whose citizens feel a sense of optimism, whose citizens make evidence-informed decisions, and whose citizens are prepared for changes in their country and their lives, we need to work on closing the education divide by helping as many people as possible get a great postsecondary education.

 

What can we do?

  1. Welcome the underprepared. They are the students who really need our help in obtaining not only economic security but the thinking skills that are the hallmark of a college education and a sense of optimism about their future. The future of our country is in their hands.
  2. Make every student want to come back, as Ken O’Donnell has said, until they complete their degree or credential. Every student we lose hurts his or her economic future and our country.
  3. Encourage actively what Ernest Boyer called the scholarship of application: using research to solve real-life problems such as regional social and economic issues.
  4. Partner with local school systems and governments to improve local grade schools. Many regions of the country need new businesses, but new businesses usually want to locate in communities with good schools for their employees and their families.
  5. Create more opportunities for students to listen to and learn from others with different backgrounds and perspectives. Many colleges seek to attract international students and encourage students to study abroad. I’d like to go farther. Do we encourage our international students to share their backgrounds and experiences with our American students, both in relevant classes and in co-curricular settings? Do we encourage returning study abroad students to share what they learned with their peers? Do we encourage our students to consider not only a semester abroad but a semester at another U. S. college in a different part of the country?
  6. Create more opportunities for students to learn about the value of courtesy, civility, respect, compassion, and kindness and how to practice these in their lives and careers.

 

 

 

Lessons from the Election for Assessment

Posted on November 21, 2016 at 2:45 PM Comments comments (0)

The results of the U.S. presidential election have lessons both for American higher education and for assessment. Here are the lessons I see for meaningful assessment; I’ll tackle implications for American higher education in my next blog post.

 

Lesson #1: Surveys are a difficult way to collect meaningful information in the 21st century. If your assessment plan includes telephone or online surveys of students, alumni, employers, or anyone else, know going in that it’s very hard to get a meaningful, representative sample.

 

A generation ago (when I wrote a monograph Questionnaire Survey Research: What Works for the Association of Institutional Research), most people had land line phones with listed numbers and without caller ID or voice mail. So it was easy to find their phone number, and they usually picked up the phone when it rang. Today many people don’t have land line phones; they have cell phones with unlisted numbers and caller ID. If the number calling is unfamiliar to them, they let the call go straight to voice mail. Online surveys have similar challenges, partly because databases of e-mail addresses aren’t as readily available as phone books and partly because browsing habits affect the validity of pop-up polls such as those conducted by Survey Monkey. And all survey formats are struggling with survey fatigue (how many surveys have you been asked to complete in the last month?).

 

Professional pollsters have ways of adjusting for all these factors, but those strategies are difficult and expensive and often beyond our capabilities.

 

Lesson #2: Small sample sizes may not yield meaningful evidence. Because of Lesson #1, many of the political polls we saw were based on only a few hundred respondents. A sample of 250 has an error margin of 6% (meaning that if, for example, you find that 82% of the student work you assessed meets your standard, the true percentage is probably somewhere between 76% and 88%). A sample of 200 has an error margin of 7%. And these error margins assume that the samples of student work you’re looking at are truly representative of all student work. Bottom line: We need to look at a lot of student work, from a broad variety of classes, in order to draw meaningful conclusions.

 

Lesson #3: Small differences aren’t meaningful. I was struck by how many reporters and pundits talked about Clinton having, say, a 1% or 2% point lead without mentioning that the error margin made these leads too close to call. I know everyone likes to have a single number—it’s easiest to grasp—but I wish we could move to the practice of reporting ranges of likely results, preferably in graphs that show overlaps and convey visually when differences aren’t really significant. That would help audiences understand, for example, whether students’ critical thinking skills really are worse than their written communication skills, or whether their information literacy skills really are better than those of their peers.

 

Lesson #4: Meaningful results are in the details. Clinton won the popular vote by well over a million votes but still lost enough states to lose the Electoral College. Similarly, while students at our college may be doing well overall in terms of their analytic reasoning skills, we should be concerned if students in a particular program or cohort aren’t doing that well. Most colleges and universities are so diverse in terms of their offerings and the students they serve that I’m not sure overall institution-wide results are all that helpful; the overall results can mask a great deal of important variation.

 

Lesson #5: We see what we want to see. With Clinton the odds-on favorite to win the race, it was easy to see Trump’s chances of winning (anywhere from 10-30%, depending on the analysis) as insignificant, when in fact these probabilities meant he had a realistic chance of winning. Just as it was important to take a balanced view of poll results, it’s important to bring a balanced view to our assessment results. Usually our assessment results are a mixed bag, with both reasons to cheer and reasons to reflect and try to improve. We need to make sure we see—and share—both the successes and the areas for concern.

 

 

Using Professional Development Funds Wisely

Posted on November 9, 2016 at 5:45 AM Comments comments (0)

 

I recently suggested to a college that it invest its professional development funds in helping faculty learn more about how to teach and assess. The response? We already do plenty—we give every faculty member funds to use however they like on professional development.

 

The problem with this approach is that there can be a difference between what people want to do and what they should do. If you gave me funds for my own personal growth and development, I’d probably use it to visit some fine restaurants instead of the gym membership that I really should get. If you gave me funds for professional development, I’d probably use it to go to a research conference in a nice location rather than organize a visit with a team of my colleagues a college that’s doing a great job teaching and assessing writing in our discipline.

 

One of the themes of my book Five Dimensions of Quality is “put your money where your mouth is.” Does your college really do this when it comes to professional development?

 

  • Does your college focus its professional development resources on the things your college says it’s focusing on? For example, if one of your college’s strategic goals is to be student-centered, do you focus professional development funds on helping faculty and staff learn what it means to be student-centered and how to incorporate student-centered practices into their teaching and other responsibilities?
  • Does your college give priority to funding sabbatical leave requests that, again, address your college’s top priorities? If your college’s mission or strategic goals include teaching excellence, for example, do you give high priority to sabbatical leaves that address the scholarship of teaching?
  • Does your college prioritize travel funding conferences and events that will help faculty and staff develop the knowledge and skills to address your college’s top priorities, such as student success?
  • Does your college prioritize sabbatical and travel funding for requests that include plans to disseminate that’s been learned to colleagues across your college?
  • Does your teaching-learning center use systematic evidence of what faculty and student development staff most need to learn when it plans its professional development offerings? For example, if assessments show that students across a variety of disciplines struggle to cite sources, does the TLC collaborate with librarians to offer programming on how to teach students to cite sources?
  • Does your assessment committee periodically review department assessment reports to identify what faculty and staff are doing well with assessment and what remains a struggle? Does it publicize successes, such as useful rubrics and prompts, to help others learn what good practices look like? Does it sponsor or recommend professional development to help faculty and staff with whatever aspects of assessment are most challenging?

An example of closing the loop...and ideas for doing it well

Posted on September 24, 2016 at 7:35 AM Comments comments (5)

I was intrigued by an article in the September 23, 2016, issue of Inside Higher Ed titled “When a C Isn’t Good Enough.” The University of Arizona found that students who earned an A or B in their first-year writing classes had a 67% chance of graduating, but those earning a C had only a 48% chance. The university is now exploring a variety of ways to improve the success of students earning a C, including requiring C students to take a writing competency test, providing resources to C students, and/or requiring C students to repeat the course.

 

I know nothing about the University of Arizona beyond what’s in the article. But if I were working with the folks there, I’d offer the following ideas to them, if they haven’t considered them already.

 

1. I’d like to see more information on why the C students earned a C. Which writing skills did they struggle most with: basic grammar, sentence structure, organization, supporting arguments with evidence, etc.? Or was there another problem? For example, maybe C students were more likely to hand in assignments late (or not at all).

 

2. I’d also like to see more research on why those C students were less likely to graduate. How did their GPAs compare to A and B students? If their grades were worse, what kinds of courses seemed to be the biggest challenge for them? Within those courses, what kinds of assignments were hardest for them? Why did they earn a poor grade on them? What writing skills did they struggle most with: basic grammar, organization, supporting arguments with evidence, etc.? Or, again, maybe there was another problem, such as poor self-discipline in getting work handed in on time.

 

And if their GPAs were not that different from those of A and B students (or even if they were), what else was going on that might have led them to leave? The problem might not be their writing skills per se. Perhaps, for example, that students with work or family obligations found it harder to devote the study time necessary to get good grades. Providing support for that issue might help more than helping them with their writing skills.

 

3. I’d also like to see the faculty responsible for first-year writing articulate a clear, appropriate, and appropriately rigorous standard for earning a C. In other words, they could use the above information on the kinds and levels of writing skills that students need to succeed in subsequent courses to articulate the minimum performance levels required to earn a C. (When I taught first-year writing at a public university in Maryland, the state system had just such a statement, the “Maryland C Standard.”)

 

4. I’d like to see the faculty adopt a policy that, in order to pass first-year writing, students must meet the minimum standard of every writing criterion. Thus, if student work is graded using a rubric, the grade isn’t determined by averaging the scores on various rubric criteria—that lets a student with A arguments but F grammar earn a C with failing grammar. Instead, students must earn at least a C on every rubric criterion in order to pass the assignment. Then the As, Bs, and Cs can be averaged into an overall grade for the assignment.

 

(If this sounds vaguely familiar to you, what I’m suggesting is the essence of competency-based education: students need to demonstrate competence on all learning goals and objectives in order to pass a course or graduate. Failure to achieve one goal or objective can’t be offset by strong performance on another.)

 

5. If they haven’t done so already, I’d also like to see the faculty responsible for first-year writing adopt a common rubric, articulating the criteria they’ve identified, that would be used to assess and grade the final assignment in every section, no matter who teaches it. This would make it easy to study student performance across all sections of the course and identify pervasive strengths and weaknesses in their writing. If some faculty members or TAs have additional grading criteria, they could simply add those to the common rubric. For example, I graded my students on their use of citation conventions, even though that was not part of the Maryland C Standard. I added that to the bottom of my rubric.

 

6. Because work habits are essential to success in college, I’d also suggest making this a separate learning outcome for first-year writing courses. This means grading students separately on whether they turn in work on time, put in sufficient effort, etc. This would help everyone understand why some students fail to graduate—is it because of poor writing skills, poor work habits, or both?

 

These ideas all move responsibility for addressing the problem from administrators to the faculty. That responsibility can’t be fulfilled unless the faculty commit to collaborating on identifying and implementing a shared strategy so that every student, no matter which section of writing they enroll in, passes the course with the skills needed for subsequent success.

 


Rss_feed