Linda Suskie

  A Common Sense Approach to Assessment & Accreditation

Blog

What does a new CAO survey tell us about the state of assessment?

Posted on January 26, 2017 at 8:40 AM

A new survey of chief academic officers (CAOs) conducted by Gallup and Inside Higher Education led me to the sobering conclusion that, after a generation of work on assessment, we in U.S. higher education remain very, very far from pervasively conducting truly meaningful and worthwhile assessment.


Because we've been working on this so long, as I reviewed the results of this survey, I was deliberately tough. The survey asked CAOs to rate the effectiveness of their institutions on a variety of criteria using a scale of very effective, somewhat effective, not too effective, and not effective at all. The survey also asked CAOs to indicate their agreement with a variety of statements on a five-point scale, where 5 = strongly agree, 1 = strongly disagree, and the other points are undefined. At this point I would have liked to see most CAOs rate their institutions at the top of the scale: either “very effective” or “strongly agree.” So these are the results I focused on and, boy, are they depressing.


Quality of Assessment Work

Less than a third (30%) of CAOs say their institution is very effective in identifying and assessing student outcomes. ‘Nuff said on that! :(


Value of Assessment Work

Here the numbers are really dismal. Less than 10% (yes, ten percent, folks!) of CAOs strongly agree that:

  • Faculty members value assessment efforts at their college (4%).
  • The growth of assessment systems has improved the quality of teaching and learning at their college (7%).
  • Assessment has led to better use of technology in teaching and learning (6%). (Parenthetically, that struck me as an odd survey question; I had no idea that one of the purposes of assessment was to improve the use of technology in T&L!)


And just 12% strongly disagree that their college’s use of assessment is more about keeping accreditors and politicians happy than it is about teaching and learning.

 

And only 6% of CAOs strongly disagree that faculty at their college view assessment as requiring a lot of work on their parts. Here I’m reading something into the question that might not be there. If the survey asked if faculty view teaching as requiring a lot of work on their parts, I suspect that a much higher proportion of CAOs would disagree because, while teaching does require a lot of work, it’s what faculty generally find to be valuable work--it's what they are expected to do, after all. So I suspect that, if faculty saw value in their assessment work commensurate with the time they put into it, this number would be a lot higher.

 

Using Evidence to Inform Decisions

Here’s a conundrum:

  • Over two thirds (71%) of CAOs say their college makes effective use of data used to measure student outcomes,
  • But only about a quarter (26%) said their college is very effective in using data to aid and inform decision making.
  • And only 13% strongly agree that their college regularly makes changes in the curriculum, teaching practices, or student services based on what it finds through assessment.


 So I’m wondering what CAOs consider effective uses of assessment data!


 Furthermore,

  • About two thirds (67%) of CAOs say their college is very effective in providing a quality undergraduate education.
  • But less than half (48%) say it’s very effective in preparing students for the world of work,
  • And only about a quarter (27%) say it’s very effective in preparing students for engaged citizens.
  • And (as I've already noted) only 30% say it’s very effective in identifying and assessing student outcomes.


How can CAOs who admit their colleges are not very effective in preparing students for work or citizenship engagement or assessing student learning nonetheless think their college is very effective in providing a quality undergraduate education? What evidence are they using to draw that conclusion?


And,

  • While less than half of CAOs saying their colleges are very effective in preparing students for work,
  • Only about a third (32%) strongly agree that their institution is increasing attention to the ability of its degree programs to help students get a good job.


My Conclusions

After a quarter century of work to get everyone to do assessment well:

  • Assessment remains spotty; it is the very rare institution that is doing assessment pervasively and consistently well.
  • A lot of assessment work either isn’t very useful or takes more time than it’s worth.
  • We have not yet transformed American higher education into an enterprise that habitually uses evidence to inform decisions.

Categories: Ideas

Post a Comment

Oops!

Oops, you forgot something.

Oops!

The words you entered did not match the given text. Please try again.

6 Comments

Reply Madeleine
2:08 PM on January 27, 2017 
Yes, I am often struck that discussions about assessment focus not so much on telling teachers to assess, but on how frustrating it is that we don't document the assessment activities we are, in fact, already doing. So the problem isn't necessarily one of teachers not assessing their students, or themselves; it's that the CAOs can't track everything all the time. The extent to which this matters or not is an interesting question. Now that we *can* know everything, if we just devote enough resources to data, it's hard not to feel like achieving some kind of institutional transparency would be a good use of time and money. Or is this just the CAO version of helicopter parenting, stifling innovation and spontaneity? I am not sure....
Reply Adam Halemano
3:34 PM on January 26, 2017 
Debbie makes a valid and very astute point. With regard to the state of assessments, within the context of what I consider to be "perceptual distortion", I too believe there is a philosophical gap between those who have advanced degrees in education and those who do not. This is not to imply that all educators trained only in their respective disciplines do not value assessments. Rather, I would assert that there isn't enough focus and emphasis on the importance of assessments, in terms of professional development training/workshops for ALL vested stakeholders, to help alter the myopic and distorted perceptions that are at the root of this cognitive dissonance issue. From a philosophical and policy driven standpoint, I can understand (to a certain degree) why CAOs are inclined to believe one thing despite conflicting contrast to the other. That said, in addition to the need for deeper focus and emphasis in assessments, is there a more meaningful and effective remedy to help cut through the "reality distortion field" that binds many within the educational community to a mindset held captive in a vacuum of delusion and denial? With a fair degree of optimism...I would like to think so.
Reply Brenda Edmonds
2:55 PM on January 26, 2017 
Or here's another interpretation of the data: The CAOs may not have a good handle on what is really happening with teaching and learning.

Although all the percents are low, where are the REALLY low ones? They are all in the questions to do with TEACHERS using the data to make decisions and improve teaching.

As a former Assessement Director who went back to teaching, my view is that teachers do this a LOT without necessarily sending in a report or documenting it, so the CAOs (and Assessment Directors) may not know about it. The (newish) CAO at my college has NO IDEA what I do, or that I work 60-hour weeks regularly during the semester, or that I am always reading research or looking at data from my own courses and my department to make changes. Part of that is my own fault, but who has time in that 60-hour work week to make sure he knows what I am doing? But he does know about the occasional professor that is not doing their job well, because someone complains. Anyway, maybe it stresses the need to let the CAOs know more about it.

Thanks, Linda, for your always interesting commentary on assessment and teaching and learning.
Reply Madeleine
2:39 PM on January 26, 2017 
It's certainly true that teachers generally don't have degrees in educational research. And since, in many institutions, it's teachers who have been asked to analyze their "assessment data," this is probably one reason why it never leads to overall change.

I would like to know, though, how much attention has been given to the criticisms of the assessment movement. Every teacher knows that when almost everyone dislikes an assignment, or gets nothing out of it, or does badly on it, the problem is with the *assignment,* not the students.

Could it not be that perhaps - just maybe - assessment doesn't really solve our problems?

For instance, I wonder if maybe the "assessment" movement is looking in the wrong place?

To take one example: If we have decided that student *outcomes* are more important than their progress through the various courses they're taking, we need to ditch the GPA system and instead require colleges to administer uniform end-of-year exams for all subjects. That's what the European colleges do. That's what all institutions do, in fact, when they really want to find out what you know. No one cares about your law school GPA; you have to pass the bar exam.

Thus, instead of getting faculty to document, record, analyze and discuss all kinds of assessment data *other* than course grades, maybe we should focus on creating a movement for abolishing the practice of instructor-grading and GPA.
Reply Craig Pepin
1:41 PM on January 26, 2017 
Thanks for sharing these results, Linda. You're a tough grader! But yes, I agree that after this long we should be prepared to be tough.
What this evidence says to me, is that it is REALLY hard to move from assessment data to meaningful curricular and pedagogical change. It's much easier to collect good (or "good enough") assessment data, than it is to get teachers and administrators to take it seriously enough to institute changes in response to the data. Charlie Blaich and Kathy Wise have written eloquently on this, as have others.
The lack of strong agreement about the success of assessment efforts might have to do both with the difficulty in using assessment results to inform teaching and curriculum, and how depressing it is to constantly see the lack of impact our efforts have.
Reply Debbie Kell
11:11 AM on January 26, 2017 
These findings and your very appropriate conclusions are sobering. It strikes me that faculty and academic leaders, while having advanced degrees, do not necessarily have degrees or credits in education or educational research. The K-12 sector teachers seem to have a better grasp on the value of using evidence to inform decisions - perhaps the teacher education programs leading to teacher certification are doing a better job of exposing their students to assessment practices that measurably inform practice.