|Posted on August 9, 2015 at 7:30 AM|
My graduate courses in educational measurement in the 1970s taught us to grade or score student papers by literally sorting them into piles—an incredibly primitive approach compared to today’s rubrics. I’ve always wondered where rubrics came from, and this summer I did some research and found out.
The grandfather of rubrics was Paul Diederich at Educational Testing Service. In 1961, he and two colleagues conducted a factor analysis of comments made by readers on thousands of papers that they had sorted into piles, as I was taught, without any guidance other than their own preferences. (Back in those days, before modern computers, doing a factor analysis so difficult and complex that it was sometimes accepted as a doctoral dissertation!) Diederich and his colleagues identified five factors related to those sorting decisions:
• Ideas: relevance, clarity, quantity, development, persuasiveness
• Form: organization and analysis
• Flavor: style, interest, sincerity
• Mechanics: specific errors in grammar, punctuation, etc.
• Wording: choice and arrangement of words
Diederich and his colleagues also found that only the last two factors—mechanics and wording—correlated with scores on the writing tests of the day. So clearly traditional writing tests were inadequate to assess overall writing skill and a new approach was needed.
By 1974, Diederich evolved these factors into a simple “rating slip” for evaluating writing. The rating slip was a simple five-point rating scale with eight criteria organized into two categories:
• General merit
Ideas and organization were given double the weight of the other criteria. Two years of testing led to descriptions for High, Middle, and Low ratings for each criterion. Thus Diederich pioneered what we now call an analytic rubric.
Two of the first people to use the term “rubric” were Charles Cooper at what is now called the University at Buffalo and Richard Lloyd-Jones at the University of Iowa, both of whom contributed chapters to a 1977 monograph called Evaluating Writing: Describing, Measuring, Judging published by the National Council of Teachers of English. Lloyd- Jones used the word “rubric” to describe a five-point scoring guide that today we would call a holistic rubric.
Cooper advocated evaluating writing with a “scoring guide which describes each feature and identifies high, middle, and low quality levels for each feature”—in other words, a rubric. Cooper chose to call this a holistic evaluation, because it considers all the attributes of effective writing, as opposed to the writing tests of his day that focused on isolated components such as vocabulary or sentence length. Cooper enumerated seven types of holistic assessment, most of which are probably rarely if ever used today. But one—the analytical scale—resembles today’s analytic rubric: “a list of the prominent features or characteristics of writing in a particular mode. The list of features ordinarily ranges from four to ten or twelve, with each feature described in some detail and with high-mid-low points identified and described along a scoring line for each feature.”
Ironically, Cooper used the word “rubric” for something different than his scoring guide. He mentioned that Advanced Placement raters followed a “rubric”’ worked out in advance, but the rubric was concerned mainly with the relevance and content of the writing sample, not its general writing features.
Also in the mid-1970s, “Standardized Developmental Ratings” were developed for use by raters of the New York State Regents Exam in Writing, but I haven’t yet unearthed the details.
Diederich’s work, incidentally, led directly to the 6+1 Trait Writing rubrics developed in the 1980s by teachers working with the Northwest Regional Educational Laboratory (now Education Northwest). The rubrics provide separate scores for:
• Word choice
• Sentence fluency
• (optionally) Presentation.
The 6+1 Trait Writing rubrics are widely used today to assess student writing at the K-12 level. Susan Brookhart has cited research demonstrating that using the rubrics has significantly increased student writing skills.
Today rubrics—and the term “rubric”—pervade higher education. Research at NILOA (National Institute for Learning Outcomes Assessment) has found that U.S. provosts identify rubrics as one of the top three most valuable or important approaches for assessing undergraduate learning outcomes, and over 60% of U.S. institutions of higher education use rubrics for institution-wide assessments and at least some academic program assessments.
We’ve come a long way from sorting papers into piles!
Categories: Clearing the Fog