Areas of Assessment for OAR/OER - COUP Framework

The COUP Framework

The COUP Framework is the Open Education Group Links to an external site.’s approach to studying the impact of open educational resources (like open textbooks) and open pedagogy in secondary and post-secondary education.

COUP stands for:

  • Cost
  • Outcomes
  • Usage
  • Perceptions

Cost

Adopting openly licensed resources can impact a range of financial and cost metrics for students and institutions. Proponents of OER frequently claim that using these resources instead of traditional publisher textbooks or digital materials will save students money in educational costs associated with pursuing an undergraduate degree.

The Cost criteria for assessment provide empirical evidence about the magnitude of the financial impacts of OER adoption.

Examples:

  • Quantifying the cost of previously assigned textbooks as savings for students
  • Costs associated with the additional purchase of study guides, flashcards, quizzes, etc.

  

Outcomes

Given the folk wisdom that “you get what you pay for,” some individuals and organizations worry that student learning will necessarily suffer when students use freely available, openly licensed resources instead of $200 textbooks. OER proponents claim that using these resources instead of traditional publisher textbooks or digital materials increases student access to critical learning materials and expands faculty’s academic freedom, consequently improving student learning outcomes.

The Outcomes criteria for assessment provide empirical evidence about the magnitude and direction of the learning impacts of OER adoption.

Examples:

  • Changes in the percentage of students receiving a C or better (in the course as a whole, or on a particular assignment)
  • Changes in rates of completion
  • Changes in drop rates
  • Changes in enrollment intensity

 

Usage

The Usage criteria for assessment provide empirical evidence about how students use OER and the degree to which impacts on learning outcomes co-vary with these uses.

Examples:

  • The frequency with which students access learning materials
  • Students report accessing learning materials to prepare for class activities

 

Perceptions

What do faculty and students think about, and feel toward, Open Educational Resources? How do they judge their effectiveness relative to traditional textbooks? Their rigor and coverage? Do they find the formats, structures, and other design features easy to use? Frustrating? What about other stakeholders, like parents or policymakers – what are their thoughts and feelings toward OER?

The Perceptions criteria for assessment provide empirical answers to questions about student valuation of learning materials.

Examples:

  • The learning materials are visually appealing (e.g., layout, colors).
  • The writing is engaging/interesting.
  • The writing is understandable/clear.
  • The learning materials use examples well to explain the subject matter.

 

Review of OER assessment studies using the COUP Framework

Howard, V. J., & Whitmore, C. B. (2020). Evaluating student perceptions of open and commercial psychology textbooks. In Frontiers in Education (Vol. 5, p. 139). 

Open educational resources confer a variety of advantages to both student and faculty users. For students, the obvious benefit is cost, as most OERs are free to access. OERs also permit students access to materials without restriction from the first day of courses (Seaman and Seaman, 2019 Links to an external site.), and students can gain access to materials in a variety of formats and across various devices with little restriction (Watson et al., 2017 Links to an external site.; Ross et al., 2018 Links to an external site.). These benefits also extend to faculty, who can adapt materials to best suit their unique courses and learning outcomes due to the shared permissions offered with open licenses (Griggs and Jackson, 2017 Links to an external site.). Despite these potential benefits, skeptics voice concerns about the quality of OER materials. To address these concerns, the impact of OERs and commercial textbooks have been extensively studied using the COUP (Cost, Outcome, Usage, and Perceptions) framework (Bliss et al., 2013 Links to an external site.).

 

Studies on cost may explore student savings, often estimating textbook cost savings for one or many courses before and after transitioning from a commercial text to OERs (e.g., Hilton et al., 2014 Links to an external site.). Cost studies may also explore ancillary benefits of more affordable textbooks, such as decreased course drop rates, greater persistence, and enrollment intensity (e.g., Fischer et al., 2015 Links to an external site.; Wiley et al., 2016 Links to an external site.). The latter studies tend to occur at scale, utilizing institutional data from one or many institutions. OER adoptions are estimated to have saved students over a billion dollars (Scholarly Publishing and Academic Resources Coalition (SPARC), 2019 Links to an external site.), are associated with greater student persistence in courses (reducing course withdrawals; Clinton and Khan, 2019 Links to an external site.), and students using OERs appear to take more courses both in semesters when using OER and in subsequent semesters compared to commercial textbooks (Fischer et al., 2015 Links to an external site.), generating increased tuition revenue for institutions where OER is used (Wiley et al., 2016 Links to an external site.).

 

Outcomes research focuses on the direct changes in student course performance, such as changes in grade achievement, students’ persistence in education, and enrollment patterns. Notable large-scale studies on OER efficacy include Fischer et al. (2015) Links to an external site. and Colvard et al. (2018) Links to an external site.. Fischer et al. (2015) Links to an external site. reported no statistically significant differences in course grades in most courses (60%), improved grades in some courses (33%), and decreased performance in only one course (6.7%) when faculty transitioned from commercial to OER textbooks. Colvard et al. (2018) Links to an external site. conducted a similar study, evaluating the impact of OER adoption on student performance for students across the university of Georgia system. Colvard et al. (2018) Links to an external site. reported statically significant improvements in final course grades and a reduction in grades of D, F, and W following the transition from commercial to OER textbooks, yet the disproportionate improvements seen in low-income students, non-white students, and part-time students are the most promising results of the study as they demonstrate that OERs may effectively level the playing field for students traditionally underrepresented in higher education. While OER adoption does not universally improve student performance, most studies indicate that learning outcomes either do not change or improve following a transition from commercial to OER textbooks. See Hilton (2016 Links to an external site., 2019) Links to an external site. for thorough metanalyses on the efficacy of OERs.

 

Usage research explores how faculty and students engage with OERs, such as evaluating student engagement with OERs and covarying learning outcomes (Gurung, 2017 Links to an external site.; Clinton, 2018 Links to an external site.; Cuttler, 2019 Links to an external site.). Related to studies explore pedagogical techniques made possible by OER (Wiley and Hilton, 2018 Links to an external site.), such as student generated or remixed OERs (Randall et al., 2013 Links to an external site.; Azzam et al., 2017 Links to an external site.; Jhangiani, 2017 Links to an external site.), and the open sharing of instructional design and pedagogical techniques (Cronin, 2017 Links to an external site.).

 

Finally, perception research explores what a variety of stakeholders – including students, faculty, and administrators – think about OERs. Despite advances in technology, the textbook remains a central feature of most courses, and perception research is particularly valuable when understanding the value of textbooks in a course.