To effectively measure the impact of OER, researchers in the field have developed frameworks. We’ll review some of these frameworks in this section to ground the remainder of our conversations in the guide. Three frameworks stood out as most popular: COUP Framework, S3 Framework, and Investment to Impact Framework.^[1]

COUP Framework

Bliss, Robinson, Hilton, and Wiley (2013) provide the COUP framework as a model for studying the impact of Open Educational Resources (OER) and open pedagogy in secondary and postsecondary education. COUP stands for:

• Cost – What are the financial impacts for students and other stakeholders of OER adoption?
• Outcomes – How does OER influence student academic performance?
• Usage – In what ways do faculty members, as well as learners use OER?
• Perceptions – What do faculty and students think about, and feel toward, OER?

John Hilton III (2017) offers an excellent overview of some of the studies conducted using the COUP framework, broken down into each category of the framework, in Special Issue: Outcomes of Openness: Empirical Reports on the Implementation of OER. This framework is widely used in the field, as numerous experts observe that the COUP framework highlights four critical aspects in OER research (Clinton, 2019).

S3 Framework

This framework (Wiley, 2020) features three components to evaluate the impact of OER: success, scale, and savings. Wiley offers an explanation of how each component is measured:

1. Success: Letter grades are used as the metric/indicator of success in this framework. Wiley suggests taking an average of grades across terms before the OER was introduced compared to after — noting that data from multiple terms would offer a more stable measurement.
2. Scale: Wiley explains that scale is determined by counting the number of students enrolled in OER course sections. This includes the total number of students in all sections of courses, such as Intro to Psychology using OER, as well as the number of students in particular portions of such courses.
3. Savings: The last element in this framework is savings. Wiley notes that “when calculated accurately, the savings measure takes into account several factors” including  diverse pricing of materials and instances of zero spending on course materials. Savings are assessed by comparing the average expenditure of OER users with that of control students.

Three questions that the S3 framework focus on are:

1. How much does this innovation improve student success?
2. How many students are benefiting from this innovation?
3. How much money does this innovation save students?

Investment to Impact Framework

Ebner et al (2022) offer a robust overview of some of the OER impact studies conducted.This includes a study by Jenkins et al. (2020) and their findings which show that OER can improve social justice in higher education.

After a robust literature review, Ebner et al (2022) highlight what  they consider “important new insights” around OER impact assessment:

• Diversity in OER purposes in universities
• Invisibility of the re-usage
• Overlooking innovations
• Differences in the OER infrastructure and maturity of the implementation
• Huge methodological variety
• Visualization and presentation of results

Ebner et al (2022) provide a framework for OER impact assessment, based on their research. They note how imperative it is to clearly define OER and distinguish the ways in which OER will be measured. They use UNESCO’s definition of OER (2019). Their framework also clearly connects to open educational practices. Ebner et al. suggest a practical approach to get started: look at existing OER policies or similar strategies for purposes, arguments or aims related to OER. This framework has developed a good argument for helping institutions understand that there’s no one way to measure impact,  rather it does show all the aspects people need to think about and need to think within their unique context, whether institutional or geographical. This will support researchers as they’re looking into the specifics of what their institution’s OER should achieve and who is the target audience.

Analysis of Frameworks

We found the S3 Framework and COUP Framework focus too much on the quantitative and looking at impact after the fact, rather than preparing for measuring impact while planning out initiatives. The Investment to Impact Framework provides a more holistic tool for examining the complex and very often context-dependent impact of OER in higher education. Ebner et al. (2022) present this framework, emphasizing the importance of looking at OER efficacy not as an after piece but rather as something that has to be part of the planning as you are setting up your initiative and it has to match your own goals/research and examination for efficacy to their own vision and goals.