This section of The Rebus Guide to Publishing Open Textbooks (So Far) will help you coordinate and implement review processes on your OER, including advice on deciding which of the review processes to include in your project, the tools to use, creating a review guide, and how you can manage revisions with an open mind to strengthen your OER.

What is review and why does it matter?

The ultimate purpose of review is to ensure that your OER is well-structured and ready to be used in the classroom. Review can help you get critical input and suggestions for change that will make your OER even stronger. The way we see it, review happens at many different stages. It can happen as feedback during the project scoping phase, or growing & managing teams, as you’re thinking about the voices that are represented throughout. We believe this approach is helpful to create valuable resources that center student experiences, create equitable classrooms, and genuinely support student success.

It’s also a chance to dispel the notion that “if it’s free, then it can’t be good,” because as we know — the OER you are creating are certainly high-quality learning resources. Review signals to prospective adopters that the work has passed through rigorous quality control, and that its content is suitable for use in the classroom. Not only does it give a public indicator of usefulness to potential adopters, but from experience we know that external reviewers in the field will very often end up adopting the text they’ve reviewed themselves, so it’s great ‘advertising.’

Finally, this stage also provides you and your teams with another valuable opportunity to be intentional and ensure that your review process is equitable. There are many ways to get feedback and input on your OER – either from peers, those tasked to focus on the accessibility of the materials, and/ or the people who work with the OER in classroom settings. At heart, review is about bringing more hands on deck to invest in and help improve your resource.

Types of review and tools: what’s right for you?

There are many ways to get feedback and input on your OER – either from peers, those tasked to focus on the accessibility of the materials, and/ or the people who work with the OER in classroom settings. Consider what you would like to get out of the review process – are there particular expectations or goals? For instance, you may want peer reviewers to ensure that the OER’s content and tone are consistent for an undergraduate audience at the first-year level. Or you may want to improve teaching and learning with the resource by gathering feedback from teachers and students using the OER in the classroom. Or you may want to allow for accountability on accessibility standards by reviewing the resource in all formats with an accessibility lens. Setting these goals will make it easier to prepare a guide for reviewers, recruit the right experts, and ensure that you get something useful out of this process.

Before you begin looking for reviewers, it’s important to decide what kind of review you would like to conduct on your project. We think all kinds of reviews are valuable and therefore ‘prestigious,’ so when deciding which is right for you, look back on the goals that you have set with your team earlier. In what follows, we want to provide you with a quick overview of three different types of reviews you can choose to conduct:

Peer review

A peer review is a critical read of your resource from another subject matter expert, useful to identify gaps and inconsistencies in your resource. ‘Peers’ can offer constructive feedback and solutions to improve the quality of educational content. We want to emphasize that the term ‘peer’ does not solely mean academics – consider what types of feedback you need and who can speak to the quality of your content besides another instructor — would an industry expert or community member be able to provide valuable input? In academic and educational publishing, there are many terms that are used for different types of peer review. At Rebus, we like to keep things simple, so for us, the biggest choice is whether your review will be anonymous or not.

In an anonymous peer review process, authors and reviewers’ identities will be unknown to one another and, potentially, editors. Conversely, if you don’t want to keep the process anonymous, these identities will be known, which might allow authors, editors, and reviewers to be in communication if need be. When deciding which is right for you, look back on the goals that you had set with your team earlier – if your priority is to follow a largely formal/traditional process for the perceived “prestige” value (we think all kinds of review are equally valuable and prestigious, but the reality is that opinions and perceptions differ), you might want to keep things anonymous. Or, if you want to be able to engage collaboratively with and publicly credit volunteers, you may want to know who they are from the start. What’s important is that you have experts looking through the book, and identifying areas for improvement.

If the purpose of review is truly to make this resource better and more accessible for all students, then we need to be looking beyond just reviewing the content and really contextualizing the resource for our students to improve the quality of the OER and the learning experience. To do this, we encourage you to reflect, recognize, and minimise biases in peer review. Think back to your Student Learning Outcomes — whose subject matter perspectives are needed to help determine whether the OER is built to help students achieve these outcomes? Below are some discussion questions for you and your team to consider while creating review processes and workflows:

Accessibility review

Accessibility is about ensuring that what you are making, whether it’s a website, drawing, video, textbook, etc. can be used and understood by all people, regardless of location, language, context, tools, disability, or more. Accessibility can be broken into two criteria: content and technical.

Content accessibility prompts us to think of questions like:

• Is the content created with the audience in mind?
• Is the content well-structured so that the learning objectives can be met?
• What’s the reading level of the content?
• Is it written in an understandable tone or vocabulary?
• Are things like images, tables, etc. used effectively, and only when necessary?

Technical accessibility, on the other hand, is a more specific way to think about this through the lens of web accessibility. According to the Web Content Accessibility Guidelines (WCAG): “Web accessibility means that websites, tools, and technologies are designed and developed so that people with disabilities can use them… [It] also benefits people without disabilities.”

A bulk of the accessibility work can take place during the content creation stage. So you need to build best practices into your workflow. Otherwise, remediation will prove to be very time-consuming, expensive, and inefficient. It’s important to recognize that we don’t have to know the best method or approach for addressing a particular concern in all situations. Instead, lean on those you are collaborating with to find solutions.

The accessibility review involves a thorough run through the different output formats of your OER looking specifically at the web accessibility in each format. A specific set of accessibility criteria can guide the people in your team who are tasked with this form of review to ensure that your resource meets the desired accessibility standards. We believe it to be a vital part of the publishing process and creating inclusive OER.

You should conduct a set of final checks on the books to see what accessibility standards are met and to identify areas for improvement, corresponding to the BCcampus Accessibility Checklist. Remember, the goal is to make as accessible an OER as you can, knowing that there is always opportunity for improvement down the road.

Classroom review

The classroom review, or piloting your resource in the classroom setting is something you won’t be doing for a while, but it is important to plan for it in advance as it will be the true test of your OER with its intended audience! Classroom review involves collecting feedback from both the instructor(s) who will be using the materials to teach and the students who will be learning with them.

This form of review is particularly powerful because it invites feedback from the students which ultimately will help your team to determine necessary improvements for future iterations.

Students can also help you pay attention to non-‘academic’ measures on the book to see the impact that your OER could have — for instance, are students finding the content more intellectually stimulating thanks to the OER? Are they persevering more in the course? Is this work or class participation joyful? Do they have more agency as learners, as individuals in a classroom?

As for the feedback you can get from instructors using the book, it might provide you with a slightly different perspective than you will get from your peer reviewers. Because instructors are knee-deep into the teaching with your resources, they could suggest changes to the flow, or suggest how particular concepts could be aided better through different examples or further explanations. Instructors might also give you ideas for what other types of supplementary materials would be most useful. As you can see, this is just as important as peer review in many ways!

Finally, once you’ve decided on what type of review(s) you will be conducting, you should discuss at what stage you would like to share the content with reviewers – before it has been edited, or after. This choice might be affected by the resources you have access to as a project, as well as what you are asking reviewers to focus on during the review. Do note, though, that other project tasks could be taking place while review is ongoing (such as accessibility checks, glossary development, permissions, etc.), so there’s no need to bring you project to a halt for this stage.

The review process

Similar to Creating & Editing Content, putting in planning and decision-making time up front is one of the most important things to do. Transparent documentation and good project management helps your teams understand expectations, keep consistency throughout the resource, and creates a positive experience for everyone involved!

Assign a review coordinator

If you’re reading this guide, chances are you’ve decided that you want your OER to go through a review. But how do you actually go about organizing this process? The first step should be to decide who will oversee the review process. It may be that you’re happy to coordinate the process, but if you need help, or you don’t want to have contact with your reviewers, we recommend recruiting a review coordinator. Whoever acts as the coordinator, they will be responsible for preparing the review guide, managing recruitment, tracking reviewers’ progress, and relaying information between the reviewers and the authors, editors, and/or project managers as needed.

Once you have a coordinator there are a few things that you and your team should discuss.

Create a review guide

Once all the major decisions regarding the review process have been made, you can turn your attention to preparing guidelines for reviewers. This may be something the review coordinator takes on, or might be a team effort, with input from the project managers and editors.

If you like, you could start with our review guide template, and fill in information about your project. Make sure to include details about the audience, guiding questions specific to the project (these should tie into the goals discussed earlier), deadlines (multiple, if review is taking place on a rolling basis), tools and how to provide feedback, and compensation (if any) or other ways that reviewers will be credited or recognized in the project and final text.

To better structure your review process and support reviewers, work with your team to identify expectations and central guiding questions in your review guide. This will direct their focus to specific criteria, and use their time judiciously in their review. There can be many different lenses/criteria to keep in mind when reviewing the resource, many of which are in our review guide. We suggest coming up with 3-5 central questions to keep things manageable. These questions could be about the DEI additions to the book, the interactive elements created, or a careful review of how materials work to help students meet the Student Learning Outcomes (SLOs). Include these types of questions in your review guide, or let them inform the focus or guiding questions you add in your review guide (eg. for a healthcare OER you could ask: Does the content reflect, value, validate, or reference non-western medicinal practices?). Crafting these questions will help your reviewers prioritise revisions in these key areas. The review guide is also a good place to offer some guidelines around compassionate reviewing — remember, the purpose of review is not critique for the sake of but rather to offer positive and constructive feedback to improve the resource!

This sample review guide follows the structure outlined above, and can be adapted — this example review guide incorporates project specific guiding questions with the core components of review. Both example guides are licensed CC BY, and you can refer to or adapt each as suits your project’s needs.

Recruiting reviewers

Finding reviewers can be a challenging task, but far from impossible. We have a separate chapter dedicated solely to recruitment efforts, which you are welcome to read through, but here are some takeaways specific to finding peer reviewers.

The key to successful recruitment is having a clear and concise call. Your call should be short, informative, and direct interested candidates to the right place. You should aim to include the following details: a brief description of the project, with a link to the public listing page; the intended audience; criteria for reviewers based on what you’d like to get out of the review process; and clear directions to volunteer. Here’s a sample call posted on a public forum, and an example of various kinds of copy used for emailing potential volunteers.

You can also include a few lines encouraging members of underrepresented and/or historically excluded groups within the community and people working in different contexts to your own to apply (e.g. different regions/countries, different kinds of institutions etc.) – having a wide range of perspectives assessing your content will help to ensure that it meets the needs of all students in the long run!

Before sending out the calls, you should also be prepared to track responses from potential reviewers. Use the team’s preferred tools, such as Google Sheets, to collect a list of interested candidates, confirmed reviewers, contact information, deadlines, and status. Doing so will make it easier for you to oversee the process, and not have to sift through emails or chats for an update. We’re prepared a review tracking template that you could copy and adapt for your project.

Once the call and the tracking sheet are created, you’re ready to send out your call! A great place to start is mailing lists or listservs that are specific to your field or discipline. You can also share the call with the Rebus Community, on social media, and in your personal networks (and encourage others on the team to do the same). A good option is to have each team member write up a list of all the people they think are qualified and who might be interested, and send them each an email asking them to act as a reviewer. The personal touch goes a long way! It may take a week or two for you to start receiving responses, so be patient, but if you find that you haven’t had the response you hoped for, you could also send out cold calls. This involves some good old fashioned internet sleuthing to find faculty you think might be suitable for the role. Sending out cold calls is time consuming, and can have low response rates, but if you pick your candidates wisely (making sure their expertise aligns with the project), you might be surprised at how willing people are to be involved.

Lastly, many open textbook projects don’t have much in the way of funding, and often none at all, so financial compensation for reviewers won’t always be possible. While we do encourage you to look for sources of funding for your reviews, we fully understand that this simply may not be possible in all cases. Reviewers may ask about compensation outright, as it is fairly common to be paid for review of other scholarly works, so make sure you have an answer for them. If no funds are available, you could let them know of any other recognition you could provide them, and to reiterate the importance and potential impact of your project. Money isn’t the only motivator, so share a bit more about the project and see whether this inspires them to participate. For example, it’s a nice gesture to send a physical copy of the book to the reviewer as thanks, if you’re printing copies of the book, and have the capacity to do so. Alternatively, a thank you card or personal email can do the trick – or an offer to return the favour and review their future open resource.

Managing the review process

Of course, the ‘review’ stage doesn’t end after the review has been submitted, and you’ve shared your thanks with the reviewers for their work. The obvious next step is to figure out how to incorporate the feedback! This is most often done by the authors themselves, with the editors or project managers coordinating, but sometimes the editors may take on the changes themselves. There are also inevitably decisions around which suggestions to take, and which to defer or leave aside. We outline what the peer review process can look like below (note: this might look slightly different on your project, or for a classroom or accessibility review).

As responses start filtering in, you can update your tracking spreadsheet and assess whether the respondents are right for the role. Make sure to respond to reviewers who have been selected, and offer a note to the ones who haven’t! You can confirm details with selected reviewers, sharing a copy of the review guide and setting a clear deadline for completion of the review (we recommend allowing at least 4-6 weeks if you’re asking them to look at a book-length resource).

During this exchange, you should also (of course) share the content with the reviewers, or get this to them as soon as possible. Ideally, you’d like to send content to reviewers immediately after they express their interest in the task, so as to avoid any waiting around.

Once all the details are confirmed and content has been sent, you can let the reviewers start the process and largely leave them be. However, we do recommend a series of check-ins leading up to the agreed upon deadline to make sure that the reviewer is all set, that they don’t have any questions, and to make sure they are still committed to completing the task. You can check-in as many (or few) times as you like. We recommend doing so:

• 2-3 weeks after you confirm the details with them
• A week before the deadline
• One (business) day after the deadline if they haven’t submitted
• Then as often as needed to get a response from them before it is clear they are no longer participating in the project

The next step is to collect the reviews as they roll in, and pass them along to authors or editors. Be sure to send thanks to the reviewers for their time and feedback. You should encourage them to adopt the book when it is available for classroom use, and let them know you’ll share updates about the project with them as things progress.

Depending on the nature of your project and the type of review process you have selected, you can also ask reviewers if they would like to communicate with authors or editors as changes are being implemented. We also suggest you send ongoing updates about the project to reviewers, especially when the OER is released. More often than not, they’re excited to hear about the progress, and (very importantly!) they could be potential adopters once the book is out. Share the announcement of the resource’s release with them, and encourage them to spread the word in their institutions and networks – as reviewers, they’re in the perfect position to vouch for the text.

And finally, it’s also good to ask reviewers to share their feedback about the process overall, so you can keep it in mind for the rest of the project!

Compassionate reviewing

Another important piece of OER review is to ensure that you are not soliciting reviews that are unduly harsh and that you manage revisions with compassion! Reviewers should keep in mind that they are providing feedback on something that has taken your teammates a lot of time. Highlight both the areas that need improvement as well as those that have been executed well — the latter helps offer a model that revisions can work towards. Have an explicit section in your review guide that highlights that reviewing isn’t just about critique – it’s a collaborative dialogue.

At the same time, remember that reviewers are people too! They might have their own reasons for not participating, stepping down from the project, or requesting delays. While you have a timeline that you’d like to follow, try to be understanding as the inevitable obstacles or delays come up. Look out for the people involved in your project, and if anyone does have to drop out of the job they signed up for, think about how else you might be able to keep them engaged.

When it comes to revisions, be sure to keep an open mind and use your student learning outcomes and primer questions in the review guide to help your team make these suggestions to strengthen your OER. Screen the comments and identify any low priority suggestions, harsh phrasing, etc. & address these challenges before it goes back to the author. This is just one of many feedback opportunities your team will have. There will be time to revisit this work in the future based on feedback from adopters, students, & their own experience of using these resources in the classroom. Remember: OER are living documents, so your resource will likely grow and evolve as it is used.

Conversely, be sure to thank reviewers for their service! Reviewers are often involved because they believe in the value of the project, especially if they’re volunteers, and as the one in charge, you can make sure they enjoy the experience and feel appreciated!

After the review

Once your OER has gone through a robust review and revision process, what happens next?

Earlier in this guide, we mentioned that how you discuss and describe your work – internally and externally – is important as it informs the decisions you make in your OER. Storytelling is a meaningful tool to help you communicate the people, purpose, and vision of your work through every stage of the OER publishing process. In fact, an underlying goal for every stage of OER development should be community building and engagement – there’s nothing better than the words of others to vouch for your resource because individual testimonials indicate a genuine stamp/seal of approval.

We recommend that once all the reviewers have shared their feedback and changes have been made to the content, it’s good to include a Review Statement in the book back matter. By creating a review statement and accessibility statement and adding it to your book’s front or back matter, you send potential adopters or readers a few clear signals:

• That the OER is high-quality and meets comparable standards as other textbooks/learning materials
• That it has been built with care and attention to student learning, especially if you name your reviewers in this statement
• That feedback and iteration is built into your approach – the OER can be improved over time

This is also a great chance to list reviewers (if agreed that they’ll be credited) and publicly recognize their work on the project. Here’s a review statement template that you can work with, and you can take a look at an example statement from another open textbook. Keeping compassion in mind, remember that real people have put in the work behind the project, and it’s vital to highlight and uplift that!

We believe the most powerful tool in OER is an active community, so down the road, you can think of this broader reviewer community as one to stay connected to to offer updates and news about the OER. And whether you work with just an internal team of reviewers or also coordinate with external reviewers, show them your thanks in whatever ways you can. It’s important to recognize the labour behind the resource and make the review processes transparent. This is especially valuable if you are looking for avenues to reward and recognize the work of contributors.

Let’s end with a quick reminder: revising OER is iterative, feedback can help this process take place, and feedback comes from people — so plan and treat those involved with this process with care. That will show through in your final OER and the connections you build around it!