Over the last few years, higher education discussions of diversity, equity, inclusion, and belonging (DEIB), inclusive instructors, and inclusive teaching have increased. Institutions have made major commitments to improve DEIB, building a strong community and student belonging on campuses. With the increased focus, we might expect a rise in course evaluation questions asking about a student’s diversity, equity, inclusion, and belonging experience.

We examined data from institutions to determine the percentage of instances where diversity, equity, and inclusion (DEI) experience questions had been included in course evaluations. Our data consists of 175,821 questions spread across 2,580,796 course sections within 310 Anthology Course Evaluations institutions over fifteen years. To determine which questions should be classified, we created and utilized an expert rule and a two-tiered expert rules approach. This methodology used a mixture of quantitative and qualitative methods, along with subject matter experts to analyze the 175,821 questions. Specifically, we were interested in a student’s experience of DEI, not their learning about diversity, equity, and inclusion. Of these questions, only 227 (.13% or about 1/10 of 1%) asked about a student’s DEI experience.

To what degree are DEI experience questions included in course evaluations?

Of the 310 institutions, only 48 (15.5%) have had at least one DEI experience question since 2006. While this highlights the absence of DEI experience questions, we believe these questions are even more scarce than indicated. We may be, in fact, over-inflating the degree to which the institution is assessing because even one instance of a question, over a 15+ year period, asked at an institution would indicate the institution has included questions measuring DEI experience. Another example is if a single faculty member adds a DEI experience question to a single course. In that case, it would not be accurate to suggest that this institution is actively trying to understand its students’ DEI experiences.

Instead, understanding the percentage of courses at a given institution with one or more DEI experience questions is a better view. Figure 1 shows the distribution of institutions’ percentage of courses containing a DEI experience question. This clearly indicates few institutions (16 of 310 or 5.2%) rise above even a 5% threshold of their courses containing this type of question.

Has the prevalence of DEI items increased after 2019?

Overall, we found that there may be a very slight upward trend in the prevalence of DEI experience questions being asked, but this still represents a very small number of schools where such practices are even evident, let alone systemic.

Let’s visualize the prevalence. First, you see the count of institutions that have included DEI experience questions by year since 2010.

Second, the following visualization shows the proportion of institutions across the five prevalence categories. The prevalence category is a binned categorization derived by examining each institution’s percentage of courses with at least one DEI experience question associated with it. We utilize course section level data to show this prevalence because this is most indicative of the number of students that could be impacted versus a simple count of questions being asked. For example, if Hometown University has 53% of courses with a DEI experience question being asked on the course evaluation, then that campus falls into the prevalence category of 50% < x <= 75% (more than 50% of courses, but less than or equal to 75% of courses, with a DEI question).

This data has been normalized using the percent of institutions within that year belonging to each prevalence category; this allows a comparison of the prevalence categories across time. The trade-off is that normalizing removes the concept of how many institutions comprise that year’s data. As the number of institutions increases, the variability of the prevalence of DEI questions will decrease because a single institution will have less impact on the overall trend. You can see that very few institutions across time ever rise above a 25% DEI course prevalence, and generally, this trend is not changing meaningfully, even after 2019. The large mass of the ‘none’ prevalence category dwarfs those institutions that ask a DEI experience question.

Lastly, we focus on just the institutions where a DEI experience question is asked by providing a zoomed view of the previous visualization, with the exclusion of the ‘none’ group. Given the higher prevalence rates groups (50% < x <= 75% and 75% < x <= 100%), it is likely that these are a result of questions at the institution or department/program-wide level. Conversely, the lower prevalence (0% < x <= 25%) rate group likely represents institutions that do not have a common DEI experience question. However, from our research, we know that a portion of the faculty added on these types of questions. We tease out the attributes of who is asking the DEI experience questions in the next section.

Are institutions or faculty more of the driving force for including DEI questions?

This is a complex question to answer. We examined question sets by counts of DEI experience questions knowing that each set type is less top-down than the one above it.

Breakdown of Question Sets:

• Core questions are a set of questions that are asked at the institution level. (e.g., Hometown University)
• Institutional Unit questions are a set of add-on questions (added on to the core set of questions) that a department or program may include beyond the core set. (e.g., In addition to core questions, these questions are added to all Department of Data Science questions at Hometown University)
• Question Catalog questions are rarely used and have been discontinued.
• Course Section Attributes questions are asked at the course section level. (e.g., In addition to core questions, these questions are added to all sections of English 101 at Hometown University)
• Faculty questions are those that a faculty member may add on beyond other question set types. (e.g., In addition to core questions, Professor Anthology adds a question to the three courses section they are teaching only)

It is obvious from these question counts that the majority of DEI experience questions are being driven by faculty, making up 63.4% of all of the DEI experience questions being asked. In contrast, institution-level core questions account for only 11.5% of these questions.

On the other hand, if we think of how much each of these questions impacts students, then looking at the number of courses that the question appears in is a better indicator of the driving force. Those meager 26 DEI core questions impact 66,732 courses, or 85.1% of all courses. That means that for every DEI question asked at the core level, we can expect that, on average, 2,567 courses will see this question.

Faculty questions, while the most frequent source of DEI questions, have a far less reaching impact. In fact, fewer courses were impacted (128 or .2% courses) than actual questions asked (144 questions) because faculty tended to ask multiple DEI experience questions per course in comparison to core type questions. That means that for every DEI question asked at the faculty level, we can expect that, on average, .9 courses will see the question.

In conclusion, very few institutions ask experience questions related to diversity, equity, inclusion, and belonging, and the largest source of these questions comes at the faculty level. Institutions should be asking DEIB experience questions at broader levels to collect student perceptions and feedback on commitments to DEIB. The data generated by asking about these experiences not only in the classroom but at the institutional level can provide an opportunity to understand how well an institution is providing DEIB across departments, programs, courses, and faculty. The inclusion of DEIB experience questions lets students know that commitments to diversity, equity, inclusion, and belonging are an important piece of the academic experience. Including these as core questions communicates to students that the institution is rigorously tracking, understanding, and continuously improving based on this feedback.

Looking to improve Course Evaluation question sets by including DEIB experience questions? Below are sample DEIB questions identified in our research. We recommend you include these questions at the core level.

1. The instructor appreciated diversity and modeled that appreciation.
2. The instructor was open and comfortable with diversity.
3. The instructor structured learning experiences for valuing diversity.
4. The instructor created an anti-bias and anti-racist learning environment.
5. The instructor valued the diversity of life experiences among students.
6. The instructor maintained dignity and respect when interacting with the students.
7. The instructor created an equitable and respectful environment in the classroom for all students.
8. The instructor modeled respect for human dignity.
9. Rank the safety, acceptance, and security of your group experience:
10. Describe how the instructor created an anti-bias and anti-racist learning environment:

To learn more about how Anthology solutions can help you seamlessly deliver evaluations at all levels for your institution to achieve desired results, visit Anthology Course Evaluations.