Degree of Difference: What Do Learning Outcomes Say About Higher Education?

This content was previously published by Campus Labs, now part of Anthology. Product and/or solution names may have changed.

White Paper overview

At the time of this writing, The United States Census Bureau states the country’s population is 328.6 million people. An interesting bit of trivia, perhaps, but all the more intriguing when one realizes that 44 million of those people (~13.4 percent) collectively owe $1.5 trillion in student loan debt. The authors ask that you indulge us as we write out that full value, zeroes-and-all, so as to truly showcase its magnitude:

$1,500,000,000,000.00

Changes to state and federal funding, massive increases in tuition, and skepticism amongst elected officials have increasingly and unsustainably shifted the cost-burden of higher education on the individual student. Missing from national conversations on cost, admissions and access more broadly, though, are concerns about what students will learn or how they will change as human beings while enrolled at an institution.

This study is an inquiry into student learning, using teaching and learning data to explore the outcomes campuses use to articulate this learning—both overall at an institution and at a program level. These outcomes are the intended “short-and-long-term changes at the individual level in terms of behaviors, knowledge, skills, or dispositions” for each student.

Learning outcomes statements are, or should be, a fundamental aspect of a student’s higher education experience. To determine if there is a meaningful difference between learning at various institutions, this analysis focuses on learning outcome statements and the methods used to assess them to examine these primary research questions:

1. What learning themes emerge from institutional and program learning outcomes statements written at two- and four-year institutions?
2. What assessment methods are used for each theme?
3. How do learning outcomes fit into a learning taxonomy?
4. What assessment methods are used at each taxonomy level?

To answer the primary research questions, the Campus Labs Data Science team gathered 15,521 institutional (ILO) and department or program level (PLO) learning outcomes statements from 73 colleges and universities—all institutions included in this analysis use the Campus Labs platform for learning assessment management. Classification as a two-year or four-year institution (IPEDS ICLEVEL-Level of Institution) was determined using the Carnegie Classification data set for 2018.

This research established outcome statement themes through researcher defined grouped expert rules regular expressions classifier model. After applying inclusion criteria to the sample, 8,428 institution level or program level outcomes remained—and 87 percent (7,320) were tagged with a theme.

Learning Outcome Themes

Three points initially stand out from viewing the results of this analysis of learning outcome themes:

1. There is agreement on the primary learning outcome theme in higher education
2. There may be misalignment between what programs do and what institutions expect
3. A campus’ values may not be displayed in their outcomes as much as they think

Not surprisingly, Intellectual Skills—which encompasses critical thinking, problem solving and reasoning—was the most prevalent theme at both institutional and program levels. Most discussion, research, and media coverage about the purpose and value of higher education centers on intellectual skills, so it is good to confirm that campus intentions communicated through learning outcomes statements also reflect this priority.

Programs communicate a stronger concentration on Technology, Personal Development, Skills, and Career Specific outcomes than institutions, as a whole. Logically speaking, one would expect disciplines to have unique content and therefore a greater focus on discipline-specific skills and technology. But, a greater representation of Personal Development in program outcomes versus institutional outcomes seems antithetical, especially considering the growth of student affairs divisions on campus and their respective work toward student development.

Themes and Institution Type (Two- and Four-year)

As stated earlier, Intellectual Skills represent the highest percentage of institutional level outcomes for both two- and four-year institutions. The same was true for four-year program learning outcomes, whilst Technology was the highest percentage theme for two-year program level outcomes.

Impact and Action: Career Specific Outcomes

As logic would dictate, the Career Specific theme was far more prevalent in program learning outcomes than in institutional learning outcomes. Schools and disciplines with specific accreditation bodies, such as Business and Nursing, are given a set of learning outcomes. Whether these are always well-conceived outcomes is a discussion for another research endeavor.

According to the Bureau of Labor Statistics (2017), it is possible, even likely, that college graduates will change jobs up to 12 times in their lifetime, and this is true for all disciplines. Engineers can become college presidents, speech-language pathologists can become hospital administrators, and English majors can become CEOs. The discipline-specific skills and knowledge learned may be more indicative of fit and potential than institutional outcomes or the brand on a diploma. Prospective students and their families may be wise to look at the learning expectations of specific programs and disciplines and bypass institutional outcomes altogether.

Impact and Action: Campus Reflection

While this study opens a lot of doors for further analysis, the most obvious being the actual achievement of learning, there is one final question to ask about these data—what do learning outcome statements say about educators and higher education?

Campuses may need to do a non-judgmental, yet honest, self-assessment to determine if some themes are not represented because of a lack of resources or comfort with the content. One cannot teach what they don’t know. Is it possible meaningful learning outcomes are avoided because of a lack of experience with or knowledge of topics in technology or global issues? Perhaps, or perhaps not. That is for each institution to decide.

As niches and name recognition become less reliable drivers of enrollment, institutions will need to differentiate from one another by the uniqueness of what learning outcomes they can provide students.

With increased public scrutiny brought upon higher education as a result of scandal, cost, and general disagreement about purpose, it is all the more imperative for institutions as a collective to have processes in place that demonstrate what learning happens on campus, and to what end. On this matter, our study’s findings suggest that most campuses indeed have a long road ahead of them.