Selected SLC Research

Policy Analysis | September 9, 2011

Higher Education Performance and Accountability

Jonathan Watts Hull

The Texas Board of Regents approved a sweeping plan last week to increase accountability for the University of Texas System. The changes are intended to enhance the system's efficiency, improving cost factors while simultaneously increasing the quality of education across all 15 units of the UT System. The framework for implementation features nine focus areas, including undergraduate student access and success; faculty, administrative, and staff excellence; research; and productivity and efficiency. This plan builds upon an existing accountability system and measures to close gaps and increase excellence in the System established by the Texas Higher Education Coordinating Board.

A similar accountability proposal announced last week by Missouri governor Jay Nixon would allocate state funds to institutions of higher education based on meeting performance measures and academic goals, including the number of degrees and certificates conferred. The governor's plan allocates future funding increases using a model based on statewide- and institution-specific goals that accounts for the differences between two- and four-year institutions.

The actions in Texas and Missouri dovetail well with calls at the regional and national level for increased numbers of people with post-secondary credentials and to improve the productivity of higher education in general. Several states have been at work on similar performance funding programs in the past few years. In 2010, Tennessee passed the Complete College Tennessee Act, which linked its higher education system and economic development through a funding formula which will, when fully implemented, allocate resources to institutions based on outcomes rather than enrollment figures. Louisiana's GRAD Act similarly allows for institutional performance to determine part of its budget, with alloances for tuition increases for those schools meeting four established performance objectives. Outside the region, several states use similar models to implement accountability in postsecondary education, often based on the National Governors Association's Complete to Compete Initiative, The Delta Project on Postsecondary Education Costs, Productivity, and Accountability, or others.

These efforts take place in the midst of a broader discussion of what accountability means in the context of higher education, particularly in times of economic distress. Early efforts at postsecondary accountability in the 1980s and 1990s focused on institutional measures of success. More recent efforts have sought to measure performance on statewide impacts in areas such as completion, learning, affordability and participation. The economic downturn has resulted in increased pressures on institutions of higher education as state support has shrunk and pressure to keep tuition increases in check has grown.

The mission, purpose, and structure of these institutions do not immediately lend themselves to broadly applicable standards by which to measure performance. Large research universities are unlike smaller liberal arts colleges in a vast range of ways, which are equally dissimilar to two-year and technical colleges. K-12 education reform has focused on standards and the crafting of a common core of standards to build upon. These standards are based on the goal of having all students college- or career-ready. Higher education does not have this common goal, not all graduates are heading for professions, or graduate or professional school. Establishing measures that meaningfully assess school performance while respecting the different missions and goals of the institutions will require a major effort.

Measuring Higher Education Productivity

For the most part, performance measures currently used include graduation rates, first year retention rates, cost per student, cost per degree, cost of first year attrition, student loan default rate (as a proxy for gainful employment), and the number of degrees and certificates conferred. These metrics (which all are publicly reported and easily compared) provide some basis for initial analysis, but serve more as blunt instruments rather than finely tuned tools.

Patrick Kelley of the National Center for Higher Education Management Systems outlines the complexity of appropriate measures and proposes a productivity scale which accounts for a range of factors, including the types of certificate or degrees conferred, weighting this by the economic impact of the various certificate or degree, and estimating the overall system productivity as a measure of how the weighting increases the overall productivity of the state system. His results focus on the relative benefit of certain fields and certain credentials, with those with higher earning potential contributing more. The effect of this is to measure the benefit to the state of its investment in the institution. Kelley's use of state funding per full-time equivalent student allows for cross comparisons between states, although it includes an internal bias against states that have large two-year systems in place, since the certificates these institutions produce are generally weighted lower. Moreover, he concludes that the link between funding and productivity (as measured by his calculus) is relatively weak.

Data Challenges

An additional challenge facing this effort is the limitation of the existing data systems for collecting, sharing and analyzing student and school performance. The U.S. Department of Education is working with states to expand the Integrated Postsecondary Education Data System (IPEDS) to provide more comprehensive information on student and institutional performance. The IPEDS model is principally a consolidation of separate surveys conducted by the Department of Education of every institution participating in federal student financial aid programs. States collect and manage a wealth of data on their own and are making steady, if uneven, progress toward the goal of being able to follow students as they move through all levels of the education system. According to the Data Quality Campaign, states, in general, now have the data to answer critical policy questions, but have not consistently linked this data across education systems or made the information available to stakeholders, limiting the utility of the data to help improve student achievement and system performance. Among all states, Texas has made the greatest progress on the Campaign's 10 State Actions and 10 Essential Elements for using data to create the conditions under which data can be used to promote student achievement.

Increased Access and Affordability

One measure of success for postsecondary education is participation, how many Americans are enrolled in or have completed studies. In order to achieve the ambitious goal set out by the president in February 2009 of having the highest proportion of college graduates in the world will require considerable efforts in increasing participation among both traditional and non-traditional students. The economic downturn has witnessed a remarkable uptick in the number of Americans seeking post-secondary credentials, but it also has seen a jump in the associated costs, with average net tuition rising nationally more than 11 percent between 2009 and 2010 alone.

Access, the ability of every qualified student to participate in higher education regardless of race, gender, or income, has been undercut by rising tuition and shrinking need-based financial aid. In recent years, students have assumed increased debt loads, something that is both unsustainable in the long run and delays other important economic activity for college graduates, creating a drag on the overall economy. Preserving access, even in times of economic uncertainty, has been the central theme of a number of organizations, including the National Center for Public Policy and Higher Education (whose Measuring Up reports often are credited with initiating the recent discussions of higher education accountability). Because the key measures of success for accountability purposes are high graduation rates and low attrition rates, investments in improving college access would be matched with proven practices to improve student retention and completion (some of which were outlined by Dr. Vasti Torres at the SLC Annual Meeting in 2010).

Increased rigor

While most accountability measures of higher education focus at some point on credentialing, graduation rates, and research productivity (as measured by faculty publication in peer-reviewed journals), there is perhaps a more telling data set that points to how successful institutions are at educating students: the National Survey Of Student Engagement and its related surveys measuring how much time and effort students apply to their studies and how institutions organize and invest resources to get students to participate activities research has indicated are linked to student learning. Limited in scope and applicability (the data are not reported publicly by institution and only a small sampling of institutions participate), the survey nonetheless provides a more nuanced window into how well schools (and individual programs within those schools) engage their students in the activities that are most likely to provide benefits. Building on this dataset and combining it with information from the Collegiate Learning Assessment, New York University professor Richard Arum and University of Virginia sociologist Josipa Roksa's study Academically Adrift, illustrates why a measure of student engagement, and not simply degrees and certificates conferred, would be a useful component of an accountability system. Arum and Roksa found that 45 percent of students participating in the assessment did not demonstrate significant improvement in learning in their first two years of college, and that 36 percent of students failed to show improvement over four years. Presumably, many of these students eventually earned credentials even though their academic commitment was limited at best.

Toward a system of accountability

Accountability, in the words of scholar David E. Leveille of the Center for Studies in Higher Education, has become the lingua franca of higher education, with organizations, institutions and influential voices advocating a range of options. As yet, however, there is not a comprehensive understanding of what is being sought to be measured. For the most part, state performance measures account for certificate and degree completions, typically with some calculation as to their total cost, as well as allowances for the beneficial research an institution may contribute and the ability of an institution to matriculate a high degree of its students. Assessments of how well aligned certificate and degree production is with state economic and social demands remains beyond the scope of current models, but cannot be ignored. Remedies for this that establish incentives for schools to confer credentials that meet state or regional needs must be carefully crafted to avoid penalizing institutions that have missions or serve specific populations.

Inevitably, it is the high degree of "moving parts" within higher education that makes establishing accountability systems so difficult. Prior to considering the practical and logistical elements of an accountability plan, however, it is important to be mindful of some of the essential components of a state level higher education accountability system (from Accountability In Higher Education):

  • A Public Agenda that articulates a state's strategic goals and drives decisions for both the state and its colleges and universities;
  • Public Trust in the institutions and undertaking of higher education;
  • Performance Indicators that measure how well institutions and systems accomplish their mission;
  • Data-Driven Decision Making that pulls a variety of sources into a comprehensive and unified system to help inform decision making; and
  • Engaged Stakeholders

Fifty years after California's master plan for higher education helped the state to catapult to the top tier of public college and university systems in the world, the state of California began looking for ways to improve and expand upon its mission. In considering the steps they might take in improving higher education in the state through accountability and sustainability, one of the places they sought answers was, perhaps unsurprisingly, Texas.