The Impact of Success Boston Coaching on College Outcomes

Antoniya Owens, The Boston Foundation, Hayley Didriksen and Beth Gamse, Abt Associates

March 30, 2017

The benefits of postsecondary education in today’s economy and society are well documented. College graduates are less likely to be unemployed or suffer job losses in economic downturns; they earn higher salaries and are more competitive in the job market. In fact, projections show that by 2020, more than 70 percent of jobs in Massachusetts will be held by workers with postsecondary credentials. And in Boston—a hub of higher education as well as leading industries like life sciences, health care, and high tech—boosting college completion is crucial for ensuring its future competitiveness and prosperity.

It is thus not surprising that in 2008—after sobering news that fewer than four in ten Boston students who enrolled in college completed a degree—Boston’s then-Mayor Thomas Menino issued a call to action to the city’s leaders. That call generated a clear, focused, and coordinated response: Success Boston, a citywide initiative seeking to improve college completion among Boston public high school graduates, particularly those traditionally underrepresented in higher education. Success Boston is a partnership of the Boston Foundation, the City of Boston, the Boston Public Schools (BPS), 37 area institutions of higher education, led by University of Massachusetts Boston and Bunker Hill Community College, and local nonprofit partners, including the Boston Private Industry Council and uAspire. Importantly, the city’s current Mayor, Martin J. Walsh, has also embraced Success Boston, making him one of the few large city mayors in the country to focus on the success of students after they leave a city’s public education system and move on to higher education and beyond.

Central to the initiative is Success Boston Coaching (SBC)—one-on-one transition coaching, in which Boston students in the last year of high school are connected with adult coaches who support them for the first two years of college. Coaches are hired by local nonprofit organizations to help students navigate tasks and challenges—both routine and unexpected—on the path to college graduation (see Exhibit 1). They offer one-on-one support on multiple topics and refer students to available services and resources; their support covers academics, time-management skills, self-advocacy, and planning pathways both for college and, eventually, a career. The SBC network, convened by the Boston Foundation, facilitates communication across organizations and provides coaches with extensive professional development.

Exhibit 1

New Results on the Impacts of Coaching

On March 23, 2017, the Boston Foundation released a new report prepared Abt Associates, an independent research organization, which shows that graduates of the Boston Public Schools who received coaching through Success Boston have significant advantages over other students.ii While previous studies focused specifically on SBC found that coached students have higher persistence rates than non-coached students,iii the rigorous design of the current evaluation provides the first causal evidence that differences in outcomes can be attributed to the students’ participation in coaching. And according to the report authors, the effects of coaching can be characterized as “positive and large.”

Using data from BPS, the Massachusetts Department of Elementary and Secondary Education, nine local colleges, and the Success Boston program database, the study analyzes whether and how transition coaching affects students’ early outcomes in college. It also examines whether and how observed impacts vary for students with different characteristics (gender, race, academic achievement, and type of college) and by features of coaching (frequency, duration, and topic). The primary college success outcomes in this evaluation fall into three domains: persistence, academic achievement, and financial aid. The analysis focuses on 2013 and 2014 BPS high school graduates who entered college in the fall immediately after their high school graduation. The evaluation uses a quasi-experimental design that creates a comparison group of students; the comparison students are as statistically similar as possible to coached students before their participation in coaching on baseline characteristics (e.g., gender, race/ethnicity, high school academic achievement, socioeconomic status). This design rules out the possibility that these baseline characteristics are what accounts for observed differences in outcomes between coached and non-coached students.

Persistence

A key premise of the SBC model is that students are more likely to persist in college as coaches help them navigate the academic, financial, logistical, and social-emotional challenges typically faced by beginning college students. To examine the impact of coaching on persistence, the study uses two outcome measures: persistence into the second and into the third years of college, respectively. These two measures capture whether students who enrolled in college after high school graduation returned to college in the fall of each successive academic year (or had already completed a certificate or degree). Results show that coached students (treatment group) are more likely than non-coached students (comparison group) to persist into their second and third years of college (see Exhibit 2). In fact, 83 percent of coached students and 75 percent of non-coached students persisted into the second year of college—a positive and statistically significant impact equivalent to an 11 percent increase in persistence. In addition, both groups persisted into the second year at higher rates than national average of 72 percent.iv Furthermore, fully three-quarters (75 percent) of SBC students persisted to the third year, compared with less than two-thirds (62 percent) of non-coached students—equivalent to a 21 percent increase in persistence.

Exhibit 2

Academic Performance

Many of the challenges students face when they get to college are academic in nature—such as learning how to select the right courses, how to manage one’s time across academic and life demands, and whether and how to access academic tutoring. Success Boston coaches connect students to campus resources, line up additional supports, and help them to stay motivated and focused. In fact, an earlier Abt Associates analysis of SBC implementation during the 2014-15 academic year found that coach-student interactions during the 2014-15 academic year most frequently addressed academic topics, such as reviewing course syllabi, selecting course, planning the path to a degree, and connecting students to on-campus tutoring services.

To examine whether coaching affects students’ academic performance in college, the current study examined two primary outcome measures of academic achievement: cumulative grade point average (GPA) and good academic standing. Cumulative GPA is based on students’ most recent completed semester, and corresponds with the following values: A=4.0, B=3.0, C=2.0, D=1.0. The study found that SBC students have higher cumulative grade point averages (2.45 vs. 2.26) than comparison students. Although fairly modest in magnitude, this impact of about one-fifth of one point is statistically significant (see Exhibit 3).

Exhibit 3

In the study, students were considered to be in good academic standing if they maintained semester-specific and cumulative GPAs of 2.0 or better, or earned at least 66 percent of credits attempted in a given semester. The measure is represented as a proportion of the number of semesters spent in good academic standing, out of the four possible semesters in the first two years of college. As Exhibit 4 shows, SBC students spend more semesters in good academic standing (78 percent of semesters) than non-coached students (71 percent of semesters). This advantage is important, as it could translate into faster pathways to college completion: for example, students in good standing are not placed on academic probation and may be less likely to lose their financial aid for the subsequent semester.

Exhibit 4

FAFSA Renewal

Coaches also help students with tasks related to locating and securing financial aid. Successful and timely navigation of the financial aid process is critical to ensuring that students have access to sufficient financial aid, and renewal of Free Application for Federal Student Aid (FAFSA) forms is a key step in this process. As a result, the study examines FAFSA renewal as a primary outcome measure. Results show that coached students renew their FAFSA at a higher rate than non-coached students: 85 percent of SBC students renewed their FAFSA for their second year of college, compared with 78 percent of non-coached students (see Exhibit 5). The difference in renewal rates is positive, statistically significant, and equivalent to a 9 percent increase.

Exhibit 5

Less than a year ago, in June 2016, a joint descriptive report by the Boston Private Industry Council and Abt Associates showed a steady increase in the overall college completion rates for BPS graduates—from 40.6% for the BPS class of 2000 to 51.3% for the class of 2009—within one percentage point of the interim 52% goal set by Success Boston in 2008.vi While it is too early to examine the causal impact of Success Boston coaching on college completion for the classes of 2013 and 2014, the current Abt study provides encouraging new evidence of positive impacts on short-term college success outcomes, such as persistence and academic performance.

The magnitude of the impacts of coaching described above is generally comparable to impacts of similar programs seeking to improve college student outcomes; however, the impacts on third-year persistence and FAFSA renewal are larger than typically seen in the literature. Overall, that means the effects of Success Boston coaching can be characterized as positive and large. SBC participants are persisting longer, achieving higher academic progress, and taking the necessary steps to maintain funding for college—all good signs of progress toward college completion. Abt Associates will continue to monitor students’ progress over the next several years and, in 2020, publish a final report on the impacts of coaching, as of five and six years after these students’ high school graduation. The final report will help address questions about whether the positive impacts reported in the present study persist, and whether Success Boston has continued to progress toward its central goal of attaining a 70 percent citywide college completion rate.

• Carnevale, A. P., N. Smith, and J. Strohl. 2013. Recovery: Job Growth and Education Requirements through 2020, State Report. Washington, DC: Center on Education and the Workforce, Georgetown University.
• Linkow, T., Gamse, B., Unlu, F., Bumgarner, E., Didriksen, H., Furey, J., Meneses, M., Sami, M., and A. Nichols. 2017. Interim Outcomes Report: Success Boston Coaching. Boston, MA: The Boston Foundation.
• Sum, A., I. Khatiwada, and S. Palma. 2014. The College Persistence and Graduation Experiences of Success Boston Coaching Participants from the BPS Graduating Class of 2009. Boston, MA: Center for Labor Market Studies, Northeastern University.
• National Student Clearinghouse (NSC). 2016. Snapshot Report – Persistence and Retention. Washington, DC: U.S. Department of Education, National Center for Education Statistics.
• Linkow, T., Dadisman, K., Gamse, B., Didriksen, H., Schwartz, G., Hillard, M., and M. Karuu. 2015. Degrees of Coaching: Implementation of Success Boston Coaching 2014-15. Boston, MA: The Boston Foundation.
• McLaughlin, J., N. Sullivan, A. Van Eaton, T. Linkow, H. Didriksen, B. Gamse, and J. Furey. 2016.
• Reaching for the Cap and Gown: Progress toward Success Boston’s College Completion Goals for Graduate of the Boston Public Schools. Boston, MA: The Boston Foundation.