Social policy research is littered with examples of small programs that show tremendous success when initially implemented, but cannot be replicated. The story of the City University of New York’s (CUNY’s) Accelerated Study in Associate Programs (ASAP) – and the work that MDRC has done with CUNY to adapt, replicate and evaluate this program – is an excellent counterexample.
Only 20 percent of students who enroll in community college will earn a degree within 3 years. [1] In poor, urban areas, the rates are even lower. Research suggests that a variety of factors contribute to these high dropout rates. [2] Many community college students lack basic math and reading skills, so they are required to take developmental education courses that do not count toward a degree and often prove a dead end. [3] While the cost of tuition at community college is minimal, students typically work to cover living expenses. Community college students must juggle work, school and family responsibilities to a greater extent than most traditional four-year college students. This exacerbates academic challenges and makes even small, unexpected expenses difficult to manage. [4]
Finally, students often struggle with the lack of structure and support they find at community college, a task that one observer describes as, “navigating a shapeless river on a dark night.” [5]
Problems like this present a unique challenge for policy researchers. In assessing the efficacy of particular programs or policies, it is critical to disentangle a true causal effect from mere correlation. This suggests that one must utilize a randomized control trial or some type of very convincing quasi-experimental design. At the same time, the problem is so complex that it is unlikely that one will find the right mix of interventions without considerable trial and error. This means that it is important for each new design and evaluation effort to build upon prior research in a systematic, strategic and conscious way. Finally, because the impact of any given intervention likely differs across student groups as well as colleges, it is important to be able to adapt models for different settings.
The ASAP program seeks to address multiple barriers to community college completion simultaneously. Students in the program must enroll full time in an Associate Degree program. They receive free tuition and textbooks as well as a MetroCard to access the city’s subway and bus system. They attend classes in cohorts designed to provide a sense of community, and are provided with frequent, personalized advising and (if necessary) tutoring.
As described last year in Evidence Speaks, the program was wildly successful. [6] A randomized control trial conducted by MDRC found that ASAP doubled the three-year graduation rate, from 22 percent to 40 percent. [7]
These results are exciting, but not unprecedented.
Recognizing the value of determining how to scale the program, CUNY and MDRC set about to test the ASAP model in other settings. Working with three community colleges in Ohio and a group of funders led by the Great Lakes Higher Education Corporation and Affiliates, the ASAP model was modified slightly to fit a more decentralized community college system that serves a different population of students. [8] Like the original model, the Ohio programs provided a suite of wrap-around services intended to simultaneously address multiple barriers facing community college students. [9] The early results from MDRC’s randomized control trial of the Ohio programs are encouraging. In their second semester, 81.7 percent of program students were still enrolled in college compared with only 69.7 percent of control students, a difference of 12 percentage points.
In both contexts, the ASAP intervention was initiated and operated by the colleges themselves. In fact, the model requires considerable institutional capacity. Schools must devote real resources to develop, implement and run the program. In each of the Ohio colleges, staff were devoted exclusively to the program, led by a full-time program director who reported directly to senior leadership in the school. Perhaps even more importantly, the program requires a willingness to rethink traditional practices, and coordinate across multiple stakeholders within the college. [10]
The higher education community has been inspired by the success of CUNY’s ASAP, and CUNY is working with colleges in California and New York State to replicate the program. In many communities across the country, however, local colleges simply do not have the financial resources or institutional capacity to implement the ASAP model. Moreover, reform efforts may be driven by stakeholders outside the colleges. In Detroit, for example, the business community and elected officials would like to support students entering a variety of local community colleges as part of a scholarship program known as the Detroit Promise. While the Promise scholarship removes some of the financial barriers to college access, it does not address other challenges students face.
This naturally raises the question: Are there other ways to provide comprehensive supports to students?
The Detroit Regional Chamber, working with MDRC, developed and is currently testing a coaching intervention known as the Detroit Promise Path, which is intended to support students receiving the Detroit Promise scholarship. [11] Students in the program are required to meet with their coach twice per month. The idea is that coaches can help students navigate a wide range of challenges they may face, ranging from financial hardship to academic difficulty to feelings of isolation. While the program operates at the Chamber, outside of the colleges, all of the coaches are familiar with the schools so that they can refer them to available campus resources and help them to navigate the bureaucracy. Moreover, coaches provide students with emotional support and encouragement. The program also helps students stay positively engaged in school or in jobs programs during the summer and provides financial incentives to attend coaching sessions.
MDRC is using a random assignment design to evaluate the impact of the program, starting with the cohort of students entering college in fall 2016. Initial program results show some promise. In their second semester, roughly 60 percent of program students were still enrolled in college compared with only 49 percent of control students, a difference of 11 percentage points. The difference in full-time enrollment was even larger.
In an effort to better understand the impact of the Detroit Promise Path, the University of Michigan Youth Policy Lab has partnered with MDRC and the Chamber. A recent survey of study participants sheds light on several mechanisms that may contribute to improved persistence. Program students report more frequent interaction with student service personnel. For example, roughly 40 percent of program survey respondents reported meeting with an academic advisor at least three times since starting college compared with only 25 percent of control respondents. While over half of program and control respondents met at least once with a career counselor, 20 percent of program students reported meeting at least three times with a career counselor compared with only 2 percent of control students.
Similarly, program students are substantially more likely to report having a good understanding of things such as financial aid, academic requirements and course selection. For example, 56 percent of control students indicate a high level of understanding of academic requirements compared with 74 percent of program students. Finally, program students are more likely to agree that they have the financial resources and other support necessary to succeed in college.
On average, earning an associate degree has a big economic payoff. Students who complete an associate degree earn $5,000-$7,000 more per year compared with observationally similar students who enter a two-year college but do not graduate. [12] Given these economic returns, all three of the models described above would likely pass a simple cost-benefit test. At roughly $5,400 per enrolled student per year, the CUNY’s ASAP program was expensive but actually lowered the cost per graduate. At less than $3,000 per enrolled student per year, Ohio’s ASAP was similarly cost effective. The Detroit Promise Path is cheaper still, at roughly $1,000 per student per year.
College promise programs like the Detroit Promise have grown dramatically in the past decade, increasing the need for programs designed to support low-income community college students. [13] Policymakers should take steps to encourage colleges to adopt successful models such as ASAP and the Detroit Promise Path. Performance-based funding can help encourage colleges to focus on completion as opposed to mere enrollment. But given that the programs described above involve significant cultural and organizational change, funding incentives alone will probably not be sufficient. The programs described above provide a good example of one path forward.
Brian Jacob is the Walter H. Annenberg Professor of Education Policy, Professor of Economics, and Director of the Center on Local, State and Urban Policy (CLOSUP) at the Gerald R. Ford School of Public Policy.
This post originally appeared as part of Evidence Speaks, a weekly series of reports and notes by a standing panel of researchers under the editorship of Russ Whitehurst
1. https://nces.ed.gov/pubs2017/2017144.pdf
2. Specifically, 26 percent of CC students earn a BA or AA within six years, 34 percent complete any credential (including certificates), and 20 percent are still enrolled at the six-year mark (46 percent have no degree and no longer enrolled). Stats are from the Beginning Postsecondary Survey 04/09.
3. https://www2.ed.gov/about/offices/list/opepd/education-strategies.pdf
4. Karp, Melinda Mechur. 2011. “How Non-Academic Supports Work: Four Mechanisms for Improving Student Outcomes. New York: Community College Research Center, Working Paper No. 28.
5. Scott-Clayton, Judith. 2011. “The Shapeless River: Does a Lack of Structure Inhibit Students’ Progress at Community Colleges?” New York, Community College Research Center, Working Paper No. 25.
6. https://www.brookings.edu/research/fulfilling-the-promise-of-community-college-the-asap-demonstrations/
7. https://www.mdrc.org/publication/doubling-graduation-rates
8. https://www.mdrc.org/publication/bringing-cuny-accelerated-study-associate-programs-asap-ohio
9. However, it is notably less expensive – at only $3,000 per student per year on average – and does not include some of the components of the CUNY model, such as course sections designed solely for program students.
10. Shortly before the ASAP demonstration in Ohio, the state implemented a 100 percent performance-based funding system for community colleges, which likely provided school leaders an incentive to explore new strategies to improve student outcomes.
11. https://www.mdrc.org/publication/enhancing-promise-programs-improve-college-access-and-success
12. Belfield, C. R. and Bailey, T. (2017). The Labor Market Returns to Sub-Baccalaureate College: A Review. Technical report, CAPSEE Working Paper). New York, NY: CAPSEE. Similar results are found for Michigan: Dynarski, Susan, Jacob, Brian and Kreisman, Daniel. “How Important are Fixed Effects and Time Trends in Estimating Returns to Schooling? Evidence from a Replication of Jacobson, Lalonde and Sullivan, 2005.” Journal of Applied Econometrics (forthcoming).
13. For more information on college promise programs, see: http://ahead-penn.org/creating-knowledge/college-promise, and Miller-Adams, Michelle (2015). Promise Nation: Transforming Communities through Place-Based Scholarships. Kalamazoo, MI: W.E. Upjohn Institute for Employment Research.
