When the Numbers are Wrong

November 5, 2020

It’s two days after Election Day 2020, and with no clear winner yet in the presidential race, like many of you I’ve been playing with different scenarios on those interactive red-and-blue maps—a game my soon to be 11-year-old daughter loves to play with me—and reading autopsies on what went wrong with the polling.

Polling, like so much else in data science, is based in part on history and past behaviors. But what if the pandemic has changed our behaviors? What if the past is no longer prologue to the future?

While reporting my book on admissions, one of the questions I constantly asked admissions deans is how they were so assured their enrollment models were right: How did they know how many students to admit so they’d end up hitting their target for a class? Did they ever worry about offering too much in financial aid given they didn’t know who would actually accept the money?

So often I was told the models are remarkably stable. “The numbers never lie,” one vice president of enrollment said.

What happens when they do? What happens when teenagers and families shift their behaviors? What if what has happened in the past isn’t as helpful for what’s coming next?

?More after this message…

Are you a high school counselor at a public or private school? If you’re in that role of counseling students about their college plans, I’m hosting a virtual meet-up today at 4 p.m. ET about admissions in the age of COVID-19 and my book. Sign up here to receive registration information. This is open only to high school counselors.

Admissions is often described as a mix of art and science, and in the last decade a new generation of enrollment leaders have started to tip the balance toward science.

• The admissions office, traditionally managed by experience and gut instinct, is now driven by spreadsheets and dashboards operated by leaders who grew up in the era of Moneyball.
• This sea of numbers tell admissions officers who is most interested in the school, who will enroll if accepted, and even how much financial aid it will take to attract them.

Figuring out that sweet spot is the job of quants who have brought sophisticated statistical approaches from Wall Street and Fortune 100 companies to higher education.

• These experts in mathematics, computer science, and psychology often work far from campus at consulting companies that colleges hire to “optimize” spending on financial aid.
• In the last decade, econometric modeling and data mining to find the best incoming class money can buy has been all the rage in admissions, helped in large part by outside consultants.

The enrollment management business is dominated by a few big players and lots of mom-and-pop shops.

• In reporting the book, I got to know one of those shops well: Human Capital Research, founded in 1990 by Brian Zucker.

On the eve of the presidential election, I called up Zucker to ask him one question: Are there any similarities between forecasting an election in a pandemic with so much early voting and forecasting the incoming class of a college next fall?

We ended up talking for 45 minutes. Three things struck me in our conversation.

1. The Volatility of Certain “Cells”

At the core of financial aid modeling is a simple cell chart. Think of a piece of paper covered with square boxes, defined by factors important to the college: SAT scores, GPAs, family income, and financial need.

• Colleges design these charts based on what they want more of (full-pay students or high GPAs, for instance) or less of (low test scores and high-need kids).
• For selective, “high-end schools, the only cells they make money on are their full pays,” Zucker told me. Some top-ranked schools might have 30% to 50% of their undergraduates paying full price.
• When there’s volatility in those cells, he said, the college’s discount rate “goes crazy.” He pointed to one selective school that saw its discount rate go from 34% to 58% between 2008 and 2009 as potential full-pay students at that school traded down on the academic front in order get a discount somewhere else.

But the 2008 recession isn’t a good model for this downturn. It was a housing recession. Even high-income families saw the value of their house drop. That was a huge problem for college since those families were planning to use home equity to finance a college education for their kids.

• Census data shows so far that the pandemic is having the biggest financial hit on families earning below $75,000. So the volatility this year might be in low- and middle-income cells, where colleges will likely need to spend more of their financial-aid dollars.

Bottom line: Colleges that can will lean into their pool of full-pay students this year, even if those students bring with them a slightly lower academic profile.

2. Merit Aid Without Test Scores

Zucker told me one of the busiest parts of his business in recent months has been working with college to build new “academic ratings” used for awarding merit aid. Those ratings in the past were typically based on a combination of grades and test scores, along with measures of how likely the student was to eventually enroll.

But with so many high school seniors lacking a test score because of the widespread cancelation of the SAT and ACT, Zucker is creating indices that exclude test scores by focusing more on the rigor of the high school curriculum and grades.

What’s happening: Zucker told me losing the test score isn’t as big of a deal as it might seem. Research shows that when colleges take test scores, high school grades, and curriculum into account, it's the test score that is usually the weakest predictor of student success in college.  

What I’m watching: Two enrollment deans at big public colleges told me recently that so far they’re receiving many fewer test scores from early applicants compared to past years. Both institutions were test optional before the pandemic, but they usually received scores from some 75% of applicants. Now they’re getting them from just 25%.

• So far, I’m hearing higher percentages from the most selective institutions—over 50%.
• Where that number ends up and how it impacts admissions this year will be critical to whether test-optional sticks.  

Bottom line: Selective colleges tend to favor students from high schools they know best anyway, and expect them to follow that strategy even more this year. Giving merit aid to students from familiar high schools — where enrollment rates have been reliable — could help colleges hedge their bets amid the uncertainty of the pandemic.

3. Actions Still Matter

While enrollment models built on the results of past years might be less accurate this time around, they still provide certain signals to college leaders.

“The models never had the level of precision that colleges thought they did,” Zucker said. 

This year, many colleges will be looking for as many signs of a student’s “demonstrated interest” in actually enrolling as they can find in this new normal. Did the student take a virtual tour? Did she show up for a Zoom meet-up with an admissions counselor? Did he open an email?

Why it matters: Before the pandemic, about one in five colleges said demonstrated interest was of “considerable importance” in their admissions decisions. That’s about the same weight they give to counselor recommendations and essays, and even more consideration than given to teacher recommendations, class rank, and extracurricular activities.

Behind the numbers: There’s evidence that the better the student, the more demonstrated interest might matter to his admissions decision—especially at colleges not at the top of the academic pecking order.

• One study of two years of contacts that a highly selective university made with more than 12,000 applicants found that students with the highest academic profile (grades and SAT scores) were really helped by showing demonstrated interest.
• Simply put, colleges which track demonstrated interest are reluctant to admit smart students who don’t engage in the admissions process for fear of being used as a safety school.

Bottom line: Even if you can’t get to campus, the more students show interest in a school this year the more it might help their chances for admission and merit aid.

SUPPLEMENTS

In my last newsletter, I shared a buzzy article about the role niche sports plays in admissions to selective colleges. Now The Atlantic says it "cannot attest to the veracity of the article.”

When we think of hybrid education we tend to think of the classroom, the mix between online and face-to-face learning. In the latest episode of the Future U podcast, Michael Horn and I talk with Marni Baker Stein, provost and chief academic officer at Western Governors University, about how the fast-growing online institution had to rethink all its services and offerings to students in the virtual world.

It’s what appears on a student’s diploma and resume, the place where undergraduates met their future spouse and lifelong friends, where they cheered the home team and, years later, donated what they could to help future generations.

Until next time, Cheers — Jeff

To get in touch, find me on Twitter, Facebook, Instagram, and LinkedIn.