Ever since AI flooded into the mainstream in late 2022, students, parents, and the counselors who support them have been worried about the inevitable day when parts of the application for admission are being reviewed by machines. That day has arrived at Virginia Tech, but it is not the AI admission apocalypse anticipated among anxious applicants. In fact, the combination of AI and human readers might well lead to better and faster decisions. Following the institution’s announcement last week about the use of AI in essay review, I sat down with Juan Espinoza, the vice provost for enrollment management, to learn more.
The Role of Essays at Virginia Tech
Espinoza explains that “grades and rigor drive the review process as well as major selection,” adding, “I think a lot of people forget our review by major, and that actually one of the heaviest weights in the process is what major you’re applying for.” For example, he says, “someone looking at aerospace engineering is going to have a very different applicant pool than someone looking at music at Virginia Tech.” They are clear about the school’s admission criteria on their website and in their programming for potential applicants.
He emphasizes the importance of the essays in their review process and says that while academics are the “driving factor in the decision, for certain majors, essays bring in an additional lens.” Virginia Tech does not use the traditional personal statement–what is often known as the college essay–from the Common App, instead opting for four short essay responses, each 120 words or fewer. These questions, called the Ut Prosim Profile (Virginia Tech’s motto is Ut Prosim or “That I May Serve”), were designed intentionally to assess an applicant’s match for the institution’s mission and priorities. As application numbers have soared in recent years, Espinoza frequently hears skepticism that they are actually reading the essays. He says, “We really believe in these essays because they’re helping us identify the best students and allow us to make the best decisions.”
Why AI and Why Now?
Virginia Tech is committed to a holistic review process. Twenty years ago, they had around 17,000 applications, and this past year they received over 57,000 (more than a 10% increase from the previous year). Reviewing those applications comprehensively becomes a greater challenge every admission cycle. Espinoza says, “It’s just that with volume and the current setup with two human readers, it’s becoming harder and harder to manage, and it’s slowing down our notification process.” He adds, “One of the biggest pieces of feedback we’re getting is we’re coming too late in our offer letter, and so we’re really hoping to accelerate that by a few weeks.” In recent years, Virginia Tech has been notifying students about their status in late February or early March, and ideally, the admission office would like to send applicants a decision by late January.
Espinoza and his team have done the math on the estimated time it takes to review the nearly 500,000 essays they receive. He says, “If each essay takes about two minutes, that would require close to 16,000 hours of labor or about eight person-years.” If the job of one of those reviewers is assigned to a trained AI tool, then potentially they are saving around 8,000 hours, or the equivalent of four years of a person’s life. The concern that often follows is the job losses when AI is replacing humans, but at Virginia Tech, the second readers on applications are volunteers from the community. Espinoza says, “Each year, we recruit and train a group that typically averages between 200 to 300 volunteers from our faculty and staff.” Imagine if those individuals are freed up to volunteer in other ways within or beyond the campus community.
How It Works
As one might suspect, this is not something that Virginia Tech is jumping into haphazardly. Espinoza emphasizes, “the reason why we’re doing this is that we’ve been tracking and doing this research for three years with our faculty.” He adds, “The AI has been trained on the last few admission cycles to make sure that they are identifying key patterns in the answers to ensure that we’re getting the answer that we’re hoping to see.” He says, “The use of AI is to confirm the decision of a human reader, and it was in the most recent year that researchers were able to finally show that their models and their system are as effective as a real person.”
In the previous system, each essay was reviewed by two trained human readers who both assigned a score from 1 to 12 based on established mission-aligned criteria. If those scores varied by more than four points, a third reader was employed to address the discrepancy. With the new AI second review, the discrepancy threshold has been lowered to two points, further ensuring accuracy in the process. The use of a third reader is not changing at all in the revised system.
Espinoza says that the tool has “gone through different iterations,” adding, “what we found that resulted in the highest level of confidence was essentially to move away from a supervised LLM (Large Language Model) to one that’s using three different LLMs and utilizing a majority vote process to essentially vote on what they think the score should be.” He continues, “We found that this results in the lowest level of bias and the highest level of confidence as far as a lower disagreement score between AI and the human reader.” Over the past three years, their faculty developers have explored different models including “embedding models trained to predict historical essay scores, additional LLMs, alternative prompting approaches, alternative approaches to fine-tuning, and alternative ensembles.” Ultimately, they found that the majority vote process was proven to be the best.
It is important to note the institution’s commitment to privacy. Espinoza explains, “We’re able to remove the personally identifying information so all the reader, whether human or AI, sees is the answers and the question.” He adds, “We’re utilizing Virginia Tech resources and systems, our supercomputer Arc Center, so we underwent a security audit through our division of information technology to ensure and show that the data flow stays internal to Virginia Tech.” Applicants and their supporters can have confidence that their information is not being shared externally and is secure.
Transparency and Best Practices
To be sure, Virginia Tech is not the only school employing AI in its review process in some manner. Espinoza feels it is critical to be transparent about how they are doing so, not only for applicants but for other institutions to benefit from their learnings. He says, “Is there a possibility somebody else is doing this kind of thing behind the curtain? Yes, but who really gains from that, right? That stays within the institution.” He adds, “We are doing this transparently, which I strongly, strongly believe is imperative to build trust in the admissions process with the general public. That was a guiding principle in this process.” They feel it is critical to show the confidence they have in this process by being public about it.
Espinoza reassures, “We’ll monitor this. There will be research papers released on this. We are going to be transparent because we feel that this approach can be modeled and adopted across higher education to the betterment of stronger decisions, better quality decisions. And so we can share this with others. We absolutely plan on doing that.”
Feedback From the Field
Virginia Tech’s admission office has heard very little apprehension since the announcement last week. Espinoza credits that to the significant time spent on providing an FAQ resource. He says, “I think it’s those that didn’t actually read it that are coming with the concerns that we’re hearing about humans not making decisions.” Emily Pacheco is the founder of the AI special interest group for The National Association for College Admission Counseling (NACAC). She is impressed by Virginia Tech’s rollout of these changes and says, “their approach models how AI can augment, not replace, human evaluators. By keeping the human review independent and stepping in only when AI and human scores diverge, the university ensures fairness and consistency while maintaining human oversight at every step.”
Brian Leipheimer is the director of college counseling at Collegiate School in Richmond, Virginia, where a number of students apply to Virginia Tech every year. He says, “While I long for the days of lower application volume and greater personal—and human—attention to students’ stories, I recognize that AI is a powerful tool that is here to stay.” He adds, “Kudos to Virginia Tech for being so transparent at a time when ambiguity reigns supreme, and for working to reduce the amount of time seniors must wait for decisions to be released.”
Jenny Hechtkopf is an ACCESS Advisor at Landstown High School in Virginia Beach, Virginia. She says, “As a former admissions, counselor, and current Access Advisor for Access College Foundation, we assist students and parents with the college application process. I believe this is a strong and forward-thinking initiative. It provides students with an unbiased ‘reader,’ which promotes fairness in the decision-making process. I especially appreciate that significant discrepancies and evaluations will still trigger a human review and ensuring oversight and balance.” She adds, “this change has a potential to greatly streamline the admissions timeline. Not only will it reduce the burden on admissions committees, but it will also allow students to receive their decisions more quickly-something that can reduce stress and improve planning for many applicants. ”
Moving Forward
Espinoza says, “Is this without risk? No, of course not. I would be lying if I said that this was 100% foolproof. But what we’re going to do is be very transparent, step by step in this process, so others can learn and continue this important work. We’re very careful with the human safeguards. We’re very careful with the monitoring system. I really have a high level of confidence that we’re doing the right thing here and we’re doing it properly.” He adds, “And hopefully we’ll start reducing some of the turnaround times to get decisions back to students.” He also emphasizes, “We were so cautious not to be driven by one thing, efficiency. We wanted to do this responsibly. And by being transparent, you introduce a level of accountability to ensure that, because if you know you’re going to be revealing everything in public, you know everything’s out for inspection.” He concludes, “We are dealing with students, and it’s a very emotional process, and we’ve worked on a process that will essentially ensure that we’re making better decisions. Why would we not use that? That is in the best interest of students, and we get decisions out faster to them. This is a win-win.”
Virginia Tech is a model for the thoughtful implementation of AI in admission review from which other institutions can learn a great deal. In a high-stakes process that is increasingly challenging for both institutions and applicants, innovation with emerging technologies is critical. Collaboration between humans and machines can support a comprehensive yet streamlined approach to admission, and the coming year will be instructive.
Source: www.forbes.com