What GPT-4 Actually Knows About Hiring (And What It Gets Wrong)

The first thing to understand about large language models and hiring is that they don't know anything about your candidates.

This sounds obvious. It's also frequently forgotten when people get excited about deploying GPT-4 or similar models in recruiting workflows. The model has no access to your company's historical performance data, no understanding of your team dynamics, and no way to assess a candidate it hasn't been given explicit information about. What it has is a very large and sophisticated sense of what things sound like.

That capability is genuinely useful. It's also genuinely limited.

What LLMs Do Well in Hiring Contexts

Writing and refining job descriptions. This is probably the highest-value, lowest-risk use of LLMs in recruiting. A good model can take a rough job brief and produce a clear, well-structured job description — and can flag language that's likely to discourage certain candidate groups. It won't know whether the requirements are realistic or appropriate for the market, but it can make a mediocre draft substantially better.

Generating structured interview questions. Given a job description and some context about what matters in the role, a capable LLM produces solid behavioral and situational questions. The quality here depends heavily on the specificity of the prompt. "Write interview questions for a software engineer" produces generic results. "Write four behavioral questions to assess how a senior backend engineer handles ambiguous technical requirements in a fast-moving startup environment" produces something more useful.

Summarizing and structuring feedback. Turning sprawling interview notes into structured, comparable summaries is tedious work that LLMs do well. The model isn't adding judgment; it's organizing what humans have already said.

Candidate communication drafts. Rejection emails, offer letters, follow-up messages, scheduling context — LLMs can produce solid drafts for all of these faster than most humans, and consistently.

Where LLMs Fail

Predicting job performance from resumes. A language model reading a resume and predicting whether someone will be a strong hire is not doing analysis — it's pattern-matching against its training data's representation of what "good" resumes look like. That representation includes every bias in the data it was trained on.

An LLM scanning resumes and scoring them is almost certainly rewarding name recognition (brands, universities, previous employers), fluency in a particular writing style, and a resume format that looks like the resumes of people who've succeeded before. None of that is performance prediction. All of it is reproducing historical patterns.

Assessing candidate fit for a specific team or culture. LLMs have no model of your team. They cannot assess whether a candidate's working style will complement your current lead, whether their communication approach will work in your particular culture, or whether their ambitions align with what the role actually offers.

An LLM can tell you what a good answer sounds like. It cannot tell you whether this specific person, in this specific role, at this specific moment in your company, will succeed.

Avoiding hallucination in evaluation contexts. LLMs confabulate. They produce fluent, confident-sounding text that is factually wrong. In an email draft, this is a nuisance. In a candidate evaluation that informs a hiring decision, it is a more serious problem. Any LLM-assisted evaluation process needs human review at consequential decision points — not as a formality, but as a real check.

What the Hype Gets Wrong

The dominant pitch for LLMs in hiring is efficiency. And it's largely true — these models can compress work that took hours into minutes. But efficiency is the wrong frame for evaluating how they affect hiring quality.

The question isn't "how much faster is this process?" It's "are we making better decisions with better information?" A faster, cheaper version of a flawed process is still a flawed process.

LLMs are particularly susceptible to something the ML community calls "specification gaming" — they optimize for what they're told to optimize for, not for what you actually want. Tell a model to score candidates on "communication clarity" and it will score them on whatever communication clarity looks like in its training data. Whether that correlates with actual job performance is a separate empirical question that most deployments don't answer.

A Practical Framework

Use LLMs for:

• Content drafting (JDs, questions, emails, summaries)
• Process acceleration (screening volume, structuring feedback)
• Consistency (applying a defined rubric uniformly)

Keep humans in the loop for:

• Consequential decisions (who advances, who gets an offer)
• Novel situations (candidates who don't fit the pattern)
• Bias auditing (checking whether the AI's outputs are equitable)

The companies using LLMs well in hiring treat them as capable assistants with specific, bounded tasks — not as judgment-replacement systems. That framing isn't pessimistic about the technology. It's accurate about what it currently does well.

---