Building an AI FAANG Interview Coach: Run the Code, Don't Let the LLM Grade It

Ask ChatGPT to “review this solution and tell me if I’d pass the interview” and it will do what it always does: read your code, sound confident, and tell you it’s clean and correct — even when it isn’t. This is the trap at the heart of every naive AI interview bot: people ask the language model to judge correctness, and language models are spectacularly bad at it. They approve buggy code, wave through quadratic solutions as “efficient”, and hand out encouraging “great job!” feedback that teaches you nothing. A FAANG interview is the opposite of that: your code has to actually run, pass the edge cases, and scale — and a human interviewer scores you on specific, explicit signals. This article is how to build a coach that does the same, so the preparation is not just motivating but actually representative.

A note up front: this is an engineering article about correctness, code execution and LLM architecture. A real interview coach should prepare you honestly — not promise to “crack” any company or leak live interview questions (more on responsible framing at the end).

The architecture: the LLM runs the interview, a sandbox judges the code

The whole design rests on one decision: the LLM never rules on whether the code is correct. It does the two things it’s genuinely good at — conducting the interview (asking, hinting, probing, keeping time) and scoring the subjective signals — and hands the verdict on correctness and performance to deterministic code.

The flow is a standard tool-using agent:

1. The interviewer LLM runs the session — picks a question from the bank, asks it, drip-feeds hints, probes (“what’s the complexity?”, “how would you test this?”), and keeps to a time box.
2. A sandbox executor judges the code — it runs the candidate’s submission against hidden tests, measures time and memory, and returns a deterministic verdict (pass/fail, which cases, how fast). This is the part the LLM must not touch.
3. A rubric scorer grades the subjective signals — approach, communication, testing instinct, and behavioural STAR answers — against an explicit rubric, with evidence.
4. A structured scorecard combines them into a real debrief: per-signal scores, the evidence behind each, and a concrete next step.

Grounding the feedback on an actual execution verdict is what turns a flattering chatbot into a coach whose judgement you can trust. It’s the same pattern as any serious tool-using agent (see the AI Agent Development course and the RAG pipeline article): the model orchestrates and explains; tools provide the facts.

Why the LLM must not grade the code

It’s worth being concrete about why you can’t skip the sandbox and let the model decide if the solution is right. Here’s a real, measured example — a Two Sum solution that looks like a textbook answer.

Here’s the “clean two-pointer” that an LLM reviewer happily approves — and the hash-map answer that’s actually correct:

# Candidate's solution: a textbook two-pointer. Reads perfectly.
# An LLM says: "clean, correct, O(n log n) — looks good!"
def two_sum_subtle(nums, target):
    s = sorted(nums)
    i, j = 0, len(s) - 1
    while i < j:
        total = s[i] + s[j]
        if total == target:
            return [i, j]          # BUG: indices into the SORTED array
        elif total < target:
            i += 1
        else:
            j -= 1
    return []

# The correct answer:
def two_sum_hash(nums, target):
    seen = {}
    for idx, n in enumerate(nums):
        if target - n in seen:
            return [seen[target - n], idx]
        seen[n] = idx
    return []

The two-pointer version reads perfectly — it’s a pattern the model has seen ten thousand times, so it pattern-matches it to “correct.” But Two Sum wants indices into the original array, and sorting destroys them. Run both against six hidden tests and the truth comes out:

SANDBOX VERDICTS (candidate code run against 6 hidden tests):
  two-pointer (looks correct) : 3/6 passed  ->  WRONG ANSWER
       e.g. nums=[3, 2, 4], target=6: expected [1, 2], got [0, 2]
  hash map                    : 6/6 passed  ->  ACCEPTED

An LLM “reading” the code never catches this reliably — it’s a plausible answer, and plausible is exactly what language models reward. The sandbox doesn’t care how it reads; it runs nums=[3,2,4], target=6, gets [0,2] instead of [1,2], and fails it. Correctness is a fact you establish by execution, not a guess you make by reading.

The same goes for performance. A real interviewer cares whether your solution scales, and that too is measurable rather than debatable — here is the brute-force O(n²) scan against the hash O(n) solution on the worst case (no pair exists, so both do maximum work):

EMPIRICAL COMPLEXITY (worst case, no matching pair):
  N=2000 : brute O(n^2)  ~155 ms     hash O(n)  ~0.47 ms   ->   ~330x faster
  N=8000 : brute O(n^2)  ~1625 ms    hash O(n)  ~1.07 ms   -> ~1516x faster

(Wall-clock numbers vary with machine load — these are from one representative run; the three-orders-of-magnitude gap is the robust, reproducible point.) The ratio grows with N — the signature of a worse complexity class — and you can show the candidate that curve instead of asserting “this is slower.” So you give each component the job it’s good at: the sandbox rules on correctness and performance; the model runs the interview and explains.

The code-execution stack: a sandbox with hidden tests

The “standard library” here is a secure code-execution sandbox plus a hidden test suite per question. The executor is conceptually simple — compile/run the submission against each test, capture output, compare, time it — but the security is the hard part, because you’re running untrusted code. The judging core looks like this:

import time

def judge(solution_fn, hidden_tests):
    """Deterministic verdict: run candidate code vs hidden tests. No LLM involved."""
    passed = 0
    first_failure = None
    for nums, target, expected in hidden_tests:
        try:
            got = solution_fn(nums, target)
        except Exception as e:
            got = f"error: {e}"
        ok = (got == expected)
        passed += ok
        if not ok and first_failure is None:
            first_failure = (nums, target, expected, got)
    verdict = "ACCEPTED" if passed == len(hidden_tests) else "WRONG ANSWER"
    return {"passed": passed, "total": len(hidden_tests),
            "verdict": verdict, "first_failure": first_failure}

def measure(solution_fn, worst_case_input):
    """Empirical complexity: time the solution on a large worst-case input."""
    t0 = time.perf_counter()
    solution_fn(*worst_case_input)
    return (time.perf_counter() - t0) * 1000   # milliseconds

Two design rules make the verdict trustworthy. First, the tests are hidden — the candidate (and the model) see the prompt and maybe a sample, never the full suite, exactly like a real online assessment; that’s what stops “coding to the examples.” Second, the judge is pure and deterministic — same code, same tests, same verdict, every time — so feedback is consistent and you can diff two attempts.

Security is non-negotiable. Candidate code is untrusted and must run isolated: a locked-down container or microVM (gVisor, Firecracker, or a hosted sandbox API), no network, a read-only filesystem, and hard limits on CPU time, wall-clock, memory and output size (so an infinite loop or a fork bomb can’t take the box down). Never exec() submissions in your API process. The sandbox is the one component where “it probably won’t do anything bad” is not acceptable engineering.

For a production coach you’d layer more on top: multi-language execution (Python, Java, C++, Go), stdin/stdout harnesses as well as function-call harnesses, and per-test resource accounting. But the engine underneath is the same: code runs, the verdict is measured, the LLM reads the result.

Score the real FAANG signals — with a rubric, not a vibe

Correctness is only one axis. A FAANG interviewer grades a set of explicit signals, and the whole point of a coach is to score those, with evidence — not to say “great job!“. This is the part the LLM is good at, provided you give it a rubric and force structured output.

Two things make this work. The first is an explicit rubric with structured output — you force the model to return scored signals with evidence, not a paragraph of encouragement:

RUBRIC_PROMPT = """You are a senior FAANG interviewer writing a debrief.
Score each signal 1-4 (1=no hire, 4=strong hire). For every score, cite
EVIDENCE from what the candidate actually said or did. The CODE CORRECTNESS
is already decided by the sandbox verdict below — do NOT re-judge it; use it.

Sandbox verdict: {sandbox_verdict}
Transcript: {transcript}

Return JSON only:
{{"problem_solving": {{"score": int, "evidence": str}},
  "communication":  {{"score": int, "evidence": str}},
  "testing":        {{"score": int, "evidence": str}},
  "complexity":     {{"score": int, "evidence": str}},
  "verdict": str, "top_fix": str}}"""

Notice the rubric tells the model the sandbox already decided correctness and forbids it from re-judging — the model scores approach, communication, testing instinct and complexity reasoning, and reads the verdict for the rest. A sample of what it returns:

{"coding": {"score": 3, "evidence": "fails on unsorted input — indices wrong"},
 "complexity": {"score": 4, "evidence": "correctly identified O(n) hash approach"},
 "communication": {"score": 3, "evidence": "thought aloud, took the second hint"},
 "verdict": "lean hire — bar not yet cleared", "top_fix": "test before you submit"}

Structured, comparable across sessions, and actionable — you can track each signal over time instead of collecting a pile of “great job!“s.

The second thing is making the model behave like an interviewer, not an answer key. The most important rule is the hint ladder: nudge → hint → strong hint, but never the full solution — coaching builds the skill, handing over the answer destroys it. Add probing (“what’s the complexity?”, “how would you test this?”, “what if the input is empty?”), time-boxing each round, and the quiet pressure of a real interview. To make the rubric scoring itself authoritative, add retrieval (the RAG pattern): index each target company’s real competencies and leadership principles and have the model score against retrieved criteria, so a behavioural answer is graded on an actual principle, applied to what the candidate actually said — not the model’s vague memory of “what FAANG wants.”

What actually makes the results impressive

The “wow” in an interview coach comes from a few compounding things, all enabled by getting the foundation right:

• A verdict you can trust. The code ran; it passed 3 of 6 hidden tests; here’s the exact failing case. Users (and skeptics) can verify it. Trust is the whole product — a coach that flatters you into a real-interview rejection is worse than useless.
• Signal-level feedback over time. Not “you did well” but “your testing signal is a 2 — you’ve submitted without dry-running in 4 of your last 5 sessions.” That’s what actually moves someone past the bar.
• A real hint ladder. It nudges like a human interviewer instead of dumping the answer, so you build the muscle you’ll need when no one’s helping.
• Pressure and realism — time boxes, follow-ups, “can you do better than O(n²)?”, behavioural rounds scored on real competencies. It feels like the thing, because the mechanics are the thing.

None of that works if the verdict is a guess. Correctness first; everything impressive is built on it.

The pitfalls (where these coaches actually break)

• Letting the LLM judge correctness — the #1 bug. The moment the model decides whether code “looks right”, you’re back to flattering buggy solutions. The sandbox decides; the model reads the verdict. Non-negotiable.
• Weak sandboxing. Running untrusted candidate code without real isolation (no network, resource caps, microVM/container) is a security incident waiting to happen. Never exec() in your app process.
• Visible or weak tests. If the hidden tests leak, or miss edge cases (empty input, duplicates, negatives, the sorted-indices trap above), the verdict is hollow. Curate adversarial tests per question, and keep them hidden.
• The model giving away the answer. Without an enforced hint ladder, an eager LLM will solve the problem for the candidate. Constrain it: hints escalate, the full solution is off-limits during the round.
• Praise instead of a rubric. Free-form “feedback” drifts into encouragement. Force structured, per-signal scores with evidence, or it isn’t coaching.
• Non-determinism in scoring. Pin the model, use low temperature for the rubric pass, and anchor every subjective score to evidence so two runs of the same transcript don’t swing wildly.

Build it responsibly

A word on framing. Market it as preparation, not a cheat code: an honest coach makes you better at interviewing — it doesn’t “guarantee” an offer, and it must never traffic in leaked or live interview questions from a specific company (that’s both unethical and a legal problem). Keep the question bank to your own problems or clearly-licensed public ones. Don’t build the inverse product either — a tool to help someone cheat during a real remote interview is exactly what you’re not making. Be honest about limits: a bot can rehearse you, but it isn’t a hiring decision. The engineering here (real execution, grounded, evidence-based feedback) is sound and genuinely useful; use it to help people actually improve.

Takeaways

• Never let the LLM decide if the code is correct — it approves subtly buggy solutions (the two-pointer Two Sum that fails 3/6 hidden tests reads as “clean and correct” to a model). Use a deterministic sandbox against hidden tests.
• Measure complexity, don’t assert it — time the solution on large worst-case inputs; brute force was ~1500× slower than the hash solution at N=8000 in a representative run, and the growing ratio is the real signal.
• Split the work: a sandbox judges correctness and performance; the LLM runs the interview (hint ladder, probing, time-boxing) and scores the subjective signals against an explicit rubric with structured JSON output and evidence. Add RAG over real competencies/leadership principles.
• Sandbox security is non-negotiable — untrusted code runs isolated (no network, resource caps, microVM/container), never exec() in your process.
• Build it responsibly: honest “preparation not guarantee” framing, your own/licensed questions, and no help cheating a live interview.

If you want the broader patterns this builds on, see the AI Agent Development course (tool-using agents), the RAG pipeline guide (grounding generation), Building Voicebots if you want to make the coach talk a mock interview, and the Cracking the System Design Interview course for the design rounds a complete coach would cover.