AI Engineer Interview Questions: 15 to Ask (With Red Flags)

By Vadym Lobariev, founder of MindHunt and Claude Certified Architect — I run the technical screen for every AI engineer we present to a client.

Nearly every list of AI engineer interview questions online is written for candidates: here is the question, here is how to answer it. That is exactly the problem. The answers are public, rehearsed, and tell you nothing.

This guide is for the person asking the questions — a founder, CTO or engineering manager who needs to tell real production experience from a confident summary of other people's blog posts, possibly without being an AI specialist themselves.

Three Rules Before You Write a Single Question

1. Ask about what happened, not what is true. "Explain chunking strategies" tests memory. "How did you choose your chunk size, and what did you change after launch?" tests experience. Every question below is built this way.
2. Push one level deeper than is comfortable. The candidates worth hiring get more specific under pressure — numbers, tool names, the thing that surprised them. The ones to avoid get more general and more confident.
3. Do not test model-training theory for an application role. If the job is building product features on top of foundation models, backpropagation and gradient-descent trivia will filter out exactly the engineers you want and pass ML graduates who have never shipped. If you are unsure which role you are hiring, settle that first: AI Engineer vs. ML Engineer.

Start Here: A Five-Minute Baseline Screen

Before the deeper questions, I run five basic ones. I added them because of what I kept seeing in real screens: a surprising share of candidates — for AI engineer roles, and for software engineers who say they work with Claude Code, Cursor or similar tools every day — cannot name the models they use. They work with AI daily and have never looked under the hood. These questions take five minutes and save you an hour.

A. "Which models do you work with day to day? Name them — and tell me which you would use for what."

A strong answer names actual models and tiers, not just a brand. For Anthropic's Claude family, for example, that means knowing it runs from the small, fast Haiku through Sonnet and Opus up to the top-tier Fable — and having a view on the split: a small model for classification, extraction, routing and anything high-volume; a mid-tier model for most everyday coding and product work; the largest model for hard reasoning, planning and orchestration. They mention price and latency without being asked.

Red flag: "I use Claude" or "whatever Cursor picks." Someone who has never chosen a model has never had to care about cost, speed or quality — which means they have not run anything in production.

B. "What is a context window, and what happens as it fills up?"

A strong answer: it is the limit on everything the model can see at once — system prompt, conversation history, files, tool results — measured in tokens. Quality starts to degrade well before the hard limit, and cost and latency grow with every token. They can name ways to manage it: summarising or compacting history, retrieving only what is relevant instead of pasting everything in, handing a subtask to a sub-agent with its own fresh context.

Red flag: confusing the context window with the model's memory or its training data.

C. "What is prompt caching, and when does it help?"

A strong answer: the provider stores the already-processed beginning of a prompt, so repeated requests that share a long prefix — system prompt, tool definitions, a large document — are much cheaper and faster. It only works if that prefix stays identical, so the stable content goes first and the part that changes goes last; and the cache expires after minutes, not days.

Red flag: never heard of it. For anyone who has paid an LLM bill, it is one of the first things they learned.

D. "What is the difference between a workflow and an agent, and how do you choose?"

A strong answer: in a workflow, your code decides the steps and calls the model at fixed points. In an agent, the model decides what to do next — which tool to call, when to stop — in a loop. Workflows are predictable, cheaper and easier to test, so they are the right choice whenever the steps are known in advance. Agents earn their cost only when the path genuinely cannot be predicted. Good engineers start with the simplest thing that works and add autonomy reluctantly.

Red flag: calling everything an agent.

E. "What is the orchestrator–worker pattern, and when does it make sense?"

A strong answer: a lead model (the orchestrator, or manager) breaks a task into parts, hands each part to worker models — often smaller, cheaper ones, often running in parallel, each with its own context — and then combines and checks the results. It pays off for broad work that splits cleanly: research across many sources, changes across many files, reviews along several dimensions. It does not pay off for small tasks or tightly coupled, step-by-step work: you multiply token cost, and workers that cannot see each other's context make inconsistent decisions.

Red flag: never heard of it, or "more agents is always better."

How to use the baseline: treat it as a gate, not a score. A candidate for an AI engineering role who misses most of these should not go further, however good the CV looks. For a general software engineer who simply uses AI coding tools, question A and question B are the minimum — if they cannot answer those, "uses AI tools daily" means autocomplete.

Retrieval and RAG

1. "Tell me about a RAG system you built that didn't work well at first. What was wrong, and how did you find out?"

Why ask it: retrieval-augmented generation is the most common pattern in production LLM products and the most commonly faked on CVs.

A strong answer separates retrieval failures from generation failures — "the right document wasn't in the top five results" is a different problem from "the right document was there and the model ignored it." Expect specifics: chunks that split tables in half, embeddings that matched on topic but not on the actual question, stale indexes, a re-ranker or hybrid keyword search added after launch. Expect a description of how they knew: a labelled set of questions, logged retrievals, user complaints they traced back.

Red flag: a tidy architecture tour — "we used a vector database and LangChain" — with no failure in it. Or blaming the model for everything.

2. "When was RAG the wrong tool, and what did you use instead?"

A strong answer names real alternatives: putting the whole document in a long context window because the corpus was small; a plain SQL query or search index because the data was structured; fine-tuning because the problem was style or format, not knowledge. Bonus for mentioning cost and latency as part of the decision.

Red flag: RAG as the answer to everything, or no opinion at all.

Evaluation

3. "How do you know your LLM feature is working? Walk me through the evaluation setup on your last project."

Why ask it: this is the single most reliable separator between engineers who ship and engineers who demo. Anyone can call a model API. Far fewer can prove the result is good.

A strong answer describes a test set built from real user inputs, including the ugly ones; what was scored and how — exact checks where possible, an LLM-as-judge with a written rubric where not, human review for a sample; and how the judge itself was validated against human ratings. They will mention running the suite before every prompt or model change.

Red flag: "we tested it manually and it looked good." Or reaching for BLEU and ROUGE scores — metrics from the translation era that tell you almost nothing about whether an assistant's answer was correct.

4. "Tell me about a regression you caught before it reached users — or one you didn't."

A strong answer is a story: a prompt tweak that fixed one case and broke three others; a model version upgrade that changed output format; what the eval suite caught and what it missed; what they added afterwards.

Red flag: never having had one. Everyone who has run an LLM feature for more than a month has had one.

Agents, Tool Use and MCP

5. "Describe an agent you put into production. What could it do, what was it not allowed to do, and how did you enforce that?"

A strong answer talks about boundaries before capabilities: which tools were read-only, where a human had to approve an action, step and budget limits, what happened when a tool call failed or returned nonsense. Good candidates describe designing the tool descriptions and error messages for the model as carefully as an API for a human developer.

Red flag: an impressive demo narrative with no mention of failure handling, permissions or cost. "It just figures it out."

6. "Have you built or integrated an MCP server? What did you expose, and what did you decide not to?"

Why ask it: the Model Context Protocol has become a standard way to connect models to tools and data, yet almost no interview guide covers it. In 2026 it is a fair baseline question for anyone claiming agent experience.

A strong answer covers the design decisions: fewer, well-described tools rather than mirroring an entire API; authentication and what the server is allowed to touch; how tool results are kept small enough not to flood the context window; the risk of connecting third-party servers you do not control.

Red flag: confusing MCP with a framework or a model, or "we connected everything."

Calibration: not having used MCP is not disqualifying — it is new. Not having heard of it, for someone claiming current agent work, is a signal about how closely they follow the field.

7. "How do you make sure a model returns data your code can rely on?"

A strong answer: structured outputs or tool calling with a schema, validation on receipt (Pydantic, Zod, JSON Schema), a retry-with-error-feedback path, and a defined behaviour when validation still fails. Real job postings list this explicitly; most interview lists skip it.

Red flag: parsing free text with regular expressions and hoping.

Architecture Decisions

8. "Prompting, RAG or fine-tuning — walk me through a time you had to choose."

A strong answer treats it as a sequence, not a menu: start with prompting and good context; add retrieval when the model lacks knowledge; consider fine-tuning only when the problem is behaviour, format or cost at scale — and only with an eval set that can prove it helped. They should mention what fine-tuning costs in maintenance when the base model is replaced six months later.

Red flag: fine-tuning as the first move, or as a badge of seriousness.

9. "How do you choose which model to use for a feature?"

A strong answer runs candidates through the same eval set and compares quality, latency and cost per request; uses a small, fast model for simple steps and a stronger one where it matters; keeps the code model-agnostic enough to switch. They have an opinion about at least two providers based on use, not on benchmarks they read.

Red flag: "we just use the best one" — with no idea what it costs per thousand requests.

10. "Tell me about a time you talked someone out of using an LLM."

A strong answer names a case where a rule, a query or a classical model was cheaper, faster and more reliable — and how they made that case to a product manager who wanted "AI."

Red flag: every problem is an LLM problem. A good AI engineer's most valuable word is no.

Production: Cost, Security, Observability

11. "What did your feature cost per request, and what did you do to bring that down?"

A strong answer knows the number, or at least the order of magnitude. Levers they might mention: prompt caching, trimming context, routing easy requests to a smaller model, batching offline work, streaming to improve perceived latency, caching identical queries.

Red flag: never having looked. At small scale that is forgivable; for someone claiming a production system with real traffic, it is not.

12. "How would someone attack the system you built, and what did you do about it?"

A strong answer distinguishes direct prompt injection from indirect injection — malicious instructions hidden in a web page, document or email the model reads. Defences they might describe: treating all retrieved content as untrusted, least-privilege tools, human approval for irreversible actions, output filtering, not putting secrets in prompts. They should also be able to say what data left the company's boundary and under what agreement.

Red flag: "we told the model in the system prompt not to do that."

13. "It's Tuesday morning and answer quality has dropped. Nobody deployed anything. How do you find out what happened?"

Why ask it: observability is the least-covered topic in AI interviewing and one of the most revealing.

A strong answer starts with what they would already have in place: traces of every request with prompt, retrieved context, tool calls and output; quality sampled continuously by an automated judge; dashboards for latency, cost and refusal rate. Then hypotheses: a silent model update by the provider, a changed upstream data source, a shifted mix of user questions, an index that stopped refreshing.

Red flag: starting from "I'd look at the logs" with no idea what is in them.

Judgment and Communication

14. "Your product manager wants to launch a feature that is wrong in about 10% of edge cases. What do you do?"

A strong answer refuses the yes/no framing. What does "wrong" cost the user here — a slightly worse summary, or a wrong medical instruction? Can the failure be made visible, reversible or routed to a human? Can launch be narrowed to the cases where it works? They explain risk in business language, not model language.

Red flag: either "ship it" or "never" without asking what the errors are.

15. "What have you changed your mind about in the last six months?"

A strong answer is specific and recent: a technique they dropped, a tool they adopted, something a new model release made unnecessary. In a field that moves this fast, a senior engineer whose views have not changed in a year has stopped paying attention.

Red flag: nothing comes to mind.

Calibrating for Seniority

The same fifteen in-depth questions work at every level. What changes is the answer you should expect.

	Mid-level	Senior	Lead / Platform
Scope of stories	A feature they built inside someone else's design	A system they designed and ran end to end	A platform other teams built on
Failures	Can describe a bug and the fix	Can explain the root cause and what they changed in the process	Can describe how they stopped a class of failure across teams
Evaluation	Has used an eval suite	Has built one and validated the judge	Has made evals a release gate for other people's work
Cost and latency	Knows the levers	Knows the numbers	Has set budgets and built the tooling to enforce them
Saying no	Raises concerns	Proposes the alternative	Has changed a roadmap

One warning on years of experience: LLM application engineering as practised today is about three years old. A requirement of "5+ years of LLM experience" excludes nearly everyone qualified. Look for strong general engineering seniority plus one to three years of real LLM production work. What that profile costs in different markets is covered in our AI engineer salary guide.

A Simple Scorecard

Score each area from 1 to 4 straight after the interview, before discussing with colleagues. Write one sentence of evidence for each score — a quote, not an impression.

Area	Questions	1 — No signal	4 — Strong signal
Retrieval	1–2	Describes architecture only	Diagnoses retrieval vs. generation failures with examples
Evaluation	3–4	Manual spot checks	Test set from real data, validated judge, regression story
Agents and tools	5–7	Demo-level narrative	Boundaries, failure handling, schema validation
Decisions	8–10	One tool for every problem	Clear trade-offs, has said no
Production	11–13	Never looked at cost, security or traces	Knows numbers, threat model and how they would debug
Judgment	14–15	Binary answers, static views	Reasons about risk in business terms; views have evolved

For an application-focused AI engineer, I weight Evaluation and Production highest. A candidate who scores 4 on both and 2 on agents can learn agents in a month. The reverse is much harder.

If You Are Not an AI Expert Yourself

You can still run most of this interview. You do not need to know the right answer to hear the difference between a specific one and a vague one. Three practical moves:

• Ask "and then what happened?" three times. Real experience has a third layer. Rehearsed answers run out after the first.
• Ask for numbers. Requests per day, cost per request, size of the eval set, how long the incident lasted. People who were there remember.
• Bring in one external technical screen. One hour with someone who builds with this technology, before your own loop, removes most of the risk. This is the part of the process clients most often ask us to run.

The full hiring process around the interview — defining the role, sourcing, and the mistakes that sink most AI searches — is in How to Hire an AI Engineer.

How MindHunt Helps

Every AI engineer we present has been through a technical screen built on the questions above, run by a founder who is a Claude Certified Architect and works with this technology daily. You receive a written assessment of each candidate's real depth — where they were specific, where they were not — alongside the shortlist, usually within two to three weeks of kickoff.

→ Talk to us about your AI engineer search

Frequently Asked Questions

What questions should I ask when interviewing an AI engineer?

Ask about failures, measurement and trade-offs rather than definitions: a RAG system that did not work and how they found out; how they evaluate output quality; what their agent was not allowed to do; what a request costs; how the system could be attacked; and a time they argued against using an LLM. Specific, experience-based answers are the signal.

How is an AI engineer interview different from a standard software engineering interview?

Keep your normal bar for coding and system design — an AI engineer is a software engineer first. Add assessment of the judgment that is specific to LLM systems: evaluating non-deterministic output, managing cost and latency, handling hallucination and prompt injection, and deciding when not to use a model at all. Algorithm puzzles do not test any of that.

Should I ask machine learning theory questions?

Only if the role involves training or fine-tuning models. For an engineer building product features on top of foundation models, theory questions about backpropagation or loss functions filter out strong practitioners and pass candidates who have never shipped.

How do I interview an AI engineer if I am not technical in AI myself?

Ask for specifics and numbers, follow every answer with "and then what happened?", and listen for whether answers get more specific or more general under pressure. Add one external technical screen by someone who builds with LLMs before your own interviews.

What basic AI questions should any engineer be able to answer?

Five that take five minutes: which models they use and which suits which task; what a context window is and what happens as it fills; what prompt caching is; the difference between a workflow and an agent; and what the orchestrator–worker pattern is and when it makes sense. In real screens, a surprising number of candidates who use AI coding tools every day cannot name the models they work with.

What are the red flags in an AI engineer interview?

No failure stories; evaluation described as "we tested it manually"; every problem solved with an LLM; no idea what the system costs to run; security handled by instructions in the system prompt; and buzzword fluency that becomes vaguer, not sharper, when you ask a follow-up.

How many years of experience should an AI engineer have?

LLM application engineering in its current form is roughly three years old, so requirements like "5+ years of LLM experience" are unrealistic. A strong senior profile is five or more years of general software engineering plus one to three years of production LLM work.

Should I use a take-home assignment?

A short one can work — two to three hours, close to your real problem, followed by a conversation about the decisions made. Assume the candidate used AI coding tools; that is the job. What you are assessing is whether they evaluated their own result and can explain the trade-offs. Long unpaid assignments lose you the best candidates, who have other offers.