How to hire an AI developer without buying a demo

An AI prototype can look convincing while hiding the work required for trustworthy production behavior. The hiring brief should therefore describe a decision system, not merely a model or chat interface.

Define the job before the role

Write the user, decision, available evidence, acceptable failure, escalation path, and measurable outcome. Separate deterministic product logic from model-generated behavior. If a conventional rules or search system can solve the problem more reliably, candidates should be free to say so.

Ask whether the engagement needs:

product discovery and workflow redesign;
data preparation, retrieval, or labeling;
prompt and model experimentation;
evaluation design and red-team testing;
application engineering and integrations;
privacy, security, and compliance controls;
monitoring, incident response, and cost management.

One person may cover several areas, but the proposal should not pretend they are the same skill.

Evidence to request

Ask candidates to explain a system they moved beyond a demonstration. Good evidence shows how they created test cases, measured quality, handled unsafe or unsupported output, controlled data access, observed cost and latency, and changed the system when production evidence disagreed with the prototype.

Avoid evaluating candidates with trivia about one fast-moving framework. Give them a small version of the real decision and ask for an approach, assumptions, failure modes, and evaluation plan. The reasoning is more durable than a memorized API.

Contract the evaluation loop

Acceptance should cover representative scenarios, known failure cases, security boundaries, latency and cost budgets, human review where required, observability, documentation, and a rollback path. Do not define “works” as a single happy-path demonstration.

Start with a short paid discovery when data quality, user behavior, or feasibility is uncertain. End discovery with a recommendation, evidence, architecture options, delivery estimate, and explicit reasons not to proceed when the case is weak.

Match the hire to the unresolved risk

“AI developer” can describe very different work. Choose the profile after identifying what can still invalidate the project.

Primary uncertainty	Capability to prioritize	Evidence to request
The workflow may not create user value	Product discovery and process design	A decision memo that changed or stopped a proposed solution
Available data may be insufficient	Data engineering and evaluation design	Data-quality analysis, representative cases, and limitations
Model behavior may be unreliable	Applied ML, retrieval, experimentation, and red teaming	Versioned evaluation results and failure analysis
The prototype may not operate safely	Production software, security, and observability	A deployed system, monitoring approach, and incident learning
Cost or latency may be unacceptable	Architecture and performance engineering	A measured tradeoff between quality, speed, and complete operating cost
Adoption may fail	Workflow integration and change design	Evidence of users completing the intended task, not only opening a feature

One candidate may cover several rows, but do not award missing capability by title. If the work spans product, data, model evaluation, application engineering, security, and operations, decide whether you need a senior generalist with specialist review or a coordinated team.

Write an AI delivery brief

Start with the decision or task the system will support. Identify the user, trigger, available context, output, downstream action, and person accountable when the system is uncertain or wrong. Include examples of acceptable, unacceptable, and ambiguous behavior.

The brief should also state:

which data may be used and where it comes from;
whether personal, confidential, regulated, or licensed material is involved;
the minimum deterministic rules that must always hold;
when a human must review, approve, or take over;
expected volume, latency, availability, and operating budget boundaries;
supported languages, accessibility needs, and high-impact user groups;
required integrations and buyer-controlled systems;
evidence required to launch and conditions that should stop the project.

Avoid choosing a model, framework, or architecture before candidates can inspect the problem. A strong candidate should be able to challenge the premise and propose a simpler route when it is safer or more economical.

Evaluate a candidate with a realistic case

Give every candidate the same anonymized workflow, constraints, and sample cases. Ask for a short written approach covering assumptions, baseline, data flow, evaluation, safety boundaries, observability, cost drivers, and the first experiment. Do not require unpaid production work.

Look for disciplined uncertainty. Strong answers distinguish what can be tested immediately from what depends on data or user evidence. They propose a non-AI baseline, identify failure severity, and explain how the system degrades safely. Weak answers begin with a favorite framework, promise accuracy without an evaluation set, or treat human review as an undefined fallback.

Useful interview questions include:

What would make you recommend not using generative AI here?
How would you create representative evaluation cases without leaking production data?
Which failures must be prevented, detected, or escalated?
How would you separate model quality from retrieval, prompt, interface, and workflow problems?
What changes when the model or upstream data changes?
How would you investigate a harmful or unsupported output?
Which metrics could improve while the user outcome gets worse?
What must the buyer own to change suppliers later?

Require an evaluation system

An evaluation set should cover normal cases, important edge cases, adversarial inputs appropriate to the system, and known unacceptable outcomes. Record where each case came from, who approved it, and why it matters. Keep protected production data out of development unless access is justified and controlled.

Define multiple dimensions rather than one vanity score: task completion, factual support, refusal or escalation behavior, security boundary adherence, consistency, latency, usage cost, and reviewer burden may all matter. Some dimensions will require human judgment. Document rubrics and disagreement instead of pretending every result is objective.

Run evaluations against versioned application configurations—not only model names. Retrieval, prompts, tools, permissions, system logic, and data all affect behavior. A production change should be traceable to the tests and approval that supported it.

Inspect security and control boundaries

Map the complete data and action flow. Identify what leaves the buyer environment, what a provider may retain, which prompts or files can contain secrets, which tools the system can call, and what an attacker or untrusted document could influence. OWASP’s LLM application risks are a useful question source, but controls must fit the actual architecture.

Use least-privilege credentials, allowlists for consequential actions, confirmation before irreversible steps, output validation, audit records, and rate or spending limits. Treat retrieved content and tool output as untrusted input. The person building the system should be able to explain the boundary without hiding behind a platform certification.

Run discovery with explicit exits

A paid discovery should answer whether to proceed and under which architecture. Useful outputs include a workflow map, data assessment, baseline, representative evaluation cases, risk register, options, prototype evidence, operating-cost model, delivery plan, and stop recommendation.

Agree on exit criteria before discovery begins. Continue only if evidence supports the user value and risk can be managed. Revise when the workflow or data needs repair. Stop when the proposed system would be unreliable, uneconomical, or unnecessary. Paying for a credible “do not build” conclusion is cheaper than converting a demonstration into an unsupported product.

Contract for production ownership

Name the owners of evaluation data, prompts, application code, configurations, logs, feedback, model accounts, vector or search indexes, cloud resources, and third-party licenses. Require documentation, deployment procedures, monitoring, incident response, cost controls, and an exit path.

Acceptance should include repeatable evaluation results, security evidence proportional to risk, operational dashboards or logs, performance within agreed boundaries, rollback, and a buyer-run handover. Avoid guarantees about model behavior that no party can control; contract for the process, evidence, response, and remediation that can be controlled.

Decide whether you are hiring capability or leasing a tool chain

Some candidates can design, evaluate, and operate a system across providers. Others are highly effective inside a particular hosted platform or proprietary stack. Either can fit, but the buyer should understand what remains when the engagement ends.

Ask the candidate to map every account, model, API, repository, vector store, data-labeling service, evaluation platform, analytics tool, deployment target, and administrative permission needed. Record who owns the account, who can export the assets, which data it receives, what setting controls retention or secondary use, and how the service can be replaced.

Test whether the proposed architecture is intentionally provider-specific or merely convenient for the developer. Provider-specific work can be the right choice when it produces a meaningful advantage, but its change and exit cost belongs in the decision. Do not accept “portable” without identifying what must be rewritten, re-embedded, reevaluated, and reapproved.

Inspect dataset and evaluation design

Ask where representative cases will come from and how the candidate will avoid sampling only easy, frequent, or already solved examples. Require coverage of ordinary behavior, consequential edge cases, ambiguous inputs, adversarial conditions appropriate to the product, permission boundaries, and known unacceptable outcomes.

For every evaluation set, record provenance, usage rights, sensitive fields, transformations, version, reviewer, and limitations. Separate development cases from a protected set used to challenge generalization. If production feedback becomes evaluation material, define who may select it, how it is minimized, and whether user or customer commitments permit that use.

Have the candidate demonstrate rubric design and reviewer calibration. Where judgment is subjective, multiple reviewers may disagree for valid reasons. The system should preserve the disagreement, escalation, and final acceptance rule instead of collapsing it into an unexplained score.

Require a model-change release packet

A change to the model, prompt, retrieval source, tool set, routing logic, guardrail, or data transformation can change behavior without altering the visible interface. Require a release packet that names the versioned configuration, reason for change, representative evaluation result, new failures, latency and cost effect, data and security impact, approval, monitoring plan, and rollback.

Ask candidates how they would respond when a provider updates an aliased model or deprecates a version. A mature answer includes detection, a controlled test environment, regression evidence, traffic migration, and a safe fallback. “We will keep the model current” is not a release process.

This packet should be reproducible by the buyer. The developer can prepare and recommend the release, but the buyer’s accountable owners need enough evidence to approve or reject it.

Test incident response and degraded operation

Use a tabletop scenario drawn from the architecture: the model service is unavailable, retrieval returns unauthorized content, a prompt injection reaches a tool, output quality drops, cost spikes, or a human-review queue is overwhelmed. Ask the candidate to identify detection, containment, decision authority, evidence, communication, restoration, and follow-up.

Require a degraded mode proportionate to the workflow. Options may include disabling an action, returning to search, requiring manual completion, using a previously approved configuration, reducing features, or stopping safely. The correct response is not always a backup model; an untested substitute can produce a different failure.

Inspect whether the application records enough version, input, retrieval, tool, output, decision, and reviewer context to investigate an event without retaining more sensitive material than necessary. Observability and minimization must be designed together.

Evaluate the delivery team around the developer

Even a strong individual needs product decisions, domain review, data access, security input, application engineering, and operational ownership. Build a responsibility map showing what the candidate supplies, what the buyer supplies, and what another specialist must review. Assign names and availability before accepting a plan.

For an agency, interview the delivery lead and named practitioners rather than only the AI specialist in sales. Verify allocation, substitution, review, escalation, and how different disciplines produce one release packet. For an individual, confirm that the buyer has enough capacity to supply missing functions without creating a hidden coordination bottleneck.

Use a paid pilot to test the real arrangement. Include one representative case, one permission boundary, an evaluation update, a failure or degraded-mode exercise, deployment into a buyer-governed environment, and a handover. Score the accepted evidence and buyer review burden, not the demo’s visual polish.

Frequently asked questions

Should we hire one AI developer or an agency?

Hire one person when a bounded risk dominates and your team can supply product, review, security, and operations. Use a coordinated team when several disciplines must deliver one outcome and you need one party accountable for integration.

What portfolio evidence matters most?

Ask for a system that reached real operation and how it was evaluated, monitored, corrected, and handed over. A polished interface or model demo is weaker evidence than a candid failure analysis and versioned test process.

Do candidates need experience with our exact model provider?

Not always. Durable skills include problem definition, data judgment, evaluation, software engineering, security, and operations. Exact platform experience matters when the integration is specialized, but framework trivia should not replace production reasoning.

How long should discovery last?

Scope it to the questions that can invalidate the project, not an arbitrary calendar promise. The proposal should identify the decisions, evidence, people, and stop criteria required to finish discovery.

Who should approve launch?

The buyer’s accountable product and risk owners should approve against visible evidence. The developer or provider can recommend launch, but should not be the only party defining success or accepting its own work.

Should the developer be allowed to choose every AI service?

They should recommend services from measured needs, but the buyer should approve material providers, data use, accounts, cost, and dependency. Keep an approved-service register and a controlled change path rather than granting unrestricted tool adoption.