How to hire a machine learning engineer for production

“Machine learning engineer” is not a standardized promise. One candidate may be strongest at training models, another at data pipelines, and another at turning an existing model into a reliable service. An agency may use the same title for several people. The buying decision becomes clearer when the role is expressed as an accountable work package rather than a list of libraries.

This guide is procurement and delivery guidance, not legal, privacy, employment, security, or regulatory advice. A U.S. buyer using an outside-U.S. team should have qualified advisers review the actual data, people, jurisdictions, sector rules, and contract.

Decide whether machine learning is the actual work

Start with the decision or product outcome. A request such as “build a prediction model” is incomplete because it does not say who acts on the output, which error is costly, whether a simpler rule works, or how the answer will be observed after launch.

Write an outcome statement with five parts:

1. the user or operational owner;
2. the decision the system will support or automate;
3. the evidence available at decision time;
4. the cost of a false positive, false negative, delay, or abstention; and
5. the observable change that would justify continuing.

A simple baseline is a procurement control. It can be a current manual process, a deterministic rule, a lookup, or an existing model. The baseline gives every candidate the same reference and makes it harder to sell model complexity that does not improve the outcome. It also gives the buyer a fallback if the trained system is too expensive, too unstable, or unsupported by the available data.

Ask what must be learned before an ML build is justified. Does the organization have lawful and usable examples? Is the target label meaningful? Can the outcome be measured after a prediction? Will the surrounding workflow change? A sophisticated algorithm cannot repair an undefined target, inaccessible data, inconsistent labels, or an organization that cannot act on the result.

NIST’s AI Risk Management Framework is useful here as an organizing reference, not as a certification or universal checklist. Its Govern, Map, Measure, and Manage functions reinforce that model quality sits inside a broader lifecycle of ownership, context, measurement, and response. NIST also states that AI RMF 1.0 is being revised, which is why this page carries a short review interval and does not present the framework as frozen.

Choose the role by the missing operating capability

Do not use titles as interchangeable search terms. Use the missing deliverable and retained buyer responsibility.

An individual can cover several columns, especially for a bounded system. The question is whether the evidence supports the combined responsibility. Do not award missing capability because a résumé says “full stack AI.” Name what the buyer will retain: product judgment, domain labeling, privacy decisions, security approval, infrastructure accounts, deployment authority, and incident command should not disappear into a vendor title.

For a durable strategic capability, compare an outside project or staff-augmentation arrangement with direct employment. Staff augmentation gives the buyer more day-to-day control but also retains more management and integration work. A project team can own a bounded result, but only when acceptance, access, and dependencies are explicit. Use the staff augmentation versus project outsourcing guide before assuming the same statement of work fits both.

Define the data boundary before sharing samples

The fastest way to create an unsafe ML engagement is to send a broad production extract so a candidate can “see what is possible.” Start with a data inventory and a progressively disclosed evaluation path.

For every proposed field, record:

• source system and accountable owner;
• collection purpose and permitted use;
• whether it contains personal, confidential, licensed, regulated, or export-sensitive information;
• label origin and known quality limitations;
• permitted countries, people, environments, subprocessors, and retention period;
• transformation, pseudonymization, aggregation, or synthetic substitute available for early work;
• deletion and return evidence required at the end; and
• what new approval is needed if the purpose, field set, model, or delivery location changes.

Begin candidate evaluation with a schema, data dictionary, small sanitized example, or synthetic fixture where practical. The goal is to see whether the candidate asks the right questions and designs a defensible experiment, not to obtain unpaid training on live records.

Training, validation, testing, and post-launch evaluation datasets need distinct purposes. Define how duplicate entities, future information, and correlated records will be handled so the candidate cannot accidentally evaluate on information the production system would not have. If the work is temporal, the evaluation split should reflect the decision date. If the same customer, device, document, or event family can appear in multiple splits, require a leakage review.

International delivery adds a location and access question, not a universal prohibition or permission. Record where data will be stored, viewed, transformed, logged, backed up, and supported; the employing or subcontracting chain; and the legal mechanism or sector approval relied upon. Link the final work order to the applicable country guide and state-specific buyer context under the U.S. hub. Do not infer compliance from a provider’s headquarters or cloud-region dropdown.

Write an evaluation contract before comparing models

An evaluation contract is a shared record of what will be measured, on which data, under which conditions, and what happens when results are ambiguous. It prevents candidates from choosing the most flattering metric after seeing the outcome.

Include:

• the baseline and why it is credible;
• primary decision metric and supporting diagnostics;
• acceptance threshold or decision rule;
• uncertainty, sample limitations, and confidence reporting appropriate to the use case;
• important cohorts, edge cases, and failure scenarios;
• latency, throughput, availability, and cost boundaries where relevant;
• human review, abstention, escalation, and override behavior;
• prohibited uses and conditions that require reapproval;
• reproducible evaluation code and immutable evaluation data reference; and
• the person authorized to accept a tradeoff or stop deployment.

Accuracy is rarely a complete acceptance criterion. A candidate should explain the consequence of each error, how class imbalance affects interpretation, whether the observed metric connects to the business decision, and what evidence can be obtained after launch. For ranking or recommendation systems, require offline and operational measures rather than assuming one historical score predicts user benefit. For generative or hybrid systems, use scenario-based evaluation and verify unsupported output, unsafe action, data exposure, cost, and escalation separately.

Keep the holdout boundary meaningful. The team tuning the system needs feedback, but repeated optimization against the final acceptance set turns that set into another development input. Name who controls the final test, when it can be run, and whether a replacement set will be needed.

The acceptance decision can be “do not deploy.” A valuable pilot may show that the baseline is already adequate, the labels are unreliable, the operational cost is unjustified, or the risk cannot be controlled. Contract for the evidence and reusable artifacts, not only for a promise that a model will win.

Buy the pipeline and operating loop, not only the artifact

Production machine learning is a changing system. Google’s current ML pipeline guidance describes recurring data, training, validation, deployment, and serving activity, while its MLOps architecture separates preparation, training, evaluation, validation, serving, and monitoring. Those sources are vendor guidance, not a required architecture, but they expose useful handover questions.

Ask the candidate to draw four connected paths:

1. Data path: acquisition, validation, transformation, feature production, labeling, lineage, and deletion.
2. Training path: code, environment, dependencies, parameters, data version, compute, artifacts, and reproducibility.
3. Serving path: interface, feature parity, deployment, fallback, capacity, authentication, and rollback.
4. Learning path: outcome collection, monitoring, investigation, approval, retraining or replacement, and retirement.

Require code and configuration to live in buyer-controlled repositories and accounts unless an approved managed-service model says otherwise. Record how environments are recreated, how dependencies are pinned and scanned, who can run expensive jobs, how artifacts are named, and which logs may contain sensitive inputs or outputs. A model file without the code, data references, environment, and evaluation record is not a transferable delivery.

Training-serving skew deserves a direct demonstration. Google’s engineering guidance uses the term for discrepancies between how data is handled during training and serving. Ask the candidate to show how shared transformations, schema checks, fixtures, and monitoring detect it. Do not accept “we use the same feature names” as proof that the values, timing, defaults, and missing-data behavior are equivalent.

Monitoring needs a response owner. Dashboard creation is not acceptance. Define which technical and decision-quality signals are observable, how delayed labels affect detection, which alert opens an incident or review, who investigates, and what rollback or safe mode exists. A shift in input distribution is not automatically harmful; a stable input distribution is not proof that decisions remain good. Monitoring should lead to a documented decision, not automatic retraining by default.

Evaluate candidates with one representative paid pilot

Use the same bounded brief for finalists. A good pilot is small enough to complete without production access but broad enough to exercise the work that usually breaks at handoff.

One example pilot is a versioned baseline-to-serving slice:

1. Provide a sanitized schema, a small approved dataset or generator, the current baseline, and the evaluation contract.
2. Ask the candidate to validate the data, identify leakage and label risks, and propose the smallest defensible experiment.
3. Require a reproducible training or model-selection run with recorded inputs and outputs.
4. Package the selected artifact behind a small batch or service interface.
5. Add data, code, and interface tests plus one deliberately failing case.
6. Produce evaluation results against the agreed baseline, including limitations and unresolved questions.
7. Demonstrate rollback or fallback and describe the production monitoring loop.
8. Hand the work to a buyer engineer using only the repository, runbook, and recorded environment.

Score what was observed. Use the provider scorecard to compare fit, evidence, transparency, protections, start readiness, international capability, and support. Download one local scorecard per finalist and keep the evidence notes with the procurement record. A score does not override a failed data, security, ownership, or acceptance gate.

Interview the person who will perform the work. Ask them to challenge the premise, explain what cannot yet be known, and identify which failure would appear only after deployment. Give them a flawed pipeline or evaluation description and ask how they would investigate it. This reveals engineering judgment more effectively than trivia about a fast-moving library.

Check references against the proposed responsibility. A candidate who improved a research model may not have owned serving, on-call response, or retraining. Ask what they personally changed, which evidence existed before and after, what failed, who operated the result, and what happened when data or requirements moved.

Contract for ownership, security, and a usable exit

The work order should turn the four pipeline paths into deliverables and responsibilities. At minimum, name:

• buyer and provider owners for product, data, ML, platform, security, privacy, and incident decisions;
• approved people, countries, systems, accounts, repositories, datasets, models, and subprocessors;
• access approval, least privilege, logging, credential rotation, and revocation;
• data-use purpose, retention, return, deletion, and evidence;
• source-code, configuration, documentation, model, feature, evaluation, and derivative-work ownership or license terms;
• rights and restrictions for third-party data, open-source components, pretrained models, hosted services, and generated assets;
• evaluation, acceptance, change-control, rollback, and production-promotion authority;
• security reporting, vulnerability handling, incident cooperation, and evidence preservation;
• knowledge transfer, staffing continuity, replacement, and subcontractor controls; and
• termination assistance, export formats, environment reconstruction, and final access removal.

Do not assume “work made for hire” or a generic IP clause resolves every jurisdiction, employment chain, model license, dataset right, and third-party component. Use the source-code and IP guide to prepare the inventory, then obtain qualified advice for the actual parties and countries.

CISA’s Software Acquisition Guide is aimed especially at government and higher-assurance acquisition, but its buyer-supplier dialogue is broadly useful: development, supply chain, deployment, and vulnerability management need evidence across the ownership lifecycle. Apply the level of diligence that matches the access and impact; do not demand a large compliance packet from a low-risk prototype while leaving a production data path undefined.

Compare complete cost and retained work

The quoted engineer rate is only one input. Compare the complete operating model:

• discovery and data preparation;
• labeling, adjudication, and subject-matter review;
• data storage, movement, and quality operations;
• training, evaluation, and experiment compute;
• serving, third-party API, observability, and support costs;
• platform, security, privacy, and legal review;
• buyer product and engineering management;
• time-zone overlap, written handoff, and rework;
• environment setup, documentation, and knowledge transfer;
• incident, retraining, migration, and exit work; and
• currency, tax, payment, renewal, and change assumptions.

Ask every candidate to quote the same work breakdown and mark buyer-supplied dependencies. Separate one-time setup, usage-variable spend, recurring operations, and optional work. Put price validity, currency, taxes, usage units, included environments, support hours, and adjustment mechanics next to the number.

Use the AI development cost guide and cost calculator to expose assumptions. Do not turn a country average into a provider quote or a pilot estimate into a permanent benchmark. Nearshore overlap can reduce coordination cost for one workflow; an offshore relay can improve cycle time for another. Compare the named team and operating plan, not geography as a quality score.

Red flags that should stop progression

Pause or reject the engagement when a candidate:

• recommends a model before understanding the decision, baseline, or data;
• requests broad live data for an unpaid or loosely defined trial;
• reports a metric without the split, sample, baseline, and error consequence;
• cannot reproduce the result outside one person’s notebook;
• treats training and serving transformations as unrelated handoffs;
• promises that monitoring, retraining, or “human in the loop” will solve risk without an owner and decision rule;
• cannot name the people, employment or subcontracting chain, and work locations;
• expects to own the only repository, cloud account, model artifact, or evaluation record;
• claims universal IP ownership without reviewing datasets, model terms, components, and jurisdictions;
• presents a certification, cloud logo, or famous client as proof of fit for the proposed system;
• prices only model development while excluding data, integration, operations, and exit; or
• refuses a representative paid pilot, evidence-based acceptance, or a usable handover.

Frequently asked questions

What is the difference between a machine learning engineer and an AI developer?

Use the work package, not the title. A machine learning engineer usually needs strong evidence across data preparation, reproducible training, evaluation, serving, and ongoing model operations. An AI developer may focus more on integrating hosted or foundation-model behavior into an application, evaluation, tools, retrieval, and product controls. Real candidates can span both; verify the actual missing capability.

Should we hire one engineer or an agency?

Hire an individual when the work is bounded, your team owns product and platform decisions, and adjacent data, security, and operations capability already exists. Use a small accountable team when delivery requires several roles at the same time or when you want one party responsible for a defined outcome. In either case, interview the named people and preserve buyer ownership.

Can an overseas engineer use our production data?

Location alone does not answer the question. Classify the data and purpose, map every access and storage location, verify the employer and subprocessors, determine applicable contractual and legal mechanisms, minimize the fields and retention, and approve access through the buyer’s security and privacy process. Start with sanitized or synthetic material where practical.

What should a machine learning pilot deliver?

A useful pilot delivers evidence: a baseline, data validation, a reproducible pipeline slice, evaluation against an agreed contract, a small serving or batch interface, tests, one failure demonstration, cost and limitation notes, and a buyer-run handover. It need not be a production launch.

Which metric should we put in the contract?

There is no universal metric. Choose a primary decision measure connected to the intended outcome and error costs, then add diagnostics for important cohorts, edge cases, latency, cost, and operational constraints. Record the data, split, threshold, uncertainty, and authorized tradeoff owner. Qualified review may be required for high-impact uses.

How do we prevent vendor lock-in?

Keep repositories, infrastructure accounts, data contracts, evaluation fixtures, model and dependency records, deployment configuration, logs, and runbooks under buyer control or in agreed exportable formats. Test environment reconstruction and handover during the pilot, not only at termination. Record third-party services and the replacement path for each.