On the first day, a pilot and a production system look almost the same: the same model and the same screens. The difference is the help a pilot gets that nobody writes down. Someone checks every result by hand, the builder fixes problems the moment they appear, and the test data was chosen by someone who knew it would work.
Production takes that help away, and that is where pilots tend to stall. When we are asked to look at a stuck pilot, we run five checks before changing any code. Each one comes from work we have done for our clients.
1. Does everyone agree on what counts as wrong?
A goal like detecting people without helmets sounds clear, but a safety team has its own rules about what counts as a violation.
On the PPE platform we built for Benchmark Data Solutions, we labelled images from their own site footage, so the classes matched their compliance rules exactly. A model trained on someone else’s definition can score well and still be of no use to the people who have to act on it.
2. Was it tested on the hardest conditions, not the average ones?
Averages hide failures. On that PPE project, the difficult cases were known in advance: lighting that changes between shifts, workers partly hidden behind one another, and equipment that looks alike from a distance.
A pilot that shows only clear frames hides all three. Ask which camera, hour or shift is the hardest, and see how the pilot performs there.
3. Who is the safety net?
In a pilot, someone is usually checking every result by hand, often the builder. That works for a small test. At full volume it does not, and by then nobody has planned for that person’s time.
On our port logistics work, rules check each figure against an expected range. Anything outside the range stops for a named reviewer, so people look only at the exceptions. Before launch, write down who catches the errors, what they will see, and how many they can handle in a day.
4. Where does the result go?
A prediction has to reach someone who can act on it. Before our PPE project, compliance was checked after the fact by people reviewing footage, so problems showed up late and inconsistently.
The detection mattered because of what we built around it: a review queue, a record for each flagged frame, and reporting for the compliance record. A pilot that ends with a spreadsheet of predictions has not yet started the real work.
5. Can it run where it will actually live?
Pilots often run on a laptop or in the cloud with a good internet connection. Real sites are different. Cameras produce a lot of video, and the connection is not always reliable.
For Benchmark, we ran the models on hardware at the site itself, so detection does not depend on an uplink. If your pilot needs a perfect connection to work, treat that as a finding, not a detail.
How to use this list
Answer each check with a plain yes or no, and be strict. A maybe counts as a no.
If three or more answers are no, do not add features or try a bigger model. Fix the basics first. It also helps to run the checks with someone who did not build the pilot, because they will ask the questions the builder stopped asking weeks ago.
Once a system is live, a different problem starts: models slowly getting worse over time. We cover that in why AI models fail in production.
Have a pilot that is stuck? Tell us where it is and we will tell you which of these checks it fails. Or see how our 2-week Discovery sprint works.