A vision pilot can appear accurate when every sample is well lit and the defects are obvious. Real production adds reflections, product variants, camera vibration and rare but important faults.
A pilot trained on clean daylight images begins missing scratches during a night shift with different lighting. The evaluation plan should include each operating condition and product variant. Fixing camera illumination may outperform another round of model tuning.
Define a narrow inspection
Pick one product family, one camera position and a few defect classes. Collect examples from different shifts and conditions, including acceptable variation. Have quality staff resolve labeling disagreements before training.
A pilot should include a fallback when the camera lens is dirty or the inspection service is unavailable. Decide whether the line pauses, switches to manual sampling or continues under an approved alternative control. These decisions belong to quality and operations leaders. Model performance on normal days is only part of the deployment case.
Evaluate consequences
Count missed defects and false rejects separately. A false reject may slow a line; a missed defect may reach a customer. Set review thresholds and an operator override. Check whether the camera and lighting can be maintained consistently.
Run a genuine pilot
Test on later production runs, not only images from the development set. Watch the response to alerts and the rate of manual review. The business result is a better inspection process with clear accountability, not a single model accuracy figure.
