What a Confidence Score Actually Measures
A computer vision model's confidence score is the output of a softmax function applied to the classification layer: it represents the model's relative probability distribution across candidate classes for a given detection. When a model reports a crack detection at 68% confidence, it is saying that 68% of its probability mass was assigned to the "structural crack" class, with the remaining 32% distributed across other possible classifications (surface corrosion, marine growth, shadow artefact, and so on).
This is meaningfully different from a statement about objective probability. The score does not say "there is a 68% chance this is a crack." It says "given this model's training distribution and the features in this image region, the crack class is the most likely interpretation, and the model's internal signal strength for that interpretation is relatively strong but not dominant." The distinction matters because the same numerical score can mean very different things depending on the model's calibration, the difficulty of the detection class, and how similar the current footage is to the training data distribution.
In a road-sign classifier trained on millions of high-resolution images in controlled conditions, a 68% score might genuinely indicate marginal uncertainty. In a subsea crack detector trained on footage captured by multiple ROV operators, at varying depths and lighting conditions, with physical camera properties that differ across survey vessels, a 68% score on a crack detection may represent a genuinely confident finding for a difficult imaging scenario. The numerical value does not translate between domains.
Calibration and What It Means for Inspection Use
A well-calibrated model is one where the confidence scores reflect observed accuracy: when the model says 70%, it should be right roughly 70% of the time across that class. Calibration can be measured by holding out a test set, running detections, and comparing stated confidence bands to actual precision at each band. A model with poor calibration may be systematically overconfident (always claims 90%+ but is right only 70% of the time) or underconfident (claims 55-65% on findings it consistently gets right).
For subsea inspection applications, our internal calibration testing on the crack and structural defect detection classes shows that the model is well-calibrated above the 75% confidence threshold: detections in that range have precision rates in the high 80s to low 90s across the validation footage. Below 60%, the picture is more heterogeneous. Low-confidence detections include both genuinely ambiguous cases (image quality degraded, defect at detection boundary) and cases where the model was uncertain for feature-space reasons that do not correspond to actual defect ambiguity.
The practical implication: high-confidence detections in good imaging conditions from this model can be reviewed by an engineer with a relatively high degree of initial trust. Low-confidence detections warrant careful review with specific attention to why the confidence is low. Is the image quality poor? Is the detection at the edge of the frame? Is there marine growth partially obscuring the region? The low confidence score is a request for an experienced eye, not a dismissal of the finding.
Detection Classes Behave Differently
One of the less-obvious aspects of working with a multi-class detector is that confidence scores are not directly comparable across detection classes. Surface corrosion and marine growth are visually distinct in most imaging conditions; these classes tend to have high confidence scores even in moderately challenging footage. Structural cracks, particularly fine fatigue cracks at weld lines, are inherently difficult to separate from surface texture artefacts and produce a wider distribution of confidence scores even for genuine findings.
Cathodic protection depletion flags tend to be lower-confidence across the board relative to the certainty with which an experienced engineer might identify a depleted anode visually, simply because the training data for this class is less voluminous and the surface cues are more variable. That is a current limitation of the model, not a fundamental claim about the detectability of CP wear from ROV footage.
What this means in practice: an engineer reviewing a report should not apply a single confidence threshold uniformly across all detection classes. A 72% corrosion detection in decent lighting conditions is well into the confident-positive range. A 72% crack detection in the same conditions warrants a more careful look. Our report interface surfaces the detection class alongside the confidence score precisely to support this kind of class-aware interpretation.
Using Confidence Scores to Triage Review Workload
A structured anomaly report from a typical 8-hour ROV survey might contain 80 to 150 flagged findings. Not all of them require the same depth of review. One of the most practical uses of confidence scores is triage: high-confidence detections of high-severity classes (structural cracks, through-coat corrosion, advanced CP depletion) should receive priority review. Lower-confidence detections of lower-severity classes can be reviewed in a second pass.
This is not a recommendation to dismiss low-confidence findings. In an engineering context where the consequences of a missed structural defect can be severe, a false negative is a more consequential error than a false positive. The correct approach is to review all flagged findings, but to sequence the review so that the most critical potential findings are confirmed or discarded first.
Low-confidence findings that cannot be resolved from the flagged frame alone can be addressed by reviewing adjacent frames in the footage sequence, which is why the report retains frame references rather than simply logging a position. A marginal detection that cannot be confirmed at Frame 2,341 may be perfectly clear at Frame 2,345 when the ROV has shifted angle or lighting has changed.
What Engineers Should Ask When Reviewing Flagged Findings
A structured review workflow for automated inspection findings is worth establishing as a team protocol. The questions that consistently improve review quality are:
Does the confidence score match the image quality at the flagged region? If the image is clear and well-lit and the confidence score is low, that is more diagnostic than a low score on a degraded frame. A clear image with a low score is worth investigating whether the model had a feature-space reason for uncertainty.
Is the finding at a structurally critical location? A corrosion flag at a weld line heat-affected zone is a different clinical priority than one on a plain face panel, regardless of confidence score. The model classification does not include structural significance judgment; that is the engineer's contribution.
Does this finding correspond to any prior-survey findings at the same location? Cross-referencing current detections against prior anomaly records from the same structure is one of the highest-value interpretive steps. A finding that has appeared in the same location across three successive surveys at stable severity is a different concern than a new finding with no prior history.
We are not suggesting that confidence scores are a substitute for engineering judgment. The intended role of automated detection is to ensure that no detectable finding in the footage is missed, and to present those findings in a structured form that makes the engineer's review as efficient as possible. The confidence score is one piece of information in that review, not the whole story.
