SOTIF Analysis of LiDAR Object Detection: Evaluating Uncertainty Threshold Transfer Across Triggering Conditions

In autonomous driving perception systems, machine learning-based LiDAR object detectors are widely used, and their safety assessment must comply with the ISO 21448 Safety of the Intended Functionality (SOTIF) standard. This standard requires analysis to cover all possible triggering conditions, not just the single scenario represented by test data. A recent paper addresses this issue by proposing an uncertainty evaluation method to determine whether a performance target set under one condition can be transferred to other triggering conditions. Based on this research, this article outlines its core methods and key findings.

Background: Statistical Guarantee Limitations of Threshold Setting

SOTIF analysis requires ensuring that ML detectors meet performance targets under various triggering conditions. A common approach is to set an uncertainty threshold and reject detections above it, thereby controlling the average per-frame proportion of false detections among accepted detections (i.e., the performance target). However, the paper points out that a threshold selected on data from one condition only has statistical guarantees valid for data from the same distribution as the selection data. When SOTIF analysis must cover all triggering conditions, this guarantee cannot be directly extended. Therefore, a method is needed to evaluate the effectiveness of thresholds after transfer.

ISO 21448 (SOTIF) is an international standard published by ISO for the safety of the intended functionality of road vehicles, specifically addressing hazards caused by functional insufficiencies or reasonably foreseeable misuse in the absence of electrical/electronic system faults. Unlike ISO 26262, which focuses on systematic faults and random hardware faults, SOTIF focuses on performance limitations and insufficient scenario coverage of functions such as perception and decision-making. For machine learning-based LiDAR object detectors, SOTIF requires identifying all triggering conditions that may cause hazards and demonstrating that the system can still meet safety goals under these conditions. Therefore, verifying performance on a single dataset is insufficient; the statistical validity after condition transfer must be considered.

Triggering condition in the SOTIF context refers to specific operational scenarios or environmental factors that can trigger or expose functional insufficiencies of the system, such as rain, long distance, low light, sensor contamination, etc. These conditions may degrade perception performance, and if not adequately evaluated, safety hazards may be missed. The threshold transfer evaluation discussed in this article is precisely to determine whether an uncertainty threshold set under one triggering condition can continue to meet performance targets under other triggering conditions.

Evaluation Method: Transfer, Testing, and Independent Unit Counting

The uncertainty evaluation proposed in the paper consists of three core steps. First, transfer the threshold to the target triggering condition; second, design a statistical test to determine, at a given error probability, whether the performance target is "applicable," "not applicable," or "undetermined" under that condition; third, calculate the lower bound on the number of independent units required for an "applicable" conclusion. Here, independent unit counting is based on recorded sequences rather than frames, because there is dependence between frames and counting must follow the dependence structure.

Independent unit refers to the smallest observation unit that is statistically independent and can be regarded as an independent and identically distributed sample. In time series or continuous frame data, adjacent frames are often highly correlated and cannot be simply treated as independent samples. Counting by frame would overestimate the effective sample size, leading to overly optimistic statistical tests. The paper uses recorded sequences as independent units, assuming that different sequences are independent of each other while allowing dependence between frames within a sequence. This yields a more conservative sample size and more reliable statistical conclusions. Calculating the lower bound on the number of independent units required for an "applicable" conclusion ensures sufficient statistical power to support the performance target and avoids misjudgment due to insufficient samples.

Experimental Setup: PointPillars Ensemble Model and Two Datasets

The study uses a PointPillars ensemble model and conducts experiments on two public datasets: KITTI and nuScenes. The evaluation covers various triggering condition transfer scenarios, including long-distance transfer, rain condition transfer, and transfer between two independently trained detectors. The experiments also compare the differences between counting by frame and counting by recorded sequence.

PointPillars is a point cloud-based 3D object detection network that divides the point cloud into vertical pillars, extracts pillar features using PointNet, and then generates detection boxes through a 2D convolutional network. Its structure is simple, inference is fast, and it is often used as a baseline model for LiDAR detection. The paper uses an ensemble model (a combination of multiple PointPillars models) to reduce the uncertainty of a single model and evaluate the performance of ensemble uncertainty in threshold transfer.

KITTI and nuScenes are two widely used public datasets in the autonomous driving field. KITTI focuses on urban road scenes and includes camera, LiDAR, and GPS/IMU data with 3D bounding box annotations; nuScenes covers complex urban scenes and includes multi-sensor data, more categories, and richer weather/lighting conditions. The two differ in data scale, scene distribution, and annotation specifications, so cross-dataset or cross-condition threshold transfer evaluation is of practical significance.

Key Findings: Insufficient Independent Units and Performance Degradation After Transfer

When counting by recorded sequence, neither dataset provides enough independent units to support any performance target limit, and no transfer yields a "not applicable" result. This indicates that under strict dependence counting, the existing data scale is insufficient to draw definitive conclusions.

When counting by frame, the same data under pre-fixed limits yields "not applicable" results in long-distance transfer, rain transfer, and transfer between two independently trained detectors. Furthermore, after transfers that meet the target limit, the threshold rejects most correct detections; under rain conditions, the area under the receiver operating characteristic curve (AUC) does not decrease. These results reveal the risk of performance degradation that threshold transfer may bring: superficially meeting the target while sacrificing recall.

Conclusions and Implications: Conditional Evidence Requires Local Testing

The paper emphasizes that a performance target met under one condition can only serve as evidence for that condition if it has been statistically tested under the target triggering condition and reported together with the share of accepted correct detections and the number of independent recorded sequences. This means that SOTIF analysis cannot simply assume that thresholds are universally applicable across conditions; each triggering condition must be evaluated independently. This method provides a more rigorous statistical framework for the safety argumentation of ML perception systems and also points out the insufficiency of current datasets in terms of the number of independent units.

For engineers engaged in autonomous driving safety assessment, this research suggests that the selection and transfer of uncertainty thresholds must be handled with care, that the achievement of performance targets must be verified in conjunction with specific triggering conditions, and that the impact of data dependence on statistical conclusions must be fully considered.