Data Cleaning

A common industry practice is to first do small-scale manual verification, then use rules or model scoring for batch cleaning, and finally sample by task difficulty and type in layers to balance training efficiency and generalization ability.

Some back-end AI (such as data cleaning and model fine-tuning platforms) is difficult to present in short videos, making it harder to replicate Cal AI's influencer marketing path.

Over-cleaning may introduce artificial bias; when annotators correct data according to their own understanding, they are in fact injecting subjective judgment into the data, which may be more harmful than the original noise.

When the sample size is large enough, random noise tends to cancel out statistically, and the marginal benefit of cleaning diminishes as the amount of data grows.

In systematic bias, data leakage and duplication, extreme long-tail, and safety-sensitive scenarios, data cleaning remains irreplaceable, because the network cannot correct structural errors on its own.

In data pipeline design, one should first ensure data scale and coverage, and only then consider fine-grained cleaning; monitoring label consistency is more important than reviewing item by item; use validation set performance to infer data quality, rather than presetting cleaning rules.

Data cleaning should include removing bidirectional mixing (retaining only messages sent by oneself as training targets), filtering out meaningless content, balancing the emotional distribution, and manually verifying representative samples.

When handling missing price values, a clear distinction should be made between a price of 0 and an unknown price, and the same default value should not be used for both.

Data obtained through web scraping tends to be unstructured and "dirty," requiring data cleaning and preprocessing.