chardet
An open-source Python character encoding detection library that automatically infers the character set encoding of a byte sequence of text using statistical methods. It is commonly used to handle text data of unknown provenance.
Timeline (last 90 days)
The ingestion engine enforces that all three dependencies chardet, pydantic v2 and orjson are present, and exits explicitly via sys.exit(1) if any is missing.
Statistical charset detection libraries like chardet have known weaknesses in production CSV streams: short records, or payloads mixing ASCII identifiers with regional character sets, receive low-confidence scores and easily trigger misclassification.
Shift_JIS and Windows-31J (CP932) tend to have edge-case conflicts when handling vendor-defined glyphs or legacy NEC/IBM extended characters, and chardet can hardly distinguish the two reliably, often returning a Shift_JIS verdict.
The engineering approach to the CP932 versus Shift_JIS conflict is to try CP932 first as the superset: decode strictly as CP932 first and fall back on failure, rather than relying on a detector to choose between the two.
Production-grade bulk ingestion systems should enforce explicit encoding metadata at the ingestion boundary (HTTP Content-Type headers, S3 object metadata tags, or schema contracts), treating automatic heuristic detection only as a fallback layer.
Most EPUBs are UTF-8 encoded, but occasionally encodings such as Windows-1252 are encountered, which ebooklib handles poorly, requiring chardet for encoding detection.
All Facts (6)
The ingestion engine enforces that all three dependencies chardet, pydantic v2 and orjson are present, and exits explicitly via sys.exit(1) if any is missing.
50%UnverifiedStatistical charset detection libraries like chardet have known weaknesses in production CSV streams: short records, or payloads mixing ASCII identifiers with regional character sets, receive low-confidence scores and easily trigger misclassification.
50%UnverifiedShift_JIS and Windows-31J (CP932) tend to have edge-case conflicts when handling vendor-defined glyphs or legacy NEC/IBM extended characters, and chardet can hardly distinguish the two reliably, often returning a Shift_JIS verdict.
50%UnverifiedThe engineering approach to the CP932 versus Shift_JIS conflict is to try CP932 first as the superset: decode strictly as CP932 first and fall back on failure, rather than relying on a detector to choose between the two.
50%UnverifiedProduction-grade bulk ingestion systems should enforce explicit encoding metadata at the ingestion boundary (HTTP Content-Type headers, S3 object metadata tags, or schema contracts), treating automatic heuristic detection only as a fallback layer.
50%UnverifiedMost EPUBs are UTF-8 encoded, but occasionally encodings such as Windows-1252 are encountered, which ebooklib handles poorly, requiring chardet for encoding detection.
50%