CLIC
A token-frequency-based visual analysis framework for characterizing programming behavior of large language models. It represents code as token-frequency feature vectors, trains interpretable decision trees, profiles coding style differences across LLMs, and introduces two new metrics: robustness and concentration.
Timeline (last 90 days)
A paper on arXiv proposes a visual analytics method called CLIC (Code Learning for Identification and Comparison), designed to characterize differences in programming behavior across different LLMs.
CLIC represents each code sample as a feature vector of token frequencies, using the frequency of various tokens as features.
CLIC trains an interpretable decision tree to distinguish between the code collections of two LLMs.
CLIC defines two new measurement dimensions: robustness and concentration.
Robustness measures whether two LLMs can still be distinguished after progressively removing the most discriminative tokens.
Concentration measures whether the differences between two models are driven by a small number of dominant tokens or spread across many tokens.
The CLIC paper conducts a case study comparing 10 LLMs on 22 Kaggle machine learning tasks.
The research team states that CLIC's case analysis reveals insights of practical value for model selection and prompt engineering.
The authors developed an interactive visual analytics system for identifying models at a macro level and drilling down to specific discriminative tokens and code context.
CLIC is currently a newly released preprint paper; its conclusions are primarily based on Kaggle machine learning task scenarios, and its generalizability to general software engineering tasks remains to be verified.
All Facts (10)
CLIC is currently a newly released preprint paper; its conclusions are primarily based on Kaggle machine learning task scenarios, and its generalizability to general software engineering tasks remains to be verified.
50%UnverifiedCLIC represents each code sample as a feature vector of token frequencies, using the frequency of various tokens as features.
50%UnverifiedCLIC trains an interpretable decision tree to distinguish between the code collections of two LLMs.
50%UnverifiedCLIC defines two new measurement dimensions: robustness and concentration.
50%UnverifiedRobustness measures whether two LLMs can still be distinguished after progressively removing the most discriminative tokens.
50%UnverifiedConcentration measures whether the differences between two models are driven by a small number of dominant tokens or spread across many tokens.
50%UnverifiedThe CLIC paper conducts a case study comparing 10 LLMs on 22 Kaggle machine learning tasks.
50%UnverifiedThe research team states that CLIC's case analysis reveals insights of practical value for model selection and prompt engineering.
50%UnverifiedThe authors developed an interactive visual analytics system for identifying models at a macro level and drilling down to specific discriminative tokens and code context.
50%UnverifiedA paper on arXiv proposes a visual analytics method called CLIC (Code Learning for Identification and Comparison), designed to characterize differences in programming behavior across different LLMs.
50%