A Standardized Framework for Evaluating Language Quality of Chinese Large Language Models
DOI:
https://doi.org/10.13052/jicts2245-800X.1436Keywords:
Chinese large language model, language quality evaluation, standardization, benchmark, LLM-as-a-judge, AI quality model, conformity assessment, pre-standardizationAbstract
Chinese large language models (CLLMs) are rapidly transitioning from research to deployed infrastructure across multiple sectors. Yet current evaluation practice remains fragmented and benchmark-centric, conflating language quality with general capability and weakening comparability across models. This is especially critical in standards-oriented contexts, where language quality must be treated as a multidimensional construct encompassing linguistic correctness, semantic adequacy, discourse coherence, style appropriateness, factual grounding, safety compliance, and robustness. In this study, we propose a standardized framework for evaluating CLLM language quality as a pre-standardization, which integrates five core elements, specifically a layered quality model, scenario-driven test specification, multi-source evidence collection (automated metrics, expert review, calibrated LLM-as-a-judge), transparent scoring and conformity schemes, and governance structures aligned with AI standards practice. Unlike conventional approaches, the framework explicitly separates language quality from broader capability, introduces a hierarchical indicator system tailored to Chinese linguistic phenomena, and defines a reproducible evaluation workflow with quality assurance and version control. To operationalize the proposal, we also develop a reference architecture for quality dimensions, indicators, scoring logic, reporting structure, and pre-standard deliverables. It further demonstrates how the framework can support both research benchmarking and practical deployment scenarios, including enterprise acceptance testing and sector-specific profile extension. The resulting framework provides a technically grounded and standards-oriented blueprint for CLLM language-quality evaluation, with potential value as both a de facto industrial evaluation specification and a foundation for future formal standardization in the information and communication technology (ICT) and artificial intelligence (AI) quality ecosystem.
Downloads
References
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., “Language Models are Few-Shot Learners,” Advances in Neural Information Processing Systems 33, 2020.
R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, et al., “On the Opportunities and Risks of Foundation Models,” arXiv preprint arXiv:2108.07258, 2021.
E. M. Bender, T. Gebru, A. McMillan-Major, and S. Shmitchell, “On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?,” in Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pp. 610–623, 2021.
P. Liang, R. Bommasani, T. Lee, D. Tsipras, D. Soylu, Y. Yao, C. Zhang, D. Narayanan, B. Wu, A. Kumar, et al., “Holistic Evaluation of Language Models,” Transactions on Machine Learning Research, 2023.
M. T. R. Laskar, S. Alqahtani, M. S. Bari, M. Rahman, M. A. M. Khan, H. Khan, I. Jahan, A. Bhuiyan, C. W. Tan, M. R. Parvez, E. Hoque, S. Joty, and J. Huang, “A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendations,” arXiv preprint arXiv:2407.04069, 2024.
D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt, “Measuring Massive Multitask Language Understanding,” in Proceedings of the International Conference on Learning Representations, 2021.
A. Srivastava, A. Rastogi, A. Rao, A. W. M. Shoeb, A. Abid, A. Fisch, A. R. Brown, A. Santoro, A. Gupta, A. Garriga-Alonso, et al., “Beyond the Imitation Game: Quantifying and Extrapolating the Capabilities of Language Models,” Transactions on Machine Learning Research, 2023.
L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica, “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena,” Advances in Neural Information Processing Systems 36, 2023.
Y. Dubois, B. Galambosi, P. Liang, and T. B. Hashimoto, “Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators,” arXiv preprint arXiv:2404.04475, 2024.
D. Kiela, M. Bartolo, Y. Nie, D. Kaushik, A. Geiger, A. H. O. Bassem, E. Dinan, J. Urbanek, A. Szlam, Z. H. Kalyan, et al., “Dynabench: Rethinking Benchmarking in NLP,” in Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4110–4124, 2021.
A. Bavaresco, M. Bernhard, M. Cettolo, F. Dell’Orletta, M. Federico, A. Guerberof, J. H. M. González, K. He, F. Heinemann, G. Houbrechts, et al., “LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks,” in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024.
Y. Huang, Y. Bai, Z. Zhu, J. Zhang, J. Zhang, T. Su, J. Liu, C. Lv, Y. Cui, Y. Yuan, et al., “C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models,” in Advances in Neural Information Processing Systems 36, 2023.
H. Li, Y. Zhang, F. Koto, Y. Yang, H. Zhao, Y. Gong, N. Duan, and T. Baldwin, “CMMLU: Measuring Massive Multitask Language Understanding in Chinese,” in Findings of the Association for Computational Linguistics: ACL 2024, pp. 9853–9879, 2024.
W. Zhong, R. Cui, Y. Guo, Y. Liang, S. Lu, Y. Wang, A. Saied, W. Chen, and N. Duan, “AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models,” in Findings of the Association for Computational Linguistics: ACL 2024, pp. 8794–8823, 2024.
Y. He, W. Sun, Y. Chen, X. Wang, S. Zhou, Z. Wang, S. Jin, Y. Wang, X. Ren, J. Zhao, et al., “A Chinese Factuality Evaluation for Large Language Models,” in Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, 2025.
Q. Cheng, T. Sun, W. Zhang, S. Wang, X. Liu, M. Zhang, J. He, M. Huang, Z. Yin, K. Chen, and X. Qiu, “Evaluating Hallucinations in Chinese Large Language Models,” arXiv preprint arXiv:2310.03368, 2023.
X. Liang, S. Song, S. Niu, Z. Li, F. Xiong, B. Tang, Y. Wang, D. He, P. Cheng, Z. Wang, and H. Deng, “UHGEval: Benchmarking the Hallucination of Chinese Large Language Models via Unconstrained Generation,” in Proceedings of the AAAI Conference on Artificial Intelligence, 2025.
Y. Tan, B. Zheng, B. Zheng, K. Cao, H. Jing, J. Wei, J. Liu, Y. He, W. Su, X. Zhu, B. Zheng, and K. Zhang, “Chinese SafetyQA: A Safety Short-form Factuality Benchmark for Large Language Models,” in Proceedings of the AAAI Conference on Artificial Intelligence, 2025.
Y. Liu, D. Iter, Y. Xu, S. Wang, R. Xu, and C. Zhu, “G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 2511–2522, 2023.
J. Li, X. Cheng, X. Zhao, J.-Y. Nie, and J.-R. Wen, “HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 6449–6464, 2023.
S. Min, K. Krishna, X. Lyu, M. Lewis, W.-t. Yih, P. Koh, M. Iyyer, L. Zettlemoyer, and H. Hajishirzi, “FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 12076–12100, 2023.
K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “BLEU: A Method for Automatic Evaluation of Machine Translation,” in Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311–318, 2002.
C.-Y. Lin, “ROUGE: A Package for Automatic Evaluation of Summaries,” in Text Summarization Branches Out, pp. 74–81, 2004.
T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi, “BERTScore: Evaluating Text Generation with BERT,” in Proceedings of the International Conference on Learning Representations, 2020.
T. Sellam, D. Das, and A. P. Parikh, “BLEURT: Learning Robust Metrics for Text Generation,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7881–7892, 2020.
Z. Li, X. Xu, T. Shen, C. Xu, J.-C. Gu, Y. Lai, C. Tao, and S. Ma, “Leveraging Large Language Models for NLG Evaluation: Advances and Challenges,” in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024.
S. Gehrmann, T. Adewumi, K. Aggarwal, P. A. Ammanamanchi, A. Anadkat, K. Anubhai, J. A. Bale, C. Bandarkar, S. B. Das, B. Gautam, et al., “The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics,” in Proceedings of the 1st Workshop on Natural Language Generation, Evaluation, and Metrics (GEM 2021), pp. 96–120, 2021.
C.-H. Chiang and H.-Y. Lee, “Can Large Language Models Be an Alternative to Human Evaluators?” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15607–15631, 2023.
C. M. Chan, W. Chen, Y. Su, J. Yu, W. Xue, S. Zhang, J. Fu, and Z. Liu, “ChatEval: Towards better LLM-based Evaluators through Multi-Agent Debate,” arXiv preprint arXiv:2308.07201, 2023.
S. Lin, J. Hilton, and O. Evans, “TruthfulQA: Measuring How Models Mimic Human Falsehoods,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3214–3252, 2022.
L. Huang, W. Yu, W. Ma, W. Zhong, Z. Feng, H. Wang, Q. Chen, W. Peng, X. Feng, B. Qin, and T. Liu, “A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions,” arXiv preprint arXiv:2311.05232, 2023.
C. Wang, X. Liu, Y. Yue, X. Tang, T. Zhang, C. Jiayang, Y. Yao, W. Gao, X. Hu, Z. Qi, and Y. Wang, “Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificity,” arXiv preprint arXiv:2310.07521, 2023.
