nlp_chinese_corpus

The Large-scale Chinese Natural Language Processing Corpus project (nlp_chinese_corpus) aims to contribute high-quality corpora to the development of Chinese natural language processing. It includes multiple structured Chinese datasets ranging from millions to tens of millions of entries, widely applicable to tasks such as large model pretraining, word vector training, text classification, question answering systems, and machine translation. It mainly contains the following datasets: 1. Wikipedia (wiki2019zh): Includes 1 million well-structured Chinese entries, providing id, url, title, and text fields, suitable as general Chinese pretraining corpus or for building knowledge-based Q&A. 2. News corpus (news2016zh): Contains 2.5 million news articles from 63,000 media sources, with titles, keywords, descriptions, main text, and publication time, suitable for news classification, title and keyword generation. 3. Encyclopedia Q&A (baike2018qa): Includes 1.5 million pre-filtered high-quality questions and answers, categorized into nearly 500 topics, suitable for supervised training, sentence similarity computation, and Q&A system construction. 4. Community Q&A (webtext2019zh): Contains 4.1 million high-quality Q&A pairs, filtered by like count (at least 3 likes), covering 28,000 topics, suitable for training ultra-large-scale NLP models, topic prediction, and cQA systems. 5. Chinese-English translation corpus (translation2019zh): Contains 5.2 million parallel Chinese-English sentence pairs, usable for machine translation system training or for extracting Chinese corpus separately.